Database and Repository Installation (Quick)

Overview

Before proceeding with this installation, it is highly recommended to read through the Render Farm Considerations documentation, including the AWS Thinkbox Deadline Shared Responsibility Model. It outlines what AWS Thinkbox is responsible for and what customers are responsible for, and should be considered before installing and using Deadline.

The Database is the global database component of the Deadline Render Farm Management System. It stores the jobs, settings, and Slave configurations. The Clients access the Database via a direct socket connection over the network. It only needs to be installed on one machine (preferably a server), and does not require a license. Deadline uses MongoDB for the Database.

The Repository is the global file system component of the Deadline Render Farm Management System. It stores the plugins, scripts, logs, and any auxiliary files (like scene files) that are submitted with the jobs. The Clients access the Repository via a shared network path. It only needs to be installed on one machine (preferably a server), and does not require a license.

The Database and Repository together act as a global system where all of Deadline’s data is stored. The Clients then connect to this system to submit, render, and monitor jobs. It is important to note that while the Database and Repository work together, they are still separate components, and therefore can be installed on separate machines if desired.

Note

The Repository installer requires an internet connection to download and install the MongoDB database. If your operating system isn’t supported, or if an internet connection isn’t available, you must manually install the database by following the Manual Database Installation documentation. When running the Repository installer, you can then choose the option to connect to an existing MongoDB database.

Installation

While the Repository can be installed on any operating system, the Repository installer is only available for Windows, Linux, and Mac OS X. However, the machine that you run the Repository installer on doesn’t have to be the same machine you’re installing the Repository to. For example, if you have an existing share on a FreeBSD server or a NAS system, you can run the Repository installer on Windows, Linux, or Mac OS X and choose that share as the install location.

To install the Repository, simply run the appropriate installer for your operating system and follow the steps. This procedure is identical for all operating systems. The Repository installer also supports silent installations.

setup directory

When choosing the Installation Directory, you can choose either a local path on the current machine, or the path to an existing network share. Note that if you choose a local path, you must ensure that path is shared on the network so that the Clients can access it.

Warning

Do NOT install over an existing installation unless it’s the same major version, or there could be unexpected results.

If you’re installing over an existing Repository installation, all previous binaries, plug-ins, and scripts will be backed up prior to being overwritten. After the installation is complete, you can find these backed up files in the Backup folder in the Repository installation root. Note that installing over an existing repository is only supported for repairing a damaged repository, or for performing a minor upgrade. Major upgrades require a fresh repository installation. See the Upgrading or Downgrading Deadline Documentation for more information.

After choosing the Repository Installation directory, you will be asked to install a new MongoDB Database, or connect to an existing one.

mongodb_setup

Installing a New MongoDB Database

If you choose to install a new MongoDB Database, you can then choose to have the installer download MongoDB for you. Note though that this requires an internet connection. Please ensure the MongoDB version is minimum 3.2.12 and maximum 3.2.18.

If you do not have an internet connection on the machine you’re running the installer on, your other choice is to simply download the MongoDB pre-packaged binaries from another machine with an internet connection. Just make sure to download the .tgz or .zip file for your operating system (and not the .msi file if you’re installing on a Windows machine). After downloading the MongoDB package, copy it to the machine you’re running the installer on, and then specify the path to it in the Repository installer.

Note

When specifying the path to the pre-packaged binaries, the Repository Installer is expecting a .tgz or .zip file that was downloaded from MongoDB’s website. It does not support binaries that were manually bundled.

mongodb_install_type

No matter your choice, you will then be asked to choose an installation location, a hostname and a port number.

Warning

Ensure you choose locally attached storage to install the Database. Do NOT install the DB to a NAS/network drive. Do NOT share out the DB directory once installed.

mongodb_settings

The MongoDB Hostname is the host name or the IP address of the machine that the MongoDB database will be running on. If desired, you can specify multiple entries and separate them with semicolons. There are a couple reasons to specify multiple entries:

  • You have machines on different subnets that need to access the database differently (ie: machines in the cloud might use a different host name than machines on the local network).
  • Some machines need to resolve the database machine by its host name, and others need to use its IP address.

Note that if there are IP addresses listed that cannot be resolved, the Deadline Command application can run slower on Linux and Mac OS X Clients because it won’t exit until the connection attempt for those IP addresses time out.

Next, you need to specify security options for the Database.

mongodb_security_settings

Although We strongly recommend enabling SSL/TLS authentication, you may choose to disable it by unchecking the Require client authentication via SSL/TLS checkbox. If it is enabled, you are presented with the following options:

  • Certificate Directory: The installer will generate the required certificates for authentication and place them in this directory. This is where you will find a client certificate that can be used to connect to the Database.
  • Certificate Password: You may associate a password with the client certificate. If you do, clients will require this password in order to connect to the Database. If you do not wish to set a password, leave this field blank.
  • Use client certificate for DB user authentication: If enabled, the client certificate will also be used for Database user authentication. The installer will create a user on the Database corresponding to the client certificate that is generated.

Note

Once the certificates have been generated, the ‘Deadline10Client.pfx’ file will need to be securely moved to client machines that are expected to connect to this Repository directly; it will be needed at connection time. In addition, the user(s) running the client processes will need to be granted Read access to the file (the installer creates the file with restrictive permissions).

Warning

It is important to be aware of who might have access to the file throughout the process of distributing it to the client machines – the operating system might change the access level of the file to be overly broad on a copy, for example. We recommend always maintaining each certificate with the minimal subset of permissions for the users that need to connect to the Deadline Repository.

When you press Next, the installer will install the database and try to connect to it and configure it for Deadline. This can take a minute or two. If an error occurs, you will be prompted with the error message. If the setup succeeds, you can then proceed with the installation of the Repository.

Connecting to an Existing MongoDB Database

Warning

Do NOT install over an existing installation unless it’s the same major version, or there could be unexpected results.

If you choose to connect to an existing MongoDB Database, you will be asked to specify the Database Settings so that the installer can connect to and configure the Database. Please ensure the MongoDB version is minimum 3.2.12 and maximum 3.2.18.

mongodb_connection_settings

These settings will also be used by the Clients to connect to the database. The following are required:

  • Database Server: The host name or the IP address of the machine that the MongoDB database is running on. If desired, you can specify multiple entries and separate them with semicolons.
  • Database Port: The port that the MongoDB database is listening on.
  • Database Name: The name of the Database. Make sure to enter the same name you used when you initially set up the Database. If you installed the Database using the Repository installer, it should be sufficient to leave it set to the default value.
  • Replica Set: If you set up your MongoDB database manually and it is part of a Replica Set, specify the Replica Set Name here. If you don’t have Replica Set, just leave this blank.

Next, you will be asked to provide security information for connecting to the Database.

mongodb_connection_ssl_settings

If the Database does not have SSL authentication enabled, uncheck the Use SSL/TLS when communicating with server option. Otherwise, provide the following options:

  • Client Certificate: This is the path to a client SSL certificate to use when connecting to the Database.
  • Certificate Password: This is the password for the Database SSL certificate. If the certificate does not require a password, leave this field blank.
  • Use client certificate for DB user authentication: Enable this if the client certificate is also used for Database user authentication. Note that a user corresponding to the client certificate must already exist on the Database.

When you press Next, the installer will try to connect to the database using these settings to configure it. This can take a minute or two. If an error occurs, you will be prompted with the error message. If the setup succeeds, you can then proceed with the installation of the Repository.

Open Firewall Ports

To ensure that the Deadline applications can communicate with MongoDB, you will need to update the firewall on the machine that MongoDB is running on. You can either disable the firewall completely (assuming it operates in an internal network), or you can open the port that you chose for the database to use during install. More information on opening ports can be found below.

Windows

Open Windows Firewall with Advanced Security. Click on Inbound Rules in the left panel to view all inbout rules, and then right-click on Inbound Rules and select New Rule to start the Inbound Rule Wizard. Select Port for the Rule Type, and then click Next.

new_rule rule_type

On the Protocol and Ports page, choose TCP, and then specify the port that you chose for the database during the install, and then press next. Then on the Action page, choose Allow The Connection and press Next.

rule_port rule_allow

On the Profile page, choose the networks that this rule applies to, and then press next. Then on the Name page, specify a name for the rule (for example, MongoDB Connection), and then press Finish.

rule_network rule_name

Linux

On RedHat and CentOS, the following commands should allow incoming connections to the Mongo database if iptables are being used. Just make sure to specify the port that you chose for the database during the install.

>>> sudo iptables -I INPUT 1 -p tcp --dport [DB_PORT] -j ACCEPT
>>> sudo ip6tables -I INPUT 1 -p tcp --dport [DB_PORT] -j ACCEPT

where [DB_PORT] is the MongoDB port such as: 27100

Ubuntu has no firewall installed by default, and we have not yet tested Fedora Core’s FirewallD.

Mac OS X

Mac OS X has its firewall disabled by default, but if enabled, it is possible to open ports for specific applications. Open up System Preferences,, choose the Security & Privacy option, and click on the Firewall tab.

../_images/mac_firewall.png

Press the Firewall Options button to open the firewall options. Press the [+] button and choose the path to the mongod application, which can be found in the database installation folder in mongo/application/bin (for example, /Applications/Thinkbox/DeadlineDatabase<VERSION>/mongo/application/bin/mongod). Then click OK to save your settings.

../_images/mac_firewall_options.png

Sharing The Repository

For Deadline to operate properly, the Repository must be visible to all machines that are connecting to it. This section explains how to share your Repository folder and configure its permissions to ensure the Clients have necessary access.

The clients must have read access to the Repository root and all of its subdirectories. In addition, the Clients should have full read/write access to the following folders in the Repository:

  • jobs: This is where job auxiliary files are copied to during submission.
  • jobsArchived: This is where archived jobs are exported to.
  • reports: This is where the physical log files for job and Slave reports are saved to.

It is possible to achieve this by setting open read/write/execute permissions to the Repository root and all of its subfolders. While this is the easiest to configure, it does introduce some possible security risks. The following folders are especially sensitive because they contain Python scripts that will be executed on the render machines. We recommend that these folders be restricted to read/execute permissions only (no write access):

  • plugins: Contains Python scripts for the various render plugins supported by Deadline.
  • events: Contains Python scripts for event plugins.
  • scripts: Contains Python scripts that can be run from the Monitor.
  • custom: This directory can house additional plugins, events, and scripts.

It’s also worth noting that jobs often have additional auxiliary files submitted with them, which are normally stored in the jobs folder in the Repository, unless an alternate directory for auxiliary files is configured in the Repository Options. These auxiliary files can range from simple configuration files to scene files. Since these files may contain sensitive information, it is recommended to restrict write access to the jobs directory only to trusted users.

The guides below describe how to share the Repository with full read/write permissions.

Windows

Note

The images shown here are from Windows 10, but the steps are virtually identical for other versions of Windows.

Note

It is assumed that you are performing these instructions as a user with elevated permissions. On Windows, this is an administrator-type account that can execute commands as an administrator.

Note

For the purpose of this setup, we will create a local user account named deadlineuser. This user will be granted appropriate access to the Repository. Other non-elevated users shouldn’t be able to access the Repository.

First we will configure the appropriate Repository folder permissions.

  1. On the machine where the Repository is installed, navigate to the installation folder with the Windows Explorer.
  2. Right-click on the Repository folder, selecting Properties from the context menu.
  3. Select the Security tab and click Edit… to modify folder permissions.
../_images/windows_repository_initial_permissions.png
  1. Remove the Authenticated Users and Everyone groups. This can be done by selecting the corresponding entry from the Group or user names list and clicking the Remove button.

  2. Add deadlineuser with restricted access:
    1. Click the Add… button.
    2. Enter deadlineuser into the Enter the object names to select text field.
    ../_images/windows_select_users_or_groups.png
    1. Click OK.
    2. Select deadlineuser from the Group or user names list.
    3. Ensure deadlineuser only has permissions to allow Read, Read & execute, and List folder contents.
    ../_images/windows_repository_final_permissions.png
    1. Click OK to save the Repostiory folder permission settings.
  3. Add full access for deadlineuser to the Repository’s jobs/, jobsArchived/, and reports/ folders. After navigating into the Repository folder, the following steps should be completed for each of these folders.
    1. Right-click on the folder and select Properties from the context menu.
    2. Select the Security tab and click Edit… to modify folder permissions.
    3. Select deadlineuser from the Group or user names list.
    4. Ensure deadlineuser has permissions to allow Modify and Write.
    ../_images/windows_repository_full_control_permissions.png
    1. Click OK to save the folder permission settings. Complete these steps for the jobs/, jobsArchived/, and reports/ folders.

Attention

At this point, your Repository should only be accessible by deadlineuser (and admin users). The Repository folder (and all subfolders) should be able to be read and executed from by deadlineuser but not written to. Only the Repository subfolders jobs/, jobsArchived/, and reports/ should be able to be written to and modified.

Now we need to make sure the Repository folder is shared. Again, the images shown here are from Windows 10, but the steps are virtually identical for other versions of Windows.

  1. On the machine where the Repository is installed, navigate to the installation folder with the Windows Explorer.
  2. Right-click on the Repository folder, selecting Properties from the context menu.
  3. Select the Sharing tab.
../_images/windows_repository_sharing.png
  1. Click the Advanced Sharing… button.
  2. In the dialog, check the Share this folder checkbox.
../_images/windows_repository_sharing_advanced.png
  1. Click the Permissions button.
  2. Ensure Everyone has permissions to allow Full Control.
../_images/windows_repository_sharing_permissions.png
  1. Press OK to apply the permissions, then press OK again to share the Repository folder.
  2. Close the Repository folder Properties dialog.

Linux

Note

It is assumed that you are performing these instructions as a user with elevated permissions. On Linux, this is a user that can execute commands through sudo.

Note

For the purpose of this setup, we will create a user named deadlineuser. This user will be granted appropriate access to the Repository. Other non-elevated users shouldn’t be able to access the Repository.

First we will configure the appropriate Repository folder permissions.

  1. Change the owner of the Repository to be deadlineuser:

    >>> sudo chown -R deadlineuser:nogroup /path/to/repository
    

Note

Some distributions of Linux use nobody instead of nogroup.

  1. Add restricted access permissions to the Repository folder for the owner (deadlineuser is the owner at this point):

    >>> sudo chmod -R 500 /path/to/repository
    

Note

This sets permissions for all subfolders and files such that only the owner is able to read/execute. Other users won’t have access to the Repository folder.

  1. Add full access permissions for the owner (deadlineuser) to the Repository’s jobs/, jobsArchived/, and reports/ folders.

    >>> cd /path/to/repository
    >>> sudo chmod -R 700 jobs/ jobsArchived/ reports/
    

Attention

At this point, your Repository should only be accessible by deadlineuser (and admin users). The Repository folder (and all subfolders) should be able to be read and executed from by deadlineuser but not written to. Only the Repository subfolders jobs/, jobsArchived/, and reports/ should be able to be written to and modified.

Now you’re ready to set up your network sharing protocol. The recommended sharing protocol for Linux is Samba.

Note

You may not have Samba installed on your Linux machine. On Ubuntu, Samba can be installed with:

>>> sudo apt-get update
>>> sudo apt-get install samba

To share the Repository directory, modifications need to be made to /etc/samba/smb.conf. Add the following to the end of smb.conf:

[DeadlineRepository]
path = /path/to/repository
writeable = Yes
guest ok = No
create mask = 0700
force create mode = 0700
force directory mode = 0700
unix extensions = No

Once these modifications are saved, we need to setup a password for our Samba user (you will be prompted for the password):

>>> sudo smbpasswd -a deadlineuser

Now we can restart the Samba sharing service:

>>> sudo service smbd restart

On Linux clients, the Samba share can be mounted using cifs-utils:

>>> sudo mkdir /mnt/repo
>>> sudo mount -t cifs -o username=deadlineuser,password=<password> //<samba_server_address>/DeadlineRepository /mnt/repo

Note

<samba_server_address> must either be the IP address or hostname of the machine hosting the Samba share (likely the machine with the Repository installed).

Mac OS X

Note

The images shown here are from OS X El Capitan (10.11.4), but the steps are virtually identical for other versions of OS X.

Note

For the purpose of this setup, we will create a user named deadlineuser. This user will be granted appropriate access to the Repository. Other non-elevated users shouldn’t be able to access the Repository.

First we will configure the appropriate Repository folder permissions.

  1. On the machine where the Repository is installed, navigate to the installation folder with the Finder.

  2. Right-click on the Repository folder and select Get Info from the context menu.

  3. Expand the Sharing & Permissions section and unlock the settings (if necessary).

  4. Change permissions for the everyone group to No Access. This will prevent undesignated users from accessing the Repository.

  5. Make sure (this should already be the case) that the admin group has full Read & Write permissions.

  6. Add deadlineuser with restricted access:
    1. Click the “+” button at the bottom left of the dialog.
    2. Select deadlineuser from the Users & Groups tab.

    Note

    Users in this list appear as their “Full Name” rather than their actual username. This must be specified when the user is created.

    1. Make sure deadlineuser has Read only permissions.
../_images/osx_repository_initial_permissions.png
  1. Apply permissions to nested files and folders.
    1. Click the gear/cog dropdown button at the bottom left of the dialog.
    2. Select Apply to enclosed items….
  2. Add full access permissions for deadlineuser to the Repository’s jobs/, jobsArchived/, and reports/ folders. After navigating into the Repository folder, the following steps should be completed for each of these folders.
    1. Right-click on the folder and select Get Info from the context menu.
    2. Expand the Sharing & Permissions section and unlock the settings (if necessary).
    3. Change permissions for deadlineuser to Read & Write (should already be Read only).

If you prefer to set the permissions from the Terminal, run the following commands:

$ sudo chown -R root:admin /path/to/repository
$ sudo chmod -R 770 /path/to/repository
$ sudo chmod -R +a "deadlineuser allow list,readattr,readextattr,readsecurity" /path/to/repository
$ cd /path/to/repository
$ sudo chmod +a "deadlineuser allow list,add_file,search,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity" jobs/ jobsArchived/ reports/

Attention

At this point, your Repository should only be accessible by deadlineuser (and admin users). The Repository folder (and all subfolders) should be able to be read and executed from by deadlineuser but not written to. Only the Repository subfolders jobs/, jobsArchived/, and reports/ should be able to be written to and modified.

Now we need to make sure the Repository folder is shared.

  1. Open System Preferences and select the Sharing settings.
  2. Make sure the File Sharing checkbox is checked/enabled.
  3. Add the Repository folder to the Shared Folders list by clicking the “+” button beneath the list.
  4. Make sure that Everyone has full Read & Write access.
../_images/osx_repository_sharing.png

Note

It is recommended to use the Samba sharing protocol. This can be specified by clicking Options… and checking the Share file and folders using SMB checkbox.