# Database and Repository Installation¶

## Overview¶

Before proceeding with this installation, it is highly recommended to read through the Render Farm Considerations documentation.

The Database is the global database component of the Deadline Render Farm Management System. It stores the jobs, settings, and slave configurations. The Clients access the Database via a direct socket connection over the network. It only needs to be installed on one machine (preferably a server), and does not require a license. Deadline uses MongoDB for the Database.

The Repository is the global file system component of the Deadline Render Farm Management System. It stores the plugins, scripts, logs, and any auxiliary files (like scene files) that are submitted with the jobs. The Clients access the Repository via a shared network path. It only needs to be installed on one machine (preferably a server), and does not require a license.

The Database and Repository together act as a global system where all of Deadline’s data is stored. The Clients then connect to this system to submit, render, and monitor jobs. It is important to note that while the Database and Repository work together, they are still separate components, and therefore can be installed on separate machines if desired.

The Repository installer can install the MongoDB database for you, but you can also choose to connect to an existing MongoDB installation.

## Installation¶

While the Repository can be installed on any operating system, the Repository installer is only available for Windows, Linux, and Mac OS X. However, the machine that you run the Repository installer on doesn’t have to be the same machine you’re installing the Repository to. For example, if you have an existing share on a FreeBSD server or a NAS system, you can run the Repository installer on Windows, Linux, or Mac OS X and choose that share as the install location.

To install the Repository, simply run the appropriate installer for your operating system and follow the steps. This procedure is identical for all operating systems. The Repository installer also supports silent installations.

When choosing the Installation Directory, you can choose either a local path on the current machine, or the path to an existing network share. Note that if you choose a local path, you must ensure that path is shared on the network so that the Clients can access it. Do not install over an existing installation unless it’s the same major version, or there could be unexpected results.

If you’re installing over an existing Repository installation, all previous binaries, plug-ins, and scripts will be backed up prior to being overwritten. After the installation is complete, you can find these backed up files in the Backup folder in the Repository installation root. Note that installing over an existing repository is only supported for repairing a damaged repository, or for performing a minor upgrade. Major upgrades require a fresh repository installation. See the Upgrading or Downgrading Deadline Documentation for more information.

After choosing the installation directory, you will be asked to install the MongoDB Database, or connect to an existing one. If you choose to install the MongoDB Database, you will be asked to choose an installation location and a port number. It is highly recommended that you choose a local directory to install the Database.

Next, you need to specify the Database Settings so that the installer can set up the Database. These settings will also be used by the Clients to connect to the database. The following are required:

• Database Server: The host name or the IP address of the machine that the MongoDB database is running on. If desired, you can specify multiple entries and separate them with semicolons. There are a couple reasons to specify multiple entries:

• You have machines on different subnets that need to access the database differently (ie: machines in the cloud might use a different host name than machines on the local network).
• Some machines need to resolve the database machine by its host name, and others need to use its IP address.
Note that if there are IP addresses listed that cannot be resolved, the Deadline Command application can run slower on Linux and OSX Clients because it won’t exit until the connection attempt for those IP addresses time out.
• Database Port: The port that the MongoDB database is listening on.
• Database Name: The name of the Database. If you are setting up a new Database, you can leave this as the default. If you are connecting to an existing Database, make sure to enter the same name you used when you initially set up the Database.
• Replica Set: If you set up your MongoDB database manually and it is part of a Replica Set, specify the Replica Set Name here. If you don’t have Replica Set, just leave this blank.

When you press Next, the installer will try to connect to the database using these settings to configure it. This can take a minute or two. If an error occurs, you will be prompted with the error message. If the setup succeeds, you can then proceed with the installation of the Repository.

### Command Line or Silent Installation¶

The Repository installer can be run in command line mode or unattended mode on each operating system. Note though that on Mac OS X, you must run the installbuilder.sh script that can be found in the Contents/MacOS folder, which is inside the Mac Repository Installer package.

To run in command line mode, pass the “–mode text” command line option to the installer. For example, on Linux:

./DeadlineRepository-X.X.X.X-linux-x64-installer.run --mode text


To run in silent mode, pass the “–mode unattended” command line option to the installer. For example, on Windows:

DeadlineRepository-X.X.X.X-windows-installer.exe --mode unattended


To get a list of all available command line options, pass the “–help” command line option to the installer. For example, on Mac OS X:

/DeadlineRepository-X.X.X.X-osx-installer.app/Contents/MacOS/installbuilder.sh --help


Note that there are a few Repository installer options that are only available from the command line, which you can view when running the “–help” command. These options include:

• --help: Display the list of valid options.
• --version: Display product information.
• --unattendedmodeui <unattendedmodeui>: Unattended Mode UI.
• --optionfile <optionfile>: Installation option file.
• --debuglevel <debuglevel>: Debug information level of verbosity.
• --mode <mode>: Installation mode.
• --debugtrace <debugtrace>: Debug filename.
• --installer-language <installer-language>: Language selection.
• --prefix <prefix>: The path to install the Repository to.
• --backuprepo <backuprepo>: If set to true, many folders in the Repository will be backed up before overwriting them.
• --mongodir <mongodir>: If MongoDB is being installed by this installer, this is the path it is installed to.
• --dbListeningPort <dbListeningPort>: MongoDB Port.
• --dbhost <dbhost>: The host name or IP address of the machine that MongoDB is running on.
• --dbport <dbport>: The port that MongoDB is listening on.
• --dbname <dbname>: The name of the MongoDB database that the data is stored in.
• --dbreplicaset <dbreplicaset>: The name of the MongoDB database replica set, if you have one set up.
• --dbuser <dbuser>: The user name to connect to MongoDB with.
• --dbpassword <dbpassword>: The password to connect to MongoDB with.
• --dbauth <dbauth>: If enabled, Deadline will use this account to connect to MongoDB.
• --dbsplit <dbsplit>: If enabled, the database collections will be split into separate databases to improve performance.

### Database Config File¶

A file called config.conf is installed to the data directory in the database installation folder. This file is used to configure the MongoDB database, and can be modified to add or change functionality. This is what you will typically see by default:

#MongoDB config file

#where to log
systemLog:
destination: file
quiet: true
#verbosity: <integer>

#port for mongoDB to listen on
#uncomment below ipv6 and REST option to enable them.
net:
port: 27070
#ipv6: true
#http:
#RESTInterfaceEnabled: true

#where to store the data
storage:

#enable sharding
#sharding:
#clusterRole
#configDB

#setup replica set with give replica set name
#replication:
#replSetName

#enable authentication
#security:
#authorization: enabled


After making changes to this file, simply restart the mongod process for the changes to take effect. See the MongoDB Configuration File Options for more information on the available options.

### Manual Database Installation¶

The Repository installer installs MongoDB with the bare minimum settings required for Deadline to operate. Manually installing the Database might be preferable for some because it gives you greater control over things like authentication, and allows you to create sharded clusters or replica sets for backup.

If you wish to install MongoDB manually, you can download MongoDB from the MongoDB Downloads Page. Once MongoDB is running, you can then run the Repository installer, and choose to connect to an existing MongoDB Database. Here are some helpful links for manually installing the MongoDB database:

MongoDB also has a management system called MMS. It’s a cloud service that makes it easy to provision, monitor, backup, and scale your MongoDB databse. Here are some helpful links for setting up and using MMS:

The Automation Agent mentioned above makes it possible to setup your MongoDB database from a web interface, and easily configure which MongoDB servers are replica sets or shards. It also allows you to easily upgrade the version of your MongoDB database. Here are some additional links for how you can use the Automation Agent:

Note though that as of this writing, the Automation Agent is only available for Linux and Mac OS X.

### Database Resource Limits¶

Linux and Mac OS X systems impose a limit on the number of resources a process can use, and these limits can affect the number of open connections to the database. It is important to be aware of these limits, and make sure they are set appropriately to avoid unexpected behaviour. Note that MongoDB will allocate 80% of the system limit for connections, so if the system limit is 1024, the maximum number of connections will be 819.

If you choose a Linux system to host the database, make sure the system limits are configured properly to avoid connection issues. See MongoDB’s Linux ulimit Settings documentation for more information, as well as the recommended system limits to use.

You can check your current Linux/OSX ulimit settings in a terminal shell:

#overall ulimit settings on the machine
ulimit -a
#number of open files allowed
ulimit -n


MongoDB provides these Recommended ulimit Settings for optimal performance of your database. Note, you must restart the Deadline Database daemon after changing these ulimit settings.

If you choose a Mac OS X system to host the database, and you use the Repository installer to install the database, the resource limits will be set to 1024. These limits can be adjusted later by manually editing the HardResourceLimits and SoftResourceLimits values in /Library/LaunchDaemons/org.mongodb.mongod.plist after the Repository installer has finished.

## Open Firewall Ports¶

To ensure that the Deadline applications can communicate with MongoDB, you will need to update the firewall on the machine that MongoDB is running on. You can either disable the firewall completely (assuming it operates in an internal network), or you can open the port that you chose for the database to use during install. More information on opening ports can be found below.

### Windows¶

Open Windows Firewall with Advanced Security. Click on Inbound Rules in the left panel to view all inbout rules, and then right-click on Inbound Rules and select New Rule to start the Inbound Rule Wizard. Select Port for the Rule Type, and then click Next.

On the Protocol and Ports page, choose TCP, and then specify the port that you chose for the database during the install, and then press next. Then on the Action page, choose Allow The Connection and press Next.

On the Profile page, choose the networks that this rule applies to, and then press next. Then on the Name page, specify a name for the rule (for example, MongoDB Connection), and then press Finish.

### Linux¶

On RedHat and CentOS, the following commands should allow incoming connections to the Mongo database if iptables are being used. Just make sure to specify the port that you chose for the database during the install.

sudo iptables -I INPUT 1 -p tcp --dport 27070 -j ACCEPT
sudo ip6tables -I INPUT 1 -p tcp --dport 27070 -j ACCEPT


Ubuntu has no firewall installed by default, and we have not yet tested Fedora Core’s FirewallD.

### Mac OS X¶

Mac OS X has its firewall disabled by default, but if enabled, it is possible to open ports for specific applications. Open up System Preferences,, choose the Security & Privacy option, and click on the Firewall tab.

Press the Firewall Options button to open the firewall options. Press the [+] button and choose the path to the mongod application, which can be found in the database installation folder in mongo/application/bin (for example, /Applications/Thinkbox/DeadlineDatabase7/mongo/application/bin/mongod). Then click OK to save your settings.

## Sharing The Repository Folder¶

In general, the Repository must have open read and write permissions for Deadline to operate properly. This section explains how to share your Repository folder and configure its permissions to ensure the Clients have full access. Without full read/write access, the Client applications will not be able to function properly.

Note that this guide is for applying full read/write permissions to the entire Repository folder structure. For the more advanced user, it is possible to enforce tighter restrictions on the Repository folders. Just make sure the Clients have full read/write access to the following folders in the Repository. The rest must have at least read access.

• jobs: This is where job auxiliary files are copied to during submission.
• jobsArchived: This is where archived jobs are exported to.
• reports: This is where the physical log files for job and slave reports are saved to.

### Windows¶

First, you need to configure the Repository folder permissions. Note that the images shown here are from Windows XP, but the procedure is basically the same for any version of Windows.

• On the machine where the Repository is installed, navigate to the folder where it is installed using Windows Explorer.
• Right-click on the Repository folder and select Properties from the menu.
• Select the Security tab.
• If there is already an Everyone item under Group or user names, you can skip the next two steps.
• Click on the Add button.
• In the resulting dialog, type Everyone and click OK.
• Select Everyone under Group or user names.
• Ensure that Modify, Read & Execute, List Folder Contents, Read, and Write are all checked under the Allow column.
• Click on the OK button to save the settings.

Second, you need to share the Repository folder. Note that the images shown here are from Windows XP, but the procedure is basically the same for any version of Windows.

• On the machine where the Repository is installed, navigate to the folder where it is installed using Windows Explorer.
• Right-click on the Repository folder and select Properties from the menu. If you’re unable to see the Sharing tab, you may need to disable Simple File Sharing in the Explorer Folder Options.
• Select the Sharing tab.
• Select the option to Share This Folder, then specify the share name.
• Click the Permissions button.
• Give Full Control to the Everyone user.
• Press OK on the Permissions dialog and then the Properties dialog.

### Linux¶

Since the Clients expects full read and write access to the repository, it’s recommended to use a single user account to mount shares across all machines. It is possible to add particular users to a ‘deadline’ group, but you will need to experiment with that on your own.

So for both of the sharing mechanisms we explain below, you’ll need to create a user and a group named ‘deadline’. They don’t need a login or credentials, we just need to be able to set files to be owned by them and for their account to show up in /etc/passwd. So, to do this use the useradd command.

sudo useradd -d /dev/null -c "Deadline Repositry User" -M deadline


This should create a user named “deadline” with no home folder, and a fancy comment. The account login should also be disabled, meaning your standard users can’t ssh or ftp into your file server using this account. Set a password using sudo passwd deadline if you need your users to login as deadline using ftp or ssh.

sudo groupadd deadline


And finally, have the Repository owned by this new user and group

sudo chown -R deadline:deadline /path/to/repository
sudo chmod -R 777 /path/to/repository


Now you’re ready to set up your network sharing protocol. There are a many ways this can be done, and this just covers a few of them.

Samba Share

This is an example entry in the /etc/samba/smb.conf file:

[DeadlineRepository]
path = /path/to/repository
writeable = Yes
guest ok = Yes
force create mode = 0777
force directory mode = 0777
unix extensions = No


NFS Share

The simplest thing that could possibly work. Note that this is not the most secure thing that could possibly work:

For Linux and BSD, open up /etc/exports as an administrator, and make one new export:

/path/to/repository                  192.168.2.0/24(rw,all_squash,insecure)


Breakdown of this command is as follows:

• /path/to/repository: The Repository folder to share. Change the path as necessary.
• 192.168.2.0/24: The IP range to allow. The zero is important for these ranges. You can also go by hostname if you have reverse DNS, or * to allow from anyone’s computer.
• rw: Allow read/write for the repository, which is required for the Clients to operate properly.
• all_squash: Make every single person who connects to the Repository share map to the nobody:nogroup user and group. This relieves a lot of permissions pain for new users at the cost of zero security. Files and folders within your repository will be fully readable and writeable by whomever is able to connect to your NFS server. The Clients require this, but it can also be achieved by creating a group and adding individual users into that group. Many studios will only need all_squash as Deadline will keep track of who submits what jobs.
• insecure: Required for Mac OS X to mount nfs shares. It simply means that NFS doesn’t need to receive requests on a port in the secure port range (a port number less than 1024).

Once that’s done, you may need to install an NFS server. To do so, open a terminal or your favourite package manager to install one. For Ubuntu Server, type the following:

sudo apt-get install nfs-kernel-server


Then start up the server (for those living in an init.d world):

sudo /etc/init.d/nfs-kernel-server start


Any time you change the exports file, you’ll need to issue the same command, but replace ‘start’ with ‘reload’.

There is an excellent tutorial here as well: https://help.ubuntu.com/community/SettingUpNFSHowTo

### Mac OS X¶

First, you need to configure the Repository folder permissions. Note that the images shown here are from Leopard (10.5), but the procedure is basically the same for any version of Mac OS X.

• On the machine where the Repository is installed, navigate to the folder where it is installed using Finder.
• Right-click on the Repository folder and select Get Info from the menu.
• Expand the Sharing & Permissions section, and unlock the settings if necessary.
• Give everyone Read & Write privileges.
• While probably not necessary, also give admin Read & Write privileges.

If you prefer to set the permissions from the Terminal, run the following commands:

$chown -R nobody:nogroup /path/to/repository$ chmod -R 777 /path/to/repository


Now you can share the folder. There are a many ways this can be done, and this just covers a few of them.

Using System Preferences

Note that the images shown here are from Leopard (10.5), but the procedure is basically the same for any version of Mac OS X.

• Open System Preferences, and select the Sharing option.
• Make sure File Sharing is enabled, and then add the Repository folder to the list of shared folders.
• Under Users, give everyone Read & Write privileges.
• If sharing with Windows machines, press the Options button and make sure the “Share files and folders using SMB (Windows)” is enabled.

Samba Share

Interestingly, Mac OS X uses samba as well. Apple just does a good job of hiding it. To create a samba share in Mac OS X, past this at the bottom of /etc/smb.conf:

[DeadlineRepository]
path = /path/to/repository
writeable = Yes
guest ok = Yes
force create mode = 0777
force directory mode = 0777
unix extensions = No


## Uninstallation¶

The Repository installer creates an uninstaller in the folder that you installed the Repository to. To uninstall the Repository, simply run the uninstaller and confirm that you want to proceed with the uninstallation.

Note that if you installed the Database with the Repository installer, it will be uninstalled as well. If you chose to connect to a Database that you manually installed, the Database will be unaffected.

### Command Line or Silent Uninstallation¶

The Repository uninstaller can be run in command line mode or unattended mode on each operating system.

To run in command line mode, pass the “–mode text” command line option to the installer. For example, on Linux:

./uninstall --mode text


To run in silent mode, pass the “–mode unattended” command line option to the installer. For example, on Windows:

uninstall.exe --mode unattended


To get a list of all available command line options, pass the “–help” command line option to the installer. For example, on Mac OS X:

./uninstall --help