Database Maintenance

Note

Some of the topics covered in this page require you to have administrator/root access to your database machine. Please consult your local IT staff if in doubt.

Overview

Deadline uses MongoDB for the database. MongoDB is a highly performant, NoSQL based database system. No formal training or qualification is required to setup, run and maintain a MongoDB infrastructure for Deadline. It really is that simple! However, it’s always good to understand some fundamentals and recommended tools and procedures for any required maintenance such as in the case of an unscheduled power outage or disaster recovery/data recovery situation. MongoDB is installed automatically by the Repository Installer or via a Manual Database Installation. Additional DB configuration can be made via a simple config.conf file, such as enabling the RESTInterfaceEnabled seen below.

../_images/db_restful_interface.png

MongoDB Logs

MongoDB’s log files can be helpful if there are issues with the MongoDB service/daemon not starting, or frequently crashing on you. These issues are most often presented in the form of frequent database connection errors appearing on machines when you try to use the client applications such as Monitor, Worker or Pulse. Whenever MongoDB is restarted, a new log file will be created. In the case of sending these log files to Thinkbox Support, ensure you grab all the logs from a particular day so that any reoccurring issue can be identified.

All MongoDB logs will be found in their default location on each of the three supported platforms, though if the install was to a different location, you would look there. These defaults are:

  • Windows: C:\DeadlineDatabase[VERSION]\mongo\data\logs

  • macOS: /Applications/Thinkbox/DeadlineDatabase[VERSION]/mongo/data/logs

  • Linux: /opt/Thinkbox/DeadlineDatabase[VERSION]/mongo/data/logs

where [VERSION] is the MAJOR version number of Deadline. When contacting Thinkbox Support, ensure you compress these log files as a single zip file.

MongoDB GUI Tools

MongoDB does not include a GUI-style administrative interface. Instead most administration is carried out from command line tools such as the Mongo Shell. However some UI’s are available as separate community projects and are listed on the official MongoDB website. Some are focused on administration, while some focus on data viewing.

MongoDB Migration

The Deadline Repository and/or the MongoDB database can be moved to a new location or machine. Please see our Relocating instructions for further information.

Database Repair

On an odd occasion such as a power outage affecting your database server, you may need to carry out a small amount of maintenance on the database machine to recover and repair any problems which may have occurred. Luckily the procedure is very straight-forward and requires you as administrator/root to stop the DB service/daemon, execute the --repair command on the MongoDB and then simply, restart the DB service/daemon again. For each OS specific example below, you may wish to copy the commands into a local script file on the DB machine and ensure it has permissions to be executed correctly.

  • Windows: Create a local batch script file such as DBRepairWin.bat containing the following commands:

    sc stop Deadline[VERSION]DatabaseService
C:\DeadlineDatabase[VERSION]\mongo\application\bin\mongod --repair --logpath C:\DeadlineDatabase[VERSION]\mongo\data\logs\repairlog.txt --dbpath C:\DeadlineDatabase[VERSION]\mongo\data
sc start Deadline[VERSION]DatabaseService

  • Linux: Create a local bash script file such as DBRepairLinux.sh containing the following commands:

    sudo service deadline[VERSION]db stop
/opt/Thinkbox/DeadlineDatabase[VERSION]/mongo/application/bin/mongod --repair --config /opt/Thinkbox/DeadlineDatabase[VERSION]/mongo/data/config.conf
sudo service deadline[VERSION]db start

  • macOS: Create a local Mac Command script file such as DBRepairMac.command containing the following commands:

    echo “Stopping database”
sudo launchctl unload /Library/LaunchDaemons/org.mongodb.mongod_Deadline[VERSION]DatabaseService.plist

echo “Repairing database”
sudo /Applications/Thinkbox/DeadlineDatabase[VERSION]/mongo/application/bin/mongod --repair --logpath /Applications/Thinkbox/DeadlineDatabase[VERSION]/mongo/data/logs/log.txt --dbpath /Applications/Thinkbox/DeadlineDatabase[VERSION]/mongo/data

echo “Restarting database
sudo launchctl load /Library/LaunchDaemons/org.mongodb.mongod_Deadline[VERSION]DatabaseService.plist

echo “Done”

where [VERSION] is the MAJOR version number of Deadline, such as: 10.

MongoDB Shell

Included in the MongoDB installation via Manual Install or Deadline’s Repository Installer, is an interactive JavaScript shell interface to MongoDB, named: mongo in your bin directory which provides a powerful interface for systems administrators as well as a way for developers to test queries and operations directly with the database. The official MongoDB mongo document addresses the basic invocation of the mongo shell and an overview of its usage. Here is an example which Thinkbox Support might ask you to execute to gather all the current farm’s Limits and LimitStubs in Deadline.

  • Dump Limit Info on Linux Example: Create a bash script file names such as DumpLimitInfo.sh containing the following commands:

    #!/bin/sh

cd /opt/Thinkbox/DeadlineDatabase[VERSION]/mongo/application/bin

echo ===================================
echo Limit Stubs:
./mongoexport --port [DB_PORT] -d deadline[VERSION]db_LimitStubs -c LimitStubs | gzip -9 > /tmp/LimitStubs.json.gz
echo ===================================
echo Limits:
./mongoexport --port [DB_PORT] -d deadline[VERSION]db_Limits -c LimitGroups | gzip -9 > /tmp/LimitGroups.json.gz
echo ===================================

where [VERSION] is the MAJOR version number of Deadline, such as: 10 and [DB_PORT] is the MongoDB port, such as: 27100.

There are a number of other useful MongoDB tools included in a standard MongoDB distribution, which may prove useful such as mongoexport, mongoimport, mongodump and mongotop.

MongoDB Backup

The official MongoDB documentation provides the best information regarding various DB Backup and DB Restore options you may wish to consider deploying to protect your Mongo database.

MongoDB Clearing Metadata

If you are running Deadline 10.1.4.1 or earlier on AWS EC2 you can experience performance issues due to instance metadata getting left behind in the database by AWS Portal Link. The following commands can be used to clear that uneccessary metadata and give you a performance boost. It is recommended to do this after upgrading to version 10.1.5.0 or later, where the issue has been fixed.

Connect to the database.

Windows: C:\DeadlineDatabase10\mongo\application\bin\deadline_mongo
Linux: /opt/Thinkbox/DeadlineDatabase10/mongo/application/bin/deadline_mongo
Mac: /Applications/Thinkbox/DeadlineDatabase10/mongo/application/bin/deadline_mongo

Clear the metadata.

metaData=db.EventPluginSettingsCollection.findOne({"Name":"AWSPortal"}).Meta;
keys = Object.keys(metaData);
keys.forEach(function(key) { if (key.startsWith("i-", 0)) { delete metaData[key] }});
db.EventPluginSettingsCollection.findAndModify({
    query: {"Name":"AWSPortal"},
    update: { $set: {"Meta": metaData} } })