Logs

Overview

Deadline provides logging in a number of areas. You can stream, visualize, filter, analyze, copy, and export the logs via Deadline. Typically, these logs will help to diagnose the root cause of an error (exception) in Deadline. It’s recommended to always provide log reports where applicable to Thinkbox when contacting for support. The application logs are stored as flat *.log files, which makes it easy to zip up and send as a single package. Job, Slave, Event, Requeue log reports can easily be exported from Deadline and shared. When exporting multiple reports, Deadline will automatically create a single zip file. Under Configure Repository Options, various Application Data settings are available to control the deletion interval of application logs, maximum number of history entries per application and finally, if a high level of verbosity should be enabled for Slave, Pulse and Balancer (very useful for render job debugging).

It is recommended to NOT redirect/pipe any Deadline logs to a remote location as this will stop job, task and slave reports from being visible in the Deadline Monitor panels. Alternatively, our API can be used to regularly ‘take a copy’ of any of the log reports and push them to an external location outside of Deadline. Please see the section Access via API for more details.

Install/Uninstall Logs

Deadline’s Installers (Bitrock) generates a log file during DB/Repository install and DB/Repository uninstall as well as during Client install and Client uninstall.

Application Logs

All Deadline application logs for a particular machine can be found locally here or via the GUI of each application (see specific application):

  • Windows: C:\ProgramData\Thinkbox\Deadline[VERSION]\logs
  • Linux: /var/log/Thinkbox/Deadline[VERSION]
  • Mac OS X: /Library/Logs/Thinkbox/Deadline[VERSION]

where [VERSION] is the MAJOR version number of Deadline. When contacting Thinkbox Support, ensure you compress these log files as a single zip file.

Launcher

../_images/logs_launcher.png

Click on Launcher (right-click on Windows), select Explore Log Folder to open a file browser on your local machine. Sort folder by Date Modified. Launcher logs are named in the following format:

deadlinelauncher-[hostname]-[date]-[increment].log

A typical Launcher log output:

2015-08-10 12:28:56:  BEGIN - mbp.local\owenm
2015-08-10 12:28:56:  Deadline Launcher 7.2 [v7.2.0.13 R  (fcc207a56)]
2015-08-10 12:28:56:  Launcher Thread - Launcher thread initializing...
2015-08-10 12:28:56:  Launcher Thread - Launcher thread listening on port 17070
2015-08-10 12:28:58:  Auto Configuration: Picking configuration based on: mbp.local / 192.168.20.144
2015-08-10 12:28:58:  Auto Configuration: No auto configuration could be detected, using local configuration
2015-08-10 12:28:58:  Launcher Thread - Remote Administration is now enabled
2015-08-10 12:28:58:  Launcher Thread - Automatic Updates is now enabled

LauncherNew

../_images/logs_launcher.png

Click on Launcher (right-click on Windows), select Explore Log Folder to open a file browser on your local machine. Sort folder by Date Modified. LauncherNew logs are named in the following format:

deadlinelaunchernew-[hostname]-[date]-[increment].log

LauncherNew logs are part of the internal Deadline automatic upgrade/downgrade bootstrapping process and may be useful to debug any issue encountered during the upgrade/downgrade process.

2015-08-27 23:57:55:  BEGIN - mbp.local\owenm
2015-08-27 23:57:55:  Deadline Launcher 7.2 [v7.2.0.14 R  (dbc900863)]
2015-08-27 23:57:55:  deadlinelaunchernew is running
2015-08-27 23:57:55:  deadlinelauncher has exited, proceeding with self upgrade
2015-08-27 23:58:00:  Reading in bootstrap ini settings...
2015-08-27 23:58:00:  Bootstrap file: /private/var/folders/7y/471pt3h95cz1c6557kgzqkj80000gn/T/LauncherBootstrap/bootstrap.ini
2015-08-27 23:58:00:  Copying /Applications/Thinkbox/DeadlineRepository7/bin/Mac/bin.zip to /private/var/folders/7y/471pt3h95cz1c6557kgzqkj80000gn/T/LauncherBootstrap/bin.zip
2015-08-27 23:58:00:  Copying /Applications/Thinkbox/DeadlineRepository7/bin/Mac/Version to /private/var/folders/7y/471pt3h95cz1c6557kgzqkj80000gn/T/LauncherBootstrap/Version
2015-08-27 23:58:00:  Decompressing /private/var/folders/7y/471pt3h95cz1c6557kgzqkj80000gn/T/LauncherBootstrap/bin.zip to /Applications/Thinkbox/Deadline7/Resources
2015-08-27 23:58:04:  Copying /private/var/folders/7y/471pt3h95cz1c6557kgzqkj80000gn/T/LauncherBootstrap/Version to /Applications/Thinkbox/Deadline7/Resources/Version
2015-08-27 23:58:04:     Launch Slave: False
2015-08-27 23:58:04:     Launch Pulse: False
2015-08-27 23:58:04:     Launch Monitor: True
2015-08-27 23:58:04:     Launch Balancer: False
2015-08-27 23:58:04:     Launch Web Service: False
2015-08-27 23:58:04:  Launching launcher
2015-08-27 23:58:04:  Self upgrade completed
2015-08-27 23:58:04:  Shutting down

Monitor

../_images/logs_monitor.png

In Monitor, select the Help drop-down. Select Explore Log Folder. Sort folder by Date Modified. Monitor logs are named in the following format:

deadlinemonitor-[hostname]-[date]-[increment].log

A typical Monitor log output (which can be viewed via the Console Panel in Monitor):

2015-09-08 11:59:14:  BEGIN - mbp.local\owenm
2015-09-08 11:59:14:  Deadline Monitor 7.2 [v7.2.0.16 R  (51587765f)]
2015-09-08 11:59:14:  Time to initialize: 129.000 ms
2015-09-08 11:59:21:  Auto Configuration: No auto configuration for Repository Path could be detected, using local configuration
2015-09-08 11:59:22:  Auto Configuration: Picking configuration based on: mbp / 192.168.43.244
2015-09-08 11:59:22:  Auto Configuration: No auto configuration could be detected, using local configuration
2015-09-08 11:59:22:  Time to connect to Repository: 610.000 ms
2015-09-08 11:59:22:  Time to check user account: 5.000 ms
2015-09-08 11:59:22:  Time to purge old logs and temp files: 2.000 ms
2015-09-08 11:59:22:  Time to synchronize plugin icons: 74.000 ms
2015-09-08 11:59:23:  Time to initialize main window: 942.000 ms
2015-09-08 11:59:23:  Main Window shown
2015-09-08 11:59:23:  Time to show main window: 150.000 ms
2015-09-08 11:59:43:  Listener Thread - OnConnect: Listener Socket has been closed.

Slave

../_images/logs_slave.png

In Slave, select the Help drop-down. Select Explore Log Folder. Sort folder by Date Modified. Slave logs are named in the following format:

deadlineslave-[hostname]-[date]-[increment].log

A typical Slave log output:

2015-08-16 19:10:23:  BEGIN - mbp.local\owenm
2015-08-16 19:10:23:  Deadline Slave 7.2 [v7.2.0.16 R  (51587765f)]
2015-08-16 19:10:27:  Auto Configuration: No auto configuration for Repository Path could be detected, using local configuration
2015-08-16 19:10:27:  Info Thread - Created.
2015-08-16 19:10:29:  Auto Configuration: Picking configuration based on: mbp.local / 192.168.0.2
2015-08-16 19:10:29:  Auto Configuration: No auto configuration could be detected, using local configuration
2015-08-16 19:10:29:  Scheduler Thread - Slave initialization complete.

Slave logs can also be streamed locally and remotely. Locally, in Slave at the bottom of the window, select Open Slave Log. This will show you dynamically what is being written into the Slave Log. Right-click to Suspend or Resume the log streaming. In Slave, if you are running concurrent tasks you can filter by Render Thread. Each concurrent task is a thread, 0 to 15. Remotely, in Monitor you can connect to a Slave by double-clicking on it in the Slave panel, or by right-clicking on it and selecting Connect To Slave Log.

Pulse

../_images/logs_pulse.png

In Pulse select the Help drop-down. Select Explore Log Folder. Sort folder by Date Modified. Pulse logs are named in the following format:

deadlinepulse-[hostname]-[date]-[increment].log

A typical Pulse log output:

2015-09-08 12:54:05:  BEGIN - mbp.local\owenm
2015-09-08 12:54:05:  Deadline Pulse 7.2 [v7.2.0.16 R  (51587765f)]
2015-09-08 12:54:07:  Auto Configuration: Picking configuration based on: mbp / 192.168.43.244
2015-09-08 12:54:07:  Auto Configuration: No auto configuration could be detected, using local configuration
2015-09-08 12:54:07:  Server Thread - Auto Configuration: Listening for auto configuration requests on :::17071:17071
2015-09-08 12:54:07:  Web Service - Web Service listening for connections on port 8080...
...
2015-09-08 12:54:18:  Web Service - Web Service shutting down...
2015-09-08 12:54:18:  Server Thread - OnConnect: Listener Socket has been closed.
2015-09-08 12:54:18:  Listener Thread - OnConnect: Listener Socket has been closed.

Pulse logs can also be streamed locally and remotely. Locally, in Pulse at the bottom of the window, select Open Pulse Log. This will show you dynamically what is being written into the Pulse Log. Right-click to Suspend or Resume the log streaming. Remotely, in Monitor you can connect to a Pulse by double-clicking on it in the Pulses panel, or by right-clicking on it and selecting Connect To Pulse Log.

Balancer

../_images/logs_balancer.png

In Balancer select the Help drop-down. Select Explore Log Folder. Sort folder by Date Modified. Balancer logs are named in the following format:

deadlinebalancer-[hostname]-[date]-[increment].log

A typical Balancer log output:

2015-09-08 12:59:26:  BEGIN - mbp.local\owenm
2015-09-08 12:59:26:  Deadline Balancer 7.2 [v7.2.0.16 R  (51587765f)]
2015-09-08 12:59:28:  Auto Configuration: Picking configuration based on: mbp / 192.168.43.244
2015-09-08 12:59:28:  Auto Configuration: No auto configuration could be detected, using local configuration
2015-09-08 12:59:28:  Initializing Balancer engine.
2015-09-08 12:59:28:  Initializing Balancer Info thread.
2015-09-08 12:59:28:  Initializing Balancer worker thread.
2015-09-08 12:59:28:  Worker Thread (8): House Keeping...
2015-09-08 12:59:28:  HouseKeeper (8): Beginning house keeping...
2015-09-08 12:59:38:  HouseKeeper (8):   House keeping completed.
2015-09-08 12:59:38:  Worker Thread (8): Updating the State...
...
2015-09-08 12:59:38:  Worker Thread (8): Executing the Algorithm: DefaultAlgorithm...
2015-09-08 12:59:38:  Algorithm (8): There are no targets for any Cloud Regions. Exiting the Balancer Algorithm.
2015-09-08 12:59:38:  Worker Thread (8): Equalizing targets...
2015-09-08 12:59:38:  Worker Thread (8): Waiting 60 seconds before next cycle.

Balancer logs can also be streamed locally and remotely. Locally, in Balancer at the bottom of the window, select Open Balancer Log. This will show you dynamically what is being written into the Balancer Log. Right-click to Suspend or Resume the log streaming. Remotely, in Monitor you can connect to a Balancer by double-clicking on it in the Balancers panel, or by right-clicking on it and selecting Connect To Balancer Log.

WebService

If using the standalone WebService, the WebService logs will be stored in the same path location as all other Deadline application logs and named in the following format:

deadlinewebservice-[hostname]-[date]-[increment].log

A typical WebService log output:

2015-09-08 12:47:18:  BEGIN - mbp.local\owenm
2015-09-08 12:47:18:  Deadline Web Service 7.2 [v7.2.0.16 R  (51587765f)]
2015-09-08 12:47:18:  Web Service - Web Service listening for connections on port 8080...
2015-09-08 12:47:30:  Web Service - Received request from: 192.168.43.244
2015-09-08 12:47:30:  Web Service - Request: GET http://mbp.local:8080/
2015-09-08 12:47:30:  Web Service - Web Service listening for requests...
2015-09-08 12:47:30:  Web Service - Received request from: 192.168.43.244
2015-09-08 12:47:30:  Web Service - Request: GET http://mbp.local:8080/favicon.ico
2015-09-08 12:47:30:  Web Service - Web Service listening for requests...
2015-09-08 12:48:04:  Web Service - Received request from: 192.168.43.244
2015-09-08 12:48:04:  Web Service - Request: GET http://mbp.local:8080/api/jobs
2015-09-08 12:48:04:  Web Service - Found API command: jobs
2015-09-08 12:48:05:  Web Service - Web Service listening for requests...

Job Reports and History

../_images/logs_job_reports.png

All reports for a job can be viewed in the Job Reports panel. This panel can be opened from the View menu or from the main toolbar in the Monitor. It can also be opened from the Job and Task panel’s right-click menu. You can use the Job Report panel’s right-click menu to save reports as files to send to Deadline Support. In addition to viewing job reports, you can also view the job’s history. The History window can be brought up from the Job panel’s right-click menu by selecting the Job History option.

../_images/job_history.png

Slave Reports and History

../_images/logs_slave_reports.png

All error reports for a Slave can be viewed in the Slave Reports panel. This panel can be opened from the View menu or from the main toolbar in the Monitor. It can also be opened from the Slave panel’s right-click menu. You can use the Slave Report panel’s right-click menu to save reports as files to send to Deadline Support. In addition to viewing Slave reports, you can also view the Slave’s history. The History window can be brought up from the Slave panel’s right-click menu by selecting the View Slave History option.

../_images/slave_history.png

Routine Processes

The Deadline queue has 3 processes that take place at regular intervals which can be configured via Repository Options.

Pending Job Scan

The Pending Job Scan will automatically log it’s output to the Deadline application log which executed the scan. This would be either the Slave, Pulse or Monitor (manually executed) logs. If the option: Write Pending Job Scan Output to Separate Log File is enabled in the Repository Options, then as a separate process, the Pending Job Scan logs would be a separate file, saved in the Deadline application logs directory and named in the following format:

pendingjobscan-[hostname]-[date].log

A typical Pending Job Scan log output:

2015-09-08 13:16:30:  BEGIN - mbp.local\owenm
2015-09-08 13:16:30:  Deadline Command 7.2 [v7.2.0.16 R  (51587765f)]
2015-09-08 13:16:30:  Purging old Pending Job Scan logs.
2015-09-08 13:16:30:  Performing pending job scan
2015-09-08 13:16:30:      Pending Job Scan - Loading pending and active jobs
2015-09-08 13:16:30:      Pending Job Scan - Loaded 7 pending and active jobs in 55.030 ms
2015-09-08 13:16:30:      Pending Job Scan - Scanning pending and active jobs
...
2015-09-08 13:16:30:      Pending Job Scan - Released 0 pending jobs and 0 pending tasks in 44.756 ms
2015-09-08 13:16:30:      Pending Job Scan - Done.
2015-09-08 13:16:30:  Processing Pending Job Events
2015-09-08 13:16:30:      Pending Job Events - Checking for pending job events
2015-09-08 13:16:30:      Pending Job Events - Processing 0 job events
2015-09-08 13:16:30:      Pending Job Events - No more job events to process
2015-09-08 13:16:30:      Pending Job Events - Done.

House Cleaning

The House Cleaning will automatically log it’s output to the Deadline application log which executed the scan. This would be either the Slave, Pulse or Monitor (manually executed) logs. If the option: Write House Cleaning Output to Separate Log File is enabled in the Repository Options, then as a separate process, the House Cleaning logs would be a separate file, saved in the Deadline application logs directory and named in the following format:

housecleaning-[hostname]-[date].log

A typical House Cleaning log output:

2015-09-08 13:16:35:  BEGIN - mbp.local\owenm
2015-09-08 13:16:35:  Deadline Command 7.2 [v7.2.0.16 R  (51587765f)]
2015-09-08 13:16:35:  Purging old House Cleaning logs.
2015-09-08 13:16:35:  Performing house cleaning
2015-09-08 13:16:35:  Performing Job Cleanup Scan...
2015-09-08 13:16:35:  Purging Unsubmitted Jobs
2015-09-08 13:16:35:  Purging Deleted Jobs
2015-09-08 13:16:35:  Purging Old Job Auxiliary Files
2015-09-08 13:16:35:  Purging Old Job Reports
2015-09-08 13:16:35:  Purging Obsolete Slaves
2015-09-08 13:16:35:  Purging Old Slave Reports
2015-09-08 13:16:35:  Purging Old Limits
2015-09-08 13:16:36:  Purging Temporary Repository Files
2015-09-08 13:16:36:  Purging Old Statistics
2015-09-08 13:16:36:  Purging Deleted Document Stubs From Database
2015-09-08 13:16:36:  Triggering House Cleaning Events

Repository Repair

The Repository Repair will automatically log it’s output to the Deadline application log which executed the scan. This would be either the Slave, Pulse or Monitor (manually executed) logs. If the option: Write Repository Repair Output to Separate Log File is enabled in the Repository Options, then as a separate process, the Repository Repair logs would be a separate file, saved in the Deadline application logs directory and named in the following format:

repositoryrepair-[hostname]-[date].log

A typical Repository Repair log output:

2015-09-08 13:16:39:  BEGIN - mbp.local\owenm
2015-09-08 13:16:39:  Deadline Command 7.2 [v7.2.0.16 R  (51587765f)]
2015-09-08 13:16:39:  Purging old Repository Repair logs.
2015-09-08 13:16:39:  Performing repository repair
2015-09-08 13:16:39:  Performing Orphaned Task Scan...
2015-09-08 13:16:39:  Performing Orphaned Limit Stub Scan...
2015-09-08 13:16:39:  Checking Available Database Connections
2015-09-08 13:16:39:  Performing Stalled Slave Scan...
2015-09-08 13:16:39:  Performing Stalled Pulse Scan...
2015-09-08 13:16:39:  Performing Stalled Balancer Scan...
2015-09-08 13:16:39:  Triggering Repository Repair Events

Remote Control Access

Monitor can remotely acccess the logs of it’s applications via the Remote Control features. If the Slaves, Pulses, or Balancers panel are not visible, see the Panel Features documentation for instructions on how to create new panels in the Monitor.

Monitor can Remotely Control Slaves, Pulses, and Balancers and the Remote Command panel shows all pending and completed remote commands that were sent from the Monitor. When sending a remote command, if this panel is not already displayed, it will be displayed automatically (assuming you have permissions to see the Remote Command panel).

Submission Logs

Submission to Deadline is logged to StdOut and StdErr via DeadlineCommand. If you are using the RESTful API or Python Standalone API (wrapper around RESTful API), then where applicable; text, JSON formatted objects and in the case of errors, HTTP status codes are returned.

Many of our Integrated Submission Scripts log any StdOut/StdErr to the current application process. Depending on the application being used, open the built-in script listener/editor to view any messages. In the case of our Submit Max To Deadline (SMTD) and Setup VRay DBR With Deadline interfaces, we log all steps to the window at the bottom of the interface and also to log files with the following format:

C:\Users\%username%\AppData\Local\Thinkbox\Deadline[VERSION]\logs\SubmitMaxToDeadline-[hostname]-[date]-[increment].log
C:\Users\%username%\AppData\Local\Thinkbox\Deadline[VERSION]\logs\SetupVRayWithDeadline-[hostname]-[date]-[increment].log

where [VERSION] is the MAJOR version number of Deadline.

In our Monitor based submitters, any StdOut/StdErr is logged to the Monitor log (Console Panel) or Launcher log, depending on what application executed the “Submit” script.

History Logs

As well as the Job and Slave history entries via Monitor, Deadline also stores the history of activity for each Pulse and Balancer instance as well as all Power Management history and overall Repository history.

Pulse

You can view a Pulse’s history by right-clicking on it in the Pulse panel and selecting the View Pulse History option.

../_images/pulse_history.png

Balancer

You can view a Balancer’s history by right-clicking on it in the Balancer panel and selecting the View Balancer History option.

../_images/balancer_history.png

Repository

You can view the Repository history by logging into Monitor as Super User Mode via the Tools menu entry and selecting View Repository History….

../_images/repo_history.png

Power Management

You can view the Power Management history by logging into Monitor as Super User Mode via the Tools menu entry and selecting View Power Management History….

../_images/pm_history.png

API Access

Deadline’s Scripting API, RESTful API and Standalone Python API all have access to Job, Task and Slave Reports. Additionally, there are functions to add or query Job, Slave and Repository history entries. Finally, Event Plugins such as OnHouseCleaningCallback can be used to regularly query and inject log/report based Deadline information to a remote location such as a dedicated logging server or perhaps an ELK stack for further downstream analysis and visualization.

License Server Logs

Thinkbox’s Flexlm license server creates a log file typically in the same directory as the executing binary files, which can help to diagnose licensing issues. Please consult the License Server documentation for further information.

Mongo Database Logs

MongoDB’s log files can be helpful if there are issues with the MongoDB service/daemon not starting, or frequently crashing on you. These issues are most often presented in the form of frequent database connection errors appearing on machines when you try to use the client applications such as Monitor, Slave, Pulse or Balancer. Whenever MongoDB is restarted, a new log file will be created. In the case of sending these log files to Thinkbox Support, ensure you grab all the logs from a particular day so that any reoccurring issue can be identified.

All MongoDB logs will be found in their default location on each of the three supported platforms, though if the install was to a different location, you would look there. These defaults are:

  • Windows: C:\DeadlineDatabase[VERSION]\mongo\data\logs
  • Mac OS X: /Applications/Thinkbox/DeadlineDatabase[VERSION]/mongo/data/logs
  • Linux: /opt/Thinkbox/Thinkbox/DeadlineDatabase[VERSION]/mongo/logs

where [VERSION] is the MAJOR version number of Deadline. When contacting Thinkbox Support, ensure you compress these log files as a single zip file.