Monitor

Overview

The Monitor application offers detailed information and control options for each job and Slave in your farm. It provides normal users a means of monitoring and controlling their jobs, and it gives administrators options for configuring and controlling the entire render farm.

../_images/monitor_preview.png

If you’re launching the Monitor for the first time on your machine, you will be prompted with a Login dialog. Simply choose your user name or create a new one before continuing. Once the Monitor is running, you’ll see your user name in the bottom right corner. If this is the wrong user, you can log in as another user by selecting File -> Change User. Note that if your administrator set up Deadline to lock the user to the system’s login account, you will have to log off of your system and log back in as the correct user.

../_images/monitor_login.png

Running the Monitor

To start the Monitor:

  • On Windows, you can start the Monitor from the Start Menu under Thinkbox\Deadline, or from the Launcher’s right-click menu.
  • On Linux, you can start the Monitor from a terminal window by running the deadlinemonitor script in the bin folder, or from the Launcher’s right-click menu.
  • On Mac OS X, you can start the Monitor from Finder by running the DeadlineMonitor application in Applications/Thinkbox/Deadline, or from the Launcher’s right-click menu.

The Monitor can also be started from a command prompt or terminal window. For more information, see the Monitor Command Line documentation.

Panel Features

Information in the Monitor is broken up into different panels, which are described further down. These panels have many features in common, which are explained here.

Customization

Monitor panels can be created from the View menu, or from the main toolbar. They can be re-sized, docked, or floated as desired. This allows for a highly customized viewing experience which is adaptable to the needs of different users.

../_images/spawn_panel.png

The current layout can be pinned to the Pinned Layouts menu so that it can be restored at a later time. This can be done from the View menu, or from the main toolbar. The current layout can also be saved to a file from the View menu, and then loaded from that file later.

../_images/monitor_layouts.png

When you pin a layout you can chose to save the location and size of the Monitor by checking the “Save Location and Size” box when pinning the layout.

../_images/monitor_new_layout.png

To prevent accidental modifications to the current layout, you can lock the layout from the View menu, by pressing “Alt-`”, or from the main toolbar. When locked, panels cannot be moved, but they can still be docked and undocked. To dock a floating panel while the layout is locked, simply double-click on the panels title. It will be docked to the same location it was originally undocked from.

../_images/monitor_lock.png

The columns in Monitor panels are customizable. The columns can be resized by simply clicking on the separator column line and moving it and can be reordered by clicking on a column and moving it. Right clicking on the column headers in a panel allows you to toggle the visibility of each column.

../_images/monitor_job_columns_click.png

In this menu you can modify the visibility and ordering of the columns by clicking the “Customize..” menu item. Moving columns to the left side list hides them, and the order that columns are listed in the right list corresponds to the order they will appear in the panel (top->bottom corresponds to left->right). You move the columns around by clicking the arrow buttons.

../_images/monitor_customize_columns.png

Once you have configured your column layout you can pin it.

pinlayout layoutname

You can also set the current list layout as the list layout to load by default, when opening new panels of the same type, by clicking “Save Current List Layout As Default”. If you want to restore the original list layout default click the “Reset Default List Layout”.

Data Filtering

Almost every panel has a search box that you can use to filter the information you’re interested in. You can simply type in the word(s) you are looking for, or use regular expressions for more advanced searching.

../_images/job_search.png

In addition, every panel that has a search box also supports a more advanced filtering system. To add a filter to a panel, select the Edit Filter option in the panel’s drop down menu, which can be found in the upper-right corner of the panel. A window will appear allowing you to specify the name the filter being created. You can select to match all of the filters added or any of the filters added. If all must match, only records where all data matches each filter will be shown, while if any can match, if a record contains one or more matches it will be shown.

job_edit_filter_menu job_edit_filter

Clicking the add filter button generates a new filter. The filter requires a column to be selected, an operation to perform, and a value to use in the operation. Filters can also be removed by clicking the minus button to the right of each filter. After all filters are are entered, press OK to apply the filter to the current panel.

A filter can be cloned and opened in a new tab within the panel through the Clone Filter option in the panel drop down menu. The Clear Filter option can be used to clear all filters from the current panel.

Finally, you can pin the current filters so that they can be restored at a later time using the Pinned Filters sub menu in the panel drop down menu. Note that the Pin Current Filter option is only available if a filter is currently being applied. If there are no filters, the Pin Current Filter option will be hidden.

Automatic Sorting and Filtering

Almost every panel has an option to do automatic sorting and filtering when data changes in the panel. When this option is disabled, sorting and filters must manually be re-applied to ensure that the data is sorted and filtered properly.

Note that automatic sorting and filtering can affect the Monitor’s performance if there are lots of jobs (10,000+) or lots of Slaves (1000+) in the farm. To improve Monitor performance in this case, it is recommended to disable automatic sorting and filtering. There is an option in the Monitor Settings in the Repository Configuration to automatically disable it by default.

Saving and Loading Panel Layouts

Every list-based panel (Jobs, Slaves, Tasks, etc) has an option to save and load the list layout, which you can find in the panel’s drop down menu. This allows you to save out a list’s filters, column order and visibility, etc, and load them again later or share them with another user.

Note that when loading a list layout, you must choose a layout that was saved from the same type of list. For example, you cannot save a layout from the Job list and then load it into the Slave list.

Graph Views

Almost every panel supports showing a graphical representation of the data. The graph can be shown by selecting the Graph View option in the panel’s drop down menu, which can be found in the upper-left corner of the panel. The graph view can be saved as an image file by right-clicking anywhere in view and selecting Save Graph As Image.

../_images/save_task_graph.png ../_images/monitor_graph_example.png

If the graph is a line graph, the following operations are available:

  • Zoom In: Use the mouse wheel or the UP arrow key to zoom in. You can also click and hold the left mouse button and drag to select a sub-area of the graph to zoom in.
  • Zoom Out: Use the mouse wheel or the DOWN arrow key to zoom out.
  • Reset Zoom: Use the right-click menu to reset the zoom level.
  • Pan: Use the middle mouse button or the LEFT and RIGHT arrow keys to pan the graph.
  • Show/Hide Series: If the line graph has a legend, you can use the right-click menu to customize which series are shown or hidden.
../_images/monitor_graph_zoom.png

If the graph is a pie chart, you can filter the data from the graph view by holding down the SHIFT key and clicking on one of the pie slices. The data will be filtered to only show records that are represented by the pie slice that was clicked on.

../_images/graph_shift_filter.png

Scripts

Almost every panel has the option to run custom scripts from the panel’s right-click menu. Many scripts are already shipped with Deadline, and additional custom scripts can be written. See the Monitor Scripts documentation for more information.

These script menus can also be customized from the Repository Options.

Information Panels

As mentioned earlier, information in the Monitor is broken up into different panels. These panels can be created from the View menu, or from the main toolbar. They can be re-sized, docked, or floated as desired. This allows for a highly customized viewing experience which is adaptable to the needs of different users.

Jobs

The Jobs panel contains a list that shows all jobs in the farm. It also displays useful information about each job such as it’s name, user, status, error count, plugin, etc. As jobs change states, their colors will change. Active jobs will appear as green, and will remain green as they continue to render without errors. But if it starts to accumulate errors, it will turn brown and then eventually red. This allows you to see at a glance which jobs are having problems. For more information on job monitoring, see the Monitoring Jobs documentation.

../_images/monitor_job_panel.png

The Jobs panel supports standard filtering, but it also has a Quick Filter option in the panel’s drop down menu to make it easier to filter out unwanted jobs. By toggling the options within the Status, User, Pool, Group, and Plugin sections, you can quickly drill down to the jobs you are interested in. There is also an Ego-Centric Sorting optino in the panel’s drop down menu which can be used to keep all of your jobs at the top of the job list.

../_images/filtering.png

The Jobs panel also supports the ability to group jobs together based on their Batch Name property. All of the job submitters that are included with Deadline will automatically set the Batch Name if they are submitting multiple jobs that are related to each other. The Batch Name for a job can be modified in the Job Properties. If you prefer to not have the jobs grouped together in the job list, you can disable the Group Jobs By Batch Name option in the Monitor and User Settings.

../_images/job_batch_example.png

Finally, the Jobs panel allows jobs to be controlled and modified using the right-click menu. You can also bring up the Job Properties window by double clicking on a job. See the Controlling Jobs documentation for more information.

Tasks

The Task panel shows all the tasks for the job that is currently selected. It displays useful information about each task such as its frame list, status, and if applicable, the Slave that is rendering it.

../_images/monitor_task_panel.png

The Task panel also allows you to control tasks from the right-click menu. See the Controlling Jobs documentation for more information. In addition, the double-click behavior in the Task panel can be set in the Monitor and User Settings, which can be accessed from the main toolbar.

Job Details

The Job Details panel shows all available information about the job that is currently selected. The information is split up into different sections that can be expanded or collapsed as desired. Job statistical information can be found by expanding the Statistics section as shown below.

details stats

Job Dependency View

This panel allows you to view and modify a job’s dependency tree in a node-based view. You can lock the view to the currently selected job, which allows you to drag & drop other jobs into the view to hook up new dependencies. In addition, you can drag & drop Python scripts or asset files directly into the view and hook them up as dependencies. See the Controlling Jobs documentation for more information.

../_images/monitor_job_dependency_panel.png

Job Report

All reports for a job can be viewed in the Job Reports panel. This includes error reports, logs, and task requeue reports. This panel can also be opened by right-clicking on a job in the Job List and selecting View Job Reports. More information can be found in the Controlling Jobs documentation.

../_images/monitor_job_report_panel.png

Slaves

The Slave panel shows all the Slaves that are in your farm. It shows system information about each Slave, as well as information about the job the Slave is currently rendering.

../_images/monitor_slave_panel.png

If you see a Slave that is colored orange in the list, this means that the Slave is unable to get a license or that the license is about to expire. When the Slave cannot get a license, it could be because there is a network issue, the license has expired, or the license limit has been reached.

If a Slave isn’t rendering a job that you think it should be, you can use the Job Render Candidates feature in the Jobs panel to try and figure out why.

The Slave panel’s right-click menu allows you to modify Slave settings and control the Slaves remotely. See the Slave Configuration documentation for more information.

Slave Reports

All log and error reports for a Slave can be viewed in the Slave Reports panel. This panel can also be opened by right-clicking on a Slave in the Slave List and selecting View Slave Reports.

../_images/monitor_slave_report_panel.png

Pulses

The Pulse panel shows which machine Pulse is running on, as well as previous machines that Pulse has run on. It also shows system information about each machine.

../_images/monitor_pulse_panel.png

Balancers

The Balancer panel shows which machines the Balancer is running on. It also shows system information about each machine.

../_images/monitor_balancer_panel.png

The Balancer panel’s right-click menu allows you to modify Balancer settings and control the Balancer remotely. See the Balancer Configuration documentation for more information.

Limits

The Limit panel shows all the Limits that are in your farm. You can access many options for the Limits by right-clicking on them. See the Limits and Machine Limits documentation for more information.

../_images/monitor_limit_panel1.png

Connection Servers

The Connection Servers panel shows which machine the Connection Server is running on, as well as previous machines that the Connection Server has run on. It also shows system information about each machine.

../_images/connection_servers_panel.png

License Forwarders

The License Forwarder panel shows which machine the License Forwarder is running on, as well as previous machines that the License Forwarder has run on. It also shows system information about each machine.

../_images/license_forwarder_panel.png

The License Forwarder panel’s right-click menu allows you to control the License Forwarder remotely.

Console

The Console panel shows all lines of text (Stdout/StdErr) that is written to the Monitor’s log. A right-click menu for copying and pasting the log is available as well as “Clear Log”, “Clear Selection” and a “Find” dialog. Note, “Clear Log” only clears the current view and the information still exists in the actual log files.

../_images/monitor_console_panel.png

Remote Commands

Note

By default, Remote Administration is disabled in Repository Configuration. Enabling Remote Administration allows the Deadline Clients to be controlled remotely from the Monitor running on another machine. Note that this can be a security risk if you are not behind a firewall.

The Remote Command panel shows all pending and completed remote commands that were sent from the Monitor. When sending a remote command, if this panel is not already displayed, it will be displayed automatically (assuming you have permissions to see the Remote Command panel). See the Remote Control documentation for more information.

../_images/monitor_remote_panel.png

Cloud

The Cloud panel shows all the instances from the cloud providers that the Monitor is connected to. This panel allows you to control and close your existing instances. See the Cloud Controls documentation for more information.

../_images/monitor_cloud_panel.png

Background Operations

Most operations that can be called from the Monitor now execute in the background. This allows the Monitor’s User Interface to continue to be used while the operations execute. The Background Operations panel shows operations that are in the process of executing. You can pause and resume operations and also cancel them.

../_images/monitor_background_panel.png

Monitor Menu Options

The available options are listed below. They are available in the Monitor’s main menu, and some are also available in the main toolbar. Note that the availability of these options can vary depending on the context in which they are used, as well as the User Group Permissions that are defined for the current user.

File Menu

Launch Monitor For Repository…

Launch another Monitor for a repository. There is also a toolbar button for this option.

Change Repository…

Connect to a different repository, or reconnect to the current repository if the Monitor becomes disconnected. There is also a toolbar button for this option.

Change User…

Change the current user. You have the choice to select a different user or create a new one. There is also a toolbar button for this option.

Import Archived Jobs…

Opens a file dialog which allows you to select a zip file containing an archived job which you would like to add back to the monitor. See the Archiving Jobs documentation for more information.

View Menu

Manual Refresh

Forces an immediate refresh of all the data in the Monitor. Manual refreshing is disabled by default, and can only be enabled in the Monitor Settings in the Repository Configuration.

New Panel

Spawn a new information panel. See the Information Panels section above for more information. There is also a toolbar button for this option.

Lock Panels

Prevents the panels from being moved. Panels can still be floated, docked, and closed. To dock a floating panel, double-click on the panel’s title. There is also a toolbar button for this option.

Main Toolbar

Whether or not to display the main icon toolbar across the top of the Monitor. Useful to hide this toolbar if Monitor is being displayed on a large TV screen with pie-charts visible only.

Pinned Layouts

You are able to save different Monitor layout for quick use. By selecting Pin Current Layout, your current layout will be added to your pinned layouts. Selecting a pinned layout will restore the monitors panels to the pinned layouts state. There is also a toolbar button for this option.

Open Layout…

Load a previously saved layout from file.

Save Layout…

Saves the current layout to file.

Save All Pinned Layouts…

Save all the pinned Monitor layouts to a zip file.

Reset Layout

Reset the current layout to the Monitor’s default layout.

Submit and Script Menus

Submission scripts can be found under the Submit menu, and general scripts can be found under the Scripts menu. Many scripts are already shipped with Deadline, and additional custom scripts can be written. Check out the Monitor Scripts documentation for more information.

Tools Menu

Super User

Enter Super User Mode, which allows you to access the administrative Monitor options. Super User mode can be password protected simply by specifying a password in the Access Control section of the Repository Configuration.

View Repository History…

View all repository history entries generated on the farm.

View Power Management History…

View all power management history entries on the farm. See the Power Management documentation for more information.

View Farm Reports…

View various repository statistical information. See the Farm Statistics documentation for more information.

Manage Pools…

Add or remove Pools, and configure which Pools are assigned to the Slaves. See the Pools and Groups documentation for more information.

Manage Groups…

Add or remove Groups, and configure which Groups are assigned to the Slaves. See the Pools and Groups documentation for more information.

Manage Users…

Add or remove users, and set user information. See the User Management documentation for more information.

Manage User Groups…

Add or remove a user group, and set user group permissions to control which features are accessible. See the User Management documentation for more information

Configure Repository Options…

Configure a wide range of global options. See the Repository Configuration documentation for more information.

Configure Slave Scheduling…

Configure the Slave scheduling options. See the Slave Scheduling documentation for more information.

Configure Power Management…

Configure the Power Management settings. See the Power Management documentation for more information.

Configure Cloud Providers…

Set up and enable cloud service providers. See the Cloud Controls documentation for more information.

Configure Asset Server…

Configure the AWS Portal Asset Server settings. See the Asset Server Options documentation for more information.

Configure Plugins…

Configure the available render plugins, such as 3ds Max, After Effects, Maya, and Nuke. See the plugin documentation for more information on the configurable settings for each plugin. Each plugin also has various generic options that can be configured globally.

Configure Events…

Configure the available event plugins such as Draft, Shotgun, FTrack and NIM. See the event plugin documentation for more information on the configurable settings for each plugin.

Perform Pending Job Scan

Pending Job Scan determines if any pending jobs should be released, or if any pending job events should be processed. If Pulse is running, it will perform the Pending Job Scan every interval. If Pulse is not running, the Slaves will perform the Pending Job Scan between tasks if the interval has passed. This operation is normally performed automatically, but you can force an immediate clean up with this option if desired.

Note

This feature is not supported when the Monitor is connected to the Remote Connection Server. See the House Cleaning docs for more information.

Perform House Cleaning

House Cleaning is responsible for purging old jobs, Slaves, and reports, and for regular maintenance of the repository and database. If Pulse is running, it will perform House Cleaning every interval. If Pulse is not running, the Slaves will perform House Cleaning between tasks if the interval has passed. This operation is normally performed automatically, but you can force an immediate clean up with this option if desired.

Note

This feature is not supported when the Monitor is connected to the Remote Connection Server. See the House Cleaning docs for more information.

Perform Repository Repair

Repository Repair is responsible for finding orphaned tasks and limit stubs, and checking for stalled Slaves. If Pulse is running, it will perform a Repository Repair every interval. If Pulse is not running, the Slaves will perform the Repository Repair between tasks if the interval has passed. This operation is normally performed automatically, but you can force an immediate clean up with this option if desired.

Note

This feature is not supported when the Monitor is connected to the Remote Connection Server. See the House Cleaning docs for more information.

Undelete Jobs

Use this to recover any deleted jobs that haven’t been purged from the database yet. You can right-click on the jobs in the Undelete Jobs list to view their history, including how the job was deleted.

Explore Repository Root

View the root directory of the current Repository.

Import Settings…

Import settings from another Repository. See the Importing Repository Settings documentation for more information.

Download Integrated Submission Scripts…

View available integrated submission scripts and download the client files for them. Useful if connecting to a Repository via a Connection Server.

Synchronize Monitor Scripts and Plugins

Rebuilds the script-specific menus, updates event plugin scripts and the local icon cache to the icon configurations in the Configure Plugins menu. Note that if any new icons are changed in the Configure Plugins menu, you will have to re-sync these plugins before the jobs in list show the new icons. This only synchronizes the scripts and plugins for the Monitor.

Local Slave Controls…

Opens the Local Slave Controls window, which allows you to control and configure the Slave that runs on your machine.

Options

Modify the Monitor and User Settings. There is also a toolbar button for this option.

Command Line Options

To run the Monitor from a command prompt or terminal window, navigate to the Deadline bin folder (Windows or Linux) or the Resources folder (Mac OS X) and run the ‘deadlinemonitor’ application. To view all available command line arguments, you can run the following:

deadlinemonitor -help

Available Options

To start a new Monitor if there already another Monitor running, use the -new option:

deadlinemonitor -new

To start the Monitor connected to a different repository, use the -repository option. You can combine this with the -new option to have different Monitors connected to different repositories:

deadlinemonitor -repository "\\repository\path"
deadlinemonitor -new -repository "\\repository\path"

To start the Monitor without the splash screen, use the -nosplash option:

deadlinemonitor -nosplash

To shutdown the Monitor if it’s already running, use the -shutdown option:

deadlinemonitor -shutdown

You can also set all of the Monitor Options using command line options. For example:

deadlinemonitor -draganddropdep True -groupjobbatches False

FAQ

I’m unable to move panels in the Monitor, or dock floating panels.

You need to unlock the Monitor layout. This can be done from the View menu or from the toolbar.

Can I dock a floating panel when the Monitor layout is locked?

Yes, you can dock the floating panel by double-clicking on its title bar. It will be docked to its previous location, or to the bottom of the Monitor if it wasn’t docked previously.

What does it mean when a Slave is orange in the Slave list?

This means that the Slave is currently unable to get a license.

How is the “Estimated Remaining Render Time” column calculated in Monitor?

It’s calculated by finding the average render time of the tasks that have been completed and multiplying that value by the number of tasks that are active (rendering or queued):

average render time per frame * frames remaining