Path Mapping (Cross-Platform Rendering)¶
Overview¶
Many of the applications that Deadline supports are available for multiple operating systems. If you have a farm of mixed configurations, you will probably run into scenarios where the machines rendering your Jobs will need to access (or save) assets differently based on:
The OS at render time (Linux, Mac OS, or Windows)
The current environment at render time
Different properties within the Job being rendered (e.g., submitting user, group, custom value)
All of these needs can be addressed with Deadline’s Path Mapping feature. While there may be other considerations to take into account, depending on the application you are rendering with, the Path Mapping feature will do most of the work for you.
When using a mixed render farm, it is likely that asset paths will be different on each operating system. In many cases, Deadline is aware of the paths being passed to the rendering application, so you can configure Path Mapping rules to swap out these paths when appropriate based on the operating system.
Path Mapping Rule Types¶
There are three kinds of Path Mapping rules that can be configured in Deadline: Job-Specific, Global, and Submission-Time rules. In most cases, you will just need to use Global Rules.
Global rules will be applied to any known paths prior to rendering any job. Any paths that start with the value in Path To Replace will have that portion substituted with the replacement path for the current operating system. These rules can be created and updated from the Repository Options dialog under the ‘Mapped Paths’ section.
Job-Specific rules will be applied only to paths related to the job in which they are defined. Any such path that starts with the value Path To Replace will have that portion substituted with the replacement path for the current operating system. These rules can be created and updated after submission from the Job Properties dialog in the Monitor; there currently is no interface to do this manually at time of submission. If you wish to automatically create Job-Specific rules at submission time, please refer to the Submission-Time Rules section below.
Submission-Time rules are defined globally, but are tied to individual jobs as Job-Specific Rules at time of submission. Because they are stored in the job at submission time, changes to these rules in the Repository Settings will not retroactively alter the rules that were already copied to previously submitted jobs; only future submissions will be affected. These rules can be created and updated from the Repository Options dialog under the ‘Mapped Paths’ section.
All rule types are configured similarly, and only differ in the ways mentioned above. Note that every path mapping rule set must have a unique Path To Replace value per Region. For example, a Global rule set and a Submission-Time rule set cannot have the same Path To Replace value if they are in the same Region.
Path Mapping Setup¶
Regardless of the rule type, the process of creating and manipulating rules is identical. For Global and Submission-Time rules, the configuration is done from the Repository Options dialog, under the ‘Mapped Paths’ section:
For Job-Specific rules, the configuration is done from the Job Properties dialog instead:
To add a new Path Mapping rule, just click the Add button. Then, you specify the path that needs to be swapped out, along with the paths that will be swapped in based on the operating system. You can also specify a region so you can have different mappings for the same path across different regions.
For best results, make sure that all paths end with their appropriate path separator (/ or \). This helps avoid mangled paths that are a result of one path with a trailing separator, and one without.
Note that these swaps only work one-way, so if you are swapping from PC to Linux and vice-versa, you will need two separate entries. For example, let’s say the PC machines use the
path \\server\share\ for assets, while the Linux machines use the path /mnt/share/. Here are what your two entries should look like:
Entry 1 (replaces the Linux path with the PC path on PCs):
Replace Path: /mnt/share/ Windows Path: \\server\share\ Linux Path: Mac Path:
Entry 2 (replaces the PC path with the Linux path on Linux):
Replace Path: \\server\share\ Windows Path: Linux Path: /mnt/share/ Mac Path:
If you have Mac machines as well, you will need three entries. For example, if the Macs use /Volumes/share/ to access the assets from the previous example, here are what your three entries should look like:
Entry 1 (replaces the Linux path with the PC path on PCs and the Mac path on Macs):
Replace Path: /mnt/share/ Windows Path: \\server\share\ Linux Path: Mac Path: /Volumes/share/
Entry 2 (replaces the PC path with the Linux path on Linux and the Mac path on Macs):
Replace Path: \\server\share\ Windows Path: Linux Path: /mnt/share/ Mac Path: /Volumes/share/
Entry 3 (replaces the Mac path with the PC path on PCs and the Linux path on Linux):
Replace Path: /Volumes/share/ Windows Path: \\server\share\ Linux Path: /mnt/share/ Mac Path:
By default, Deadline just uses regular string replacement to swap out the paths. When doing so, Deadline takes care of the path separators (/ and \) automatically.
If you want more flexibility, you can configure your path mapping rules to use regular expressions, but note that you will need to handle the path separators manually using [/\\\\] in your regular expressions.
General Tips and Recommendations¶
Here are a number of useful tips and recommendations to keep in mind when setting up your Path Mapping ruleset:
Initially, create simple path mapping rules with Case Sensitive and Regular Expression both set to False and Region set to All. Once the mapped paths have proven to all be successful, then consider tightening up the configuration with these settings ONLY if necessary later.
Create the minimal number of path mapping rules to cover your needs. De-duplicate paths and leverage recursive rule application to obtain the minimal ruleset required for your use-case.
You do not have to enter replacement paths for all platforms; any empty replacement paths will be skipped when applying the rule. Note that the rule will not be applied at all if you do not enter at least one replacement paths.
Create 2-way or 3-way path mapping rules only if you need multi-directional path mapping. A 1-way rule is perfectly valid if that is all that is required for your setup.
Always ensure you have trailing path separators in each of your paths. This will help ensure you do not have partial matches on a folder of similar name.
Do not specify an identical path in the replacement field for the platform. E.g., if you are looking to replace the path
T:\, do not also enterT:\in the Windows Path field; simply leave it blank.On Windows, looking for the mapped drive:
Z:\will not matchZ:/.
Application-Specific Considerations¶
For some applications, like Maya and Nuke, simply configuring Path Mapping rules is enough to allow for Deadline to apply them at render time. For other applications however, like After Effects and Cinema 4D, additional setup is required. You can find more information on how to render with these applications on mixed farms can be found in their Cross-Platform Rendering Considerations sections in the Plugins documentation.
Advanced Setup¶
This section will cover advance configuration and use-cases of Path Mapping rules. While not needed in the general case, they can be quite useful to fine tune Path Mapping to help simplify your submission pipeline.
Regions¶
Regions can be used to setup different mappings for the same path across your farm. For example, let’s say we have a local farm and a remote farm, and we want to map the path /mnt/share/ in our remote farm but not in our local farm. All we have to do is set the region of our mapping to the same region our remote Workers are in. Workers in the region will replace /mnt/share/ but all the other Workers will use /mnt/share/ normally. We could also setup an alternate path for the Workers in our local farm.
A mapping in the All region will apply to every region, even unrecognized ones. It should be noted that a region’s mapping is applied before the All region, and if you are configured to the region none, then the path mapping with the regions configured to All will apply.
A mapping in the unrecognized region will only apply to the regions that are unrecognized.
Recursively Apply Path Mapping Rules¶
This is an option that is available in the Repository Options dialog’s ‘Mapped Paths’ section. When enabled, it will simulate recursive/repeated application of the current set of Path Mapping rules whenever they are applied.
In practice, this allows you to ‘chain’ multiple simpler rules together.
For example, if you had a rule mapping F:\ to \\fs-01, and another rule mapping U:\ to F:\users\, Deadline would automatically derive and apply the implicit U:\ -> \\fs-01\users\ rule with this option on.
Token Substitution¶
Token substitution allows for Replacement Paths to vary based on variables available in the current context, other than the OS and Region.
This enables use-cases where, for example, the current user or job ID need to be included as part of the replacement path. In order to use these tokens, you simply need to include them within the replacement paths of your Path Mapping rules, and the value of the token will be determined dynamically when the rule is applied.
Tokens are always of the following format: ${TOKEN_TYPE:TOKEN_NAME}
TOKEN_TYPE: The token type defines where the tokens will be pulling their values from. See Types of Tokens for more information.
TOKEN_NAME: The name of the token value we want to find and substitute.
Note
While tokens can be used in all Path Mapping rule types, they behave slightly differently for Submission-Time rules. Tokens within Submission-Time rules will be swapped for their values at time of submission. This means that the value of the tokens at submission time will be persisted within the job-specific copy of the Path Mapping rule.
Types of Tokens¶
The token type defines where the tokens will be pulling their values from. We currently support the following token types:
- env
The value for these tokens will be pulled from the current process’s Environment Variables.
You can use any environment variable as a token. Note however that if the environment variable does not exist, it will cause the path mapping process to fail.
- job
The value for these tokens will be pulled from the current Job. What the ‘current’ Job is largely depends on context: - At render time, this would be the Job that is currently rendering. - In the Monitor, this will typically be the currently-selected Job.
See the Token List for a list of currently supported job property tokens.
- sys
This type of token will be pulled from the underlying OS by Deadline.
While some of these may be retrievable from the environment, the idea is to simplify these by making the retrieval consistent across OSes.
See the Token List for a list of currently supported system property tokens.
The possible values for a token name will be specific to the token type above, and tell the substitution code how to retrieve the value in question.
For example, ${job:name} would be swapped with the current Job’s Name, and ${env:USER} would be swapped with the current value for the USER environment variable.
Token List¶
Token |
Description |
|---|---|
${job:name} |
The Job’s name. The Name Job Property. |
${job:batch} |
The name of the Batch that this Job belongs to. The BatchName Job Property. |
${job:user} |
The owner of the job. The UserName Job Property. |
${job:region} |
The region the job was submitted from. The RegionName Job Property. |
${job:department} |
The Department to which the Job belongs. The Department Job Property. |
${job:group} |
The Group to which this Job belongs. The Group Job Property. |
${job:pool} |
The Pool to which this Job belongs. The Pool Job Property. |
${job:secondary_pool} |
The Secondary Pool in which this Job belongs. The SecondaryPool Job Property. |
${job:submit_machine} |
The machine that submitted the job. The SubmitMachineName Job Property. |
${job:submit_date} |
The date/time at which the job was submitted. The SubmitDateTimeString Job Property. |
${job:started_date} |
The date/time at which the job started rendering. The StartedDateTimeString Job Property. |
${job:plugin} |
The name of the Deadline plugin the job uses. The PluginName Job Property. |
${job:id} |
The job’s ID. The JobId Job Property. |
${job:status} |
The job’s current state. The JobStatus Job Property. |
${sys:user} |
The operating system user. |
Example 1: Using Tokens with Global Rules¶
As a first example, let us assume that users will be submitting jobs to Deadline that rely on assets in project-specific folders that are mounted to the network drive P:locally.
In this case, a Global Rule could be used to map the network drive to actual project-specific folder, assuming the project is stored in a property that is exposed as a Path Mapping token. For the sake of example, we will assume the Job’s pool corresponds to the parent project.
Global Path Mapping Rule example:
Path: P:/
Windows Path: \\shared\public\projects\${job:pool}\
Linux Path: /mnt/shared/public/projects/${job:pool}/
OSX Path: /Volumes/public/projects/${job:pool}/
Region: All
When the path mapping is performed, the ${job:pool} tag will be replaced with the associated job’s Pool name (named after the project).
Example 2: Using Tokens with Submission-Time Rules¶
As another example, let us instead assume that assets will live in user-specific network folders, that are mounted to the U:drive on user machines.
In this case, a Submission Time Rule with a ${sys:user} could be used to allow render nodes to find these assets relative to the canonical network location at render time.
To do this, you would simply create a Submission-Time rule similar to the one below:
Submission Time Path Mapping Rule example:
Path: S:/
Windows Path: \\shared\public\users\${sys:user}\
Linux Path: /mnt/shared/public/users/${sys:user}/
OSX Path: /Volumes/public/users/${sys:user}/
Region: All
Because tokens in Submission Time rules are replace at submission time, the submitted job will have a copy of the above rules where all the tokens have already been swapped. For example, if ‘Job A’ is submitted by user ‘Bob’, ‘Job A’ would contain the following job-specific rule:
Path: S:/ Windows Path: \sharedpublicusersBobLinux Path: /mnt/shared/public/users/Bob/ OSX Path: /Volumes/public/users/Bob/ Region: All
Example 3: Using Tokens with AWS Portal¶
For tokens to be used in Path Mapping on your AWS Portal instances, you will be required to setup Mapped Paths with tokens explicitly for the AWS Portal instances.
For the purpose of this example, we are assuming that you have setup the AWS Portal Asset Server configured with \\shared\public\users\ as a root directory.
The simplest way to set this up is to create a rule that swaps S:\ to //shared/public/users/${sys:user}/ and enable Recursive Rule Application.
This would then automatically derive the above rule by leveraging the pre-defined rule that was generated for you when defining your Asset Roots.
If instead you would rather leave Recursive Rule Application disabled, you can still create an explicit rule that has the same effect. First you will need to locate the existing Global Path Mapping Rule that corresponds to your Asset Server Root (which should have been created for you). It should look similar to the following:
Path: //shared/public/users/
Windows Path: E:\sharedpublicusersfabb8458bf6fd0d757f7010b1511eb61\
Linux Path: /mnt/Data/sharedpublicusersfabb8458bf6fd0d757f7010b1511eb61/
Region: <stack id>
You can then pull the Windows and Linux Paths from the above mapping to use in token specific mappings. In order to leverage a token in the way previously described in the Submission Time Path Mapping Rules example, you would have to setup a separate mapping like so for each infrastructure stack’s region:
Path: S:/
Windows Path: E:\sharedpublicusersfabb8458bf6fd0d757f7010b1511eb61\${sys:user}\
Linux Path: /mnt/Data/sharedpublicusersfabb8458bf6fd0d757f7010b1511eb61/${sys:user}/
Region: <stack id>
