Partitioning Rollout

Available in: Krakatoa v1.0.0 and higher.

• Particle partitioning is a way to multiply the total particle count of a particle system through the repeated running of the particle system with different seed values.
  • Generally speaking, particle partitions are related subsets of particles processed independently and cached to disk files.
• Each partition is saved using different random seeds to generate a slightly different solution.
• The partitions can be combined at render time using PRT Loaders to produce an aggregate result containing more particles than 3ds Max can normally generate in a single pass.
• In addition, partition generation can be optionally split between multiple machines using the Deadline Network Manager or by manually running multiple copies of Krakatoa on multiple workstations to generate the same high amount of particles in a fraction of the time needed to process in a single pass.
  • For example, instead of creating 10 million particles per frame on one machine, you could save 1 million particles on 10 machines in 1/10th of the time.
• Krakatoa contains tools to automate the process of creating particle partitions.
• These tools are only available when Particle Render Mode in the Main Controls rollout is set to “Save Particles To File Sequence”.

Partition Count

• Partition Count controls the total number of partitions to be created.
• Default value is 10.
• Value Range is 2 to 100.
• The number will be included in the saved particle file’s name, for example three partitions of “particles_.prt” will be saved as particles__part1of3_.prt, particles__part2of3_.prt and particles__part3of3_.prt
• For each partition, the Random Seeds of Particle Flow operators and Legacy Particle Systems will be incremented as discussed below.

NOTE: The first partition will always use the original Random Seeds, the second partition the original Random Seeds Plus One and so on. Thus, the content of Partition 1 would always be identical to the content of a particle file sequence saved without partitioning.

• See Krakatoa FAQ for discussion of partition counts.

Partition Range From M To N

• The Partition Range From and To spinners can be used to generate one or more partitions of the full range of partitions.
  • This can be useful when the partitioning had to be canceled prematurely after N partitions have been generated and you want to continue from a higher partition than 1, or to replace a single partition with a fresh file sequence.
  • To generate only one partition out of the whole range, simply enter two identical values in the From and To fields.

Random Seed Toggles

• Particle Flow uses pseudo-random numbers to create variation in particle behavior. To change the results, you can enter different seed values to alter random number computation.
• The Random Seed field appears in several Particle Flow operators under Uniqueness: Seed, as well in the Legacy Particle Systems and some Thinking Particles operators.
• When creating particle partitions, you can toggle the following options to select which seeds are automatically incremented:
  • PFlow Position - affects Position operators like Icon Position and Object Position
  • PFlow Spawn - affects Spawn operators like Spawn and Collision Spawn.
  • PFlow Speed - affects Speed operators like Speed and Speed By Surface
  • PFlow Rotation - affects Rotation operators like Rotation
  • PFlow FumeFX - affects FumeFX operators like FumeFX Birth
  • PFlow Others - affects any other PFlow operators that have a Random Seed value but are not in any of the above categories.
  • Legacy Particles - affects 3ds Max legacy particles of second generation (Blizzard, PArray, PCloud, SuperSpray) which provide a random seed value. First generation Legacy systems (Spray and Snow) do not provide a random seed and cannot be partitioned correctly.
  • Thinking Particles - affects Cebas Thinking Particles operators that have a random seed.
  • Box#3 Geometry - affects Geometry sub-operators inside Orbaz Particle Flow Tools Box #3 Data Operators and Tests.
  • Box#3 Random - affects Random sub-operators inside Orbaz Particle Flow Tools Box #3 Data Operators and Tests.
  • Box#3 Sclr/Vector - affects Scalar and Vector sub-operators inside Orbaz Particle Flow Tools Box #3 Data Operators and Tests.
  • Box#3 Icon - affects Icon sub-operators inside Orbaz Particle Flow Tools Box #3 Data Icon Operators and Tests.
• Note that in addition to the built-in Seed Incrementation, Krakatoa provides a way to execute arbitrary MAXScript functions defined by the user as described here.

Modifier Random Seed Toggles

Available in Krakatoa v1.1.2 and higher

• There are checkbuttons which can be used to affect the random seeds of regular Modifiers that use random number generation.
• Currently, the only such Modifier shipping with 3ds Max is the Noise Modifier.
• Using a Noise Modifier, Krakatoa can partition any particle source that does not support random seeds including pre-saved PRT, CSV and RealFlow BIN file sequences and geometry vertices from scene geometry.
• In addition, the Random Seeds of PRT Volume and PRT FumeFX base objects can also be affected when the respective options are enabled:
  • PRT Loaders - affects Random Seeds of supported Modifiers applied to PRT Loader objects.
  • PRT Volumes - affects Random Seeds in the Base Object and/or Modifiers applied to PRT Volume objects (added in v1.5.0)
  • PRT FumeFX - affects Random Seeds in Base Object and/or Modifiers applied to PRT FumeFX objects (added in v1.6.0)
  • Geometry - affects Random Seeds of Modifiers applied to any other Geometry objects.
• See Partitioning Existing File Sequences and Vertices Tutorial for more details on using a Noise Modifier to randomize particle positions.
  • This method is much more straight-forward than the hack described in the Partitioning Existing File Sequences tutorial, but produces very similar results.

Excluding Seeds Of Specific Objects, Events and Operators

• Krakatoa allows you to control the seed incrementation at a finer level using a special naming convention for objects, Particle Flow Events and Operators, as well as Thinking Particles Systems and Dynamic Sets.
  • If an Object Name contains the string “NOSEED” (case-insensitive, “noSeed”, “NoSeed”, “noseed” etc. would also work), the object will be skipped even if it matches a category that is enabled for Seed Incrementation. The objects include Particle Flow Emitters and Legacy Particle Systems with Random Seeds, as well as Thinking Particle systems.
  • If a Particle Flow Event contains the string “NOSEED” in its name, all operators inside the event will be excluded from Random Seed incrementation.
  • If a Particle Flow Operator contains the string “NOSEED” in its name, the single operator will be excluded from Random Seed incrementation.
  • If a Sub-Operator inside a Orbaz Particle Flow Tools Box #3 Data Operator or Test contains the string “NOSEED” in its name, the single sub-operator will be excluded from Random Seed incrementation.
  • If a Thinking Particles Dynamic Set contains the string “NOSEED” in its Comments field (the edit text field on top of the set), any Operators inside the Dynamic Set will be excluded from Random Seed incrementation.
  • If a deformation Modifier with a Seed property (like Noise Modifier) is applied to a PRT Loader or Geometry Object and the name of the Modifier contains “NOSEED”, it will be ignored.
• In addition, if the >Affect Seeds of Selected Objects Only option is checked, any objects that are not selected in the scene will be excluded from Random Seed incrementation.
  • This is supported in both local and network partitioning mode - the current selection state will be recorded and submitted with the Deadline job.

Combining Partitioning with Particle Flow Toolbox #3 Disk Cache Operators

• The Disk Cache operator included in Box #3 and also sold as a stand-alone tool can be used as a “safety net” when performing Partitioning. It can cache particles to disk files while the Particle Flow is updating and reuse the saved data if the Flow has to be recalculated again from the beginning, for example after a crash or power loss in the render farm.
• When the >Increment Box #3 Disk Cache Filenames option is checked, Krakatoa will change the output file names of all Disk Cache operators to match the same partitioning naming convention as Krakatoa partitions. If the Box #3 is set to cache “Render”, cache files with the original file name plus the partition signature will be saved to disk. If the partitioning has to be canceled and continued at a later point, all frames that have already been partitioned will use the cached data instead of calculating everything from the first frame, resulting in much shorter pre-roll times.
• Once a partition is finished, the original file name will be restored.
• Like with Seeds, adding “NOSEED” to a Disk Cache operator’s name, event or flow will exclude it from being affected by Krakatoa partitioning.

Local Partitioning

• Krakatoa allows you to calculate partitions on your local workstation.
  • Of course, other than calculating partitions on Deadline, this option requires the partitions to be saved sequentially, which means a longer processing time.
• The Generate Partition Range Locally… button will save only the Partitions defined by the Partition Range From … To … values.
• The name of the partition files will still use the Partition Count as the highest count, but only a subset of all possible partitions will be processed.
  • For example, if you are saving 10 partitions (Partition Count is set to 10) but you want to process only partitions 2 and 3, you can enter 2 in the Partition Range From field and 3 in the To field and press this button.
  • The sequences will be named …_part_2_of_10_… and …_part_3_of_10_… and you can process the rest of the partitions (1 and 4 to 10) later using the respective From and To values.

Tip: If your workstation has 2 or more CPUs or Cores and you want to partition in parallel without using the Deadline network manager, you can launch multiple copies of 3ds Max, load the same scene and save multiple sub-ranges with the same Partition Count, for example one copy of 3ds Max could save the range from 1 to 5 and the other from 6 to 10. Since Particle Flow is single-threaded, each CPU will be fully loaded and the only limitation would be the memory.

• The Generate All Partitions Locally… button will save all partitions from 1 to Partition Count, completely ignoring the Paritition Range settings.
  • If you have set the Partition Count to 10, Krakatoa will process from 1 to 10, if it is set to 5, Krakatoa will process from 1 to 5 and so on.
  • This option is useful when leaving your workstation unattended and you want all partitions calculated in sequence.

Note: In Krakatoa prior to v1.5.1, the local partitioning was performed by calling the MAXScript render() function once for each frame in the specified time range or frame sequence. This caused Thinking Particles systems to evaluate the complete simulation from the first frame on each frame, resulting in exponentially longer frame times. In Krakatoa v1.5.1, this has been fixed by calling the renderer just once for the whole frame range (except when saving “pickup” Frames from the 4th option of the Render Scene Dialog). As result, Partitioning of Thinking Particles is significantly faster, and partitioning of all other sources is also faster due to more optimal memory management.

Skip Existing Files

• The >Skip Existing Files When Saving option used to affect only local partitioning prior to v1.5.1.
• Since v1.5.1, it applies to both local and Deadline partitioning.
  • When checked, Krakatoa will not save frames which already exist on disk (but it will still have to ask Particle Flow and other particle sources to update the frames that have to be saved which can cause these sources to update all frames up to the one to be saved due to their history-dependent nature).

For example, if you were saving a partition and canceled in the middle of the processing, instead of setting the output range to start at the first missing frame, you can just check this option and save the original time range. Any existing frames will be skipped and only the missing ones will be saved. Also, if you want to update only a couple of frames from an existing sequence, or if some frames were corrupted, you can delete them from disk and then run partitioning with >Skip Existing Files When Saving on - only the deleted frames will be saved again.

Potentially Missing Texture Channels Warning

• When saving particles from Particle Flow, a check will be performed to ensure that all Mapping Channels used by the flow are set for saving in the Channels Rollout.
• If a channel is defined by the Particle Flow, for example via a Mapping or Mapping Object operator or Box #3 Sub-Operator, the following warning will be issued:

• Pressing Yes will add the missing channels to the list of channels to save and continue with the partitioning.
• Pressing No will not add the missing channels and partition without saving them.

Partitioning One Frame Only

• In almost all cases, the user wants to partition a file sequence.
• Very often though, the Time Output settings in the Render Scene Dialog of 3ds Max are set to a single frame.
• In addition, saving a single frame while partitioning is not really possible - a “feature” in Particle Flow prohibits the changing of Random Seeds when a single frame has been requested for saving. Krakatoa has to save at least TWO frames to work around this limitation, so saving frame 0 results always in saving both frames 0 and 1.
• To make sure the correct time segment is partitioned, Krakatoa 1.1.0 introduces a Sanity Check for single frame partitioning:

Local Partitioning Progress

• During a local partitioning, a set of two progress bars and two text lines will appear in the area between the buttons and the Deadline Settings.
• The top bar will show the progress of the current partition in red.
• The bottom bar will show the total progress of the partitions in blue.
• The first line will show:
  • The partition Number
  • The partition Count so far out of the total requested partitions count
  • The requested partition Range
  • The current Frame number
  • The frame count so far out of the frame count for the current partition
  • The current total frame count out of the total frame count for all requested partitions
  • The total number of frames left to save.
• The second line will show:
  • The Last Frame Time in Hours:Min:Seconds
  • The Elapsed Time so far in Hours:Min:Seconds
  • The Remaining (Estimated) Time based on the progress so far.
  • The text “Press Esc To Cancel.”
• You can press the Esc key at any time during the local partitioning (as long as 3ds Max has focus) to stop the process.
• In Krakatoa v1.5.1, the Render Progress dialog of 3ds Max will also be displayed during partitioning.
  • Pressing its Cancel button will cancel not only the current partition but the whole partitioning process.

Deadline

• Krakatoa takes special advantage of Thinkbox Software’s Deadline render management system.
• Krakatoa requires Deadline 3 or higher - support for Deadline 2.x was removed in v1.6.0.

Submit All Partitions to Deadline

• Click this button to submit automatic partitions to Deadline.
  • Krakatoa will create the number of partitions specified in the Partition Count field, and will automatically name files based upon the base name defined in the File Picker.
  • Random number seeds will be incremented based upon which seed buttons are toggled on and off.
  • Job priorities and render pools will be based upon settings in the Submit Max to Deadline dialog.

Submit Current Partition to Deadline

• Click this button to submit the current particle system to Deadline as a partition.
• The resulting particle file will be named according to the Current Partition #, Partition Count value and the base file name entered in the File Picker in the “Save Particles To File Sequence” text field in the Main Controls rollout.

Deadline Submission Mode

One Job Only, Partitions As Tasks

• This is the default submission mode.
• A single Deadline MAXScript job will be created.
  • This script job will run a Krakatoa control script which will perform partitioning in a very similar way to the Local Partitioning described above.
  • Each Frame of the Job will represent a partition.
  • If the Concurrent Tasks option is enabled in the submission settings, this mode allows a single network machine to open multiple copies of 3ds Max and partition multiple sequences on the multiple cores.
• The main drawback of this mode is that should a task crash, the whole partitioning will have to start from the first frame since it is performed by a script calling the renderer just once for the whole sequence.

One Job Per Partition, Frames As Tasks

• This mode produces one Deadline job for each partition, thus resulting in a lot of jobs in the Deadline monitor.
• Other than the other mode, this one does not run a script but performs regular rendering in Save Particles To File Sequence mode.
  • Each frame to be saved will be represented as a task (or multiple frames could be represented as a task) that can be assigned to a network machine.
• The main drawback of this method is that the Monitor will get cluttered easily and that if multiple machines work on frames of the same partition, they have to preroll any PFlow or TP frames saved by the other machines working on the job, making it less effective.
• This mode is provided as a fallback (a Plan B of sorts) for cases where the first method fails and should not be considered in typical production scenarios.

Deadline Job Controls

• Krakatoa uses the functions provided by the Submit Max To Deadline (SMTD) script shipping with Deadline.
• It also exposes all relevant controls to the Particle Partitioning rollout:

Job Name:

• This field will be automatically filled with the current .MAX file name.
• If the name has changed due to a new scene being opened, you can press the [>>] button to get the current MAX file name or base the job name on the save file name.
• You can also enter any name you want in the field.

Comment:

• This field will be automatically filled with the Max version and Krakatoa version number and a note that this is a Partition Submission job.
• You can enter any text you like in this field though.

Name:

• This field will contain the name of the user currently logged in Deadline.
• You can enter any other user name here.

Department:

• This field will be taken from the current SMTD script settings (if available).
• Changing the Department in SMTD or Krakatoa will automatically change it in the other script.
• This is the ONLY value shared between the two scripted GUIs.

Group:

• This drop-down list can be used to specify an existing Group of machines.
• Groups do not control scheduling, but can limit a job to a specific group of network machines - for example based on hardware characteristics or installed software.

Pool:

• This drop-down list will contain all pools available on Deadline.
• You have to select a valid pool for the submission to succeed.
• The Pool controls what machines will pick the job and affects scheduling.

Priority:

• This spinner and color slider can be used to specify the priority of the job - the higher the number, the higher the priority.

Render Task Chunk Size (Frames Per Task):

• This value is set to 5 by default and specifies how many frames will be created within a single task.

Limit Number Of Machines Rendering Concurrently:

• This value is set to 1 by default and specifies the number of machines to work on a single Partition job.
• Since particle calculation is history-dependent, it is advisable to leave only one machine to calculate the whole partition job.

Note The Machine Limit will be enforced (other than in SMTD where it can be disabled completely) and the “Sequential Rendering” flag will be set for faster particle calculation.

• You can use the Generate Partitions Locally options to get the same results, but it will take longer.

Deadline Detection And Integration

Available since Krakatoa v1.1.2 Moved from Preferences to Partitioning rollout in Krakatoa v1.6.0

• This group of controls provides means to specify the version of Deadline to be used when submitting Partition jobs.

Deadline Detection drop-down list

• This option controls whether Deadline will be searched for automatically or not.
• The state of the drop-down list will be stored in an INI file and respect in future sessions.
• The drop-down list provides two options:

Do Not Detect Deadline

• Deadline will not be searched for.
• This is useful when a machine that used to be connected to a Deadline farm has been temporarily removed from it - the detection could take a while before Krakatoa figures out the Repository is not accessible.
  • You can always switch to Detect and Use Deadline at a later point if the need arises, or press the Detect Deadline Now! button to detect manually.

Detect and Use Deadline

• Krakatoa will attempt to detect Deadline 3 or higher and will use it automatically if found.
  • Deadline 2.7 and older are NOT supported since Krakatoa v1.6.0 and won’t be detected/used.
  • When first started and if the option Do Not Detect Deadline is not explicitly specified, Krakatoa will attempt to detect Deadline.

Detect Deadline Now! button

• The button can be used to enforce a new detection of Deadline.

SEE ALSO

• Partitioning Workflow
• Creating Particle Partitions Tutorial