Partitioning Particles

Overview

Partitioning is the term used in KRAKATOA lingo to describe the generation of similar yet slightly different versions of the same particle system to produce a final particle count beyond the capabilities of the source particle system.

Concept

  • KRAKATOA was designed to efficiently render very large numbers of particles.
  • However most particle systems in the host 3D applications (3ds Max, Maya, CINEMA 4D) require a lot more memory to simulate such high particle counts.
  • This means that a computer system capable of rendering 100 or 500 million particles with KRAKATOA would not be able to simulate more than a small fraction of these counts (usually 1 to 10 million particles would be the upper limit).
  • Most particle systems let you define the random seed value controlling the initial distribution of the particles within the emitter, and any other random decisions like random speed generation etc.
  • Instead of simulating 100 million particles in one go, Partitioning lets you process several simulations with a fraction of the final count (say, 100 simulations of 1 million particles), each with a different random seed.
  • The simulations are cached to disk as multiple numbered file sequences which can be loaded automatically using a single PRT Loader object to constitute a single particle dataset with the desired particle count.

Benefits

Besides working around the system limitations of the host hardware and software, Partitioning offers a number of other practical benefits.

Parallel Processing

  • Many particle systems are single-threaded, using only one CPU or Core of a machine that often has 4, 8 or more Cores.
  • Partitioning can be processed sequentially on a single machine, but if submitted to the Deadline network compute manager, it can be processed
    • On multiple machines, or
    • As multiple Concurrent Tasks on the same machine (if memory permits).
  • As result, if simulating 100 frames of 1 million particles takes 10 minutes, simulating 100 Partitions sequentially would require 100 times longer, or 1000 minutes (16.6+ hours!)
  • Running the same simulation on a single machine with enough memory and 8 cores as 8 Concurrent Tasks under Deadline’s control would take less time (usually around 4 to 6 times less due to other overheads)
  • Running the same simulation on 10 machines under Deadline’s control would finish the simulation in about 1.6 hours
  • Running the same simulation on 100 machines under Deadline’s control would finish all 100 Partitions in about 10 minutes.

Predicting Particle Count Requirements

  • In many cases, the KRAKATOA artist cannot know in advance how many particles will be needed to achieve the desired render quality.
  • Partitioning lets you perform the particle generation in multiple stages and perform intermediate test renders to determine what amount of particles is enough.
  • Let’s say you have a limited time to simulate and render a project with a fixed delivery deadline.
    • You can set up the number of partitions to 100 and start processing them.
    • After 10 partitions have been saved, you can perform a test render on a representative frame to get an idea of the quality and coverage.
    • After 50 partitions you might perform another test render and discover that the render quality is “good enough” - at this point, you could start rendering the project, saving 50 partitions less than initially intended.
    • If you suddenly get a call from the client that the delovery deadline has moved a week later (this would never happen in real life), you could decide to process a few (or all) of the remaining partitions to get an even higher quality, even if it is not absolutely needed.

Data Safety

  • Very rarely, a file from a particle data sequence might get corrupted.
  • Regenerating a single file around the end of a sequence would require pre-rolling the whole simulation up to that time to just save one file.
  • When saving a single cache without Partitioning, such a file corruption could cause significant delays.
  • However, if you have simulated 50 Partitions and one file from one of these partitions is corrupted or accidentally deleted, you can likely disable that single sequence and render the remaining 49 sequences without affecting the final look of the rendered output!

Free Viewport Proxies

  • The PRT Loader object of KRAKATOA lets you switch each file sequence being loaded to show up in the Viewport, in the Renderer, in neither, or in both.
  • When you load 100 Partitions, you have 100 potential Proxy sequences at no additional processing cost.
  • You can enable any of the Partitions to be visible in both the Viewport and the Renderer, while the rest would be visible only in the Renderer.
  • This way, you can set the Viewport display to 100% and load all particles sequentially, but only show 1% of the total render count.
  • This gives you the equivalent of loading 1% with Every Nth Particle mode, but with the pefromance benefit of 1% First N Particles, and none of the drawbacks.

Parallel Loading (3ds Max only)

  • The PRT Loader of KRAKATOA MX offers an additional ability that can be turned on to benefit from a very fast storage medium like an SSD or Fusion-IO drive.
  • When the Fast PRT Loading option is enabled, each sequence of the PRT Loader can be processed on a separate thread.
  • Since the loading of a PRT sequence is CPU-bounb due to the unzipping of the stream being single-threaded, this means that the loading of Partitions can be parallelized on multi-core machines.
  • This capability is usually beneficial when performing local iterative render tests and should not be used when network rendering using the render farm, as it could easily saturate even high-bandwidth networks.

Implementation Details

The Partitioning feasture of KRAKATOA has been implemented slightly differently in the various host 3D applications.

3ds Max

  • The KRAKATOA MX Partitioning logic was implemented completely using MAXScript as part of the KRAKATOA GUI.
  • Fine granularity random seed controls are available for all supported particle sources (Particle Flow, Thinking Particles etc.) and their components.
  • Custom MAXScript functions can be registered and run whenever the random seeds of the scene particle sources are changed to add additional user-defined logic.
  • Native Particle Flow or Thinking Particles caching must be disabled for the random seed manipulation to have any effect.

Maya

  • The KRAKATOA MY Partitioning logic was implemented completely using MEL as part of the PRT Saver tool.
  • Technical directors and artists can potentially modify the script if necessary.
  • Maya’s native particle system caching must be turned off for the random seed manipulation to have any effect.
  • Additional controls to jitter initial positions and velocities have been implemented to allow Initial State Partitioning to be peformed without random seed modifications.

CINEMA 4D

  • The KRAKATOA C4D Partitioning logic was implemented by the connection plugin in C++ and cannot be modified by the user.
  • Native particle system caching (e.g in X-Particles) must be disabled for the random seed manipulation to have any effect.
  • Additional controls to jitter initial positions and velocities have been implemented to allow Partitioning to be peformed without random seed modifications.