XMesh Introduction

What Is XMesh?

  • XMesh is a mesh caching solution.
  • The XMesh system consists of
    • an XMesh Saver utility for caching geometry objects to XMesh file sequences, and
    • an XMesh Loader object for loading the saved data back into the 3D application.
  • The XMesh Saver stores the render-time representation of one or more geometry objects to disk files.
  • The XMesh Loader can restore the saved data for viewport playback and rendering, while preserving all relevant information including vertex deformations, vertex velocity data, topology changes, mapping coordinates, smoothing information and material assignments.
  • XMesh is currently available as a plugin for
    • Autodesk Maya (XMesh Saver and Loader)
    • Autodesk 3ds Max (XMesh Saver and Loader)
    • The Foundry NUKE (XMesh Loader only)
  • XMesh has been used in heavy visual effects production since 2006.
  • It works reliably with multi-million polygon meshes and complex particle systems setups.
  • XMesh Saver stores the data in a compact and extensible channel-oriented, multi-file format.
  • XMesh Saver supports caching of geometry on the network using Thinkbox’s Deadline Network Manager, including the parallel processing of the same animation sequence on multiple network machines.

Why Use XMesh?

  • In a visual effects pipeline, working with dynamic scenes at all stages of production can be a problem.
  • For example,
    • The results of the Animation, Simulation and Effects departments have to be passed to the Lighting And Rendering department
    • But a lighting artist does not need access the actual animation rigs, simulation data or effects/particles setups to perform his work.
    • In fact, having access to the underlying rigs from the previous stages could be dangerous since some vital elements like nulls/helpers or particle system components could be accidentally moved, modified or deleted, affecting the final render output.
  • XMesh provides a way to cache the geometry data and removes the dependency on animation, simulation and effects rigs, giving the Lighting and Rendering department the precise results it needs in an unchangeable form.
  • In addition, XMesh allows the combining of multiple meshes into a single mesh, significantly reducing the performance hit on the 3D scene where one object with millions of polygons is typically faster to work with than thousands of objects with a few polygons each.
  • The data is externalized to file sequences outside of the main 3D scene, allowing for easy updates to the scene geometry data by simply resaving the caches.
  • Finally, thanks to the support for 3D applications like Autodesk Maya and Autodesk 3ds Max, XMesh can be used to exchange mesh data between the packages.

How Does It Work?

  • An XMesh file sequence uses multiple files per frame, resulting in a large number of files per sequence.
  • This has several positive implications:
    • Unchaging channel data can be stored just once for the whole sequence, reducing disk usage
    • Multiple machines can save to the same sequence in parallel for faster turnaround
    • A sequence can be saved in multiple runs, adding more frames to existing ones if needed
    • Sub-ranges of an existing sequence can be resaved / updated without the need to resave the whole sequence
    • Corrupted data can be fixed easily by resaving the individual frame without the need to resave the whole sequence
    • Multiple CPUs can compress and save individual data channels of the same frame in parallel
    • Multiple CPUs can decompress and load individual data channels of the same frame in parallel

To learn more about the structure and content of the XMesh file sequence, see the topic XMesh MY File Sequence.

Saving Modes

The XMesh Saver provides several saving modes:

  • Saving of multiple objects as a single mesh in world space - this mode lets you combine many objects into a single mesh to optimize the performance of heavy scenes when access to the individual entities is not needed.
  • Saving of multiple objects as individual caches in world space - thos mode preserves the individual objects by saving one XMesh sequence per object, but bakes the world transforms into the mesh cache, making the recreation of the original scene as easy as placing all XMesh Loaders at the world origin.
  • Saving of multiple objects as individual caches in object space - this mode bakes only the object space transforms and allows each XMesh Loader to replace the respective source object in an existing hierarchy, while caching object-space deformations only.

What Data Does XMesh Support?

Autodesk 3ds Max

  • XMesh MX can save and load arbitrary named data channels, however the current implementation of XMesh for 3ds Max supports a hard-coded set of channels.
  • Only the Vertex and Face lists are madatory and always included, all other channels are optional and can be disabled when saving.
  • XMesh MX Saver supports:
    • Vertex list channel - always included, contains the Vertex positions as float32[3]
    • Face list channel - always included, contains the Face or Polygon indices into the Vertex list
    • Per-face SmoothingGroup channel - a 32 bit integer defining 32 groups - if two faces share a group, their edge will be smoothed
    • Vertex Normals channel - overrides Smoothing Groups when loading if the SmoothingGroup channel is also available
    • Per-face MaterialID channel - index into the Multi/Sub-Material list of materials
    • Per-vertex Color channel (a.k.a. Mapping Channel 0) - the original 3ds Max R2 Vertex Color channel
    • TextureCoord channel (a.k.a. Mapping Channel 1) - the original 3ds Max R1 Texture Coordinates channel
    • Mapping channels (from Mapping2 to Mapping99) - the 3ds Max R3 additional 98 mapping channels
    • Vertex Velocity channel - used by the Loader to extrapolate vertex positions on sub-frames, if requested via the Loader’s settings
    • Selection channel - vertex soft selection data
    • FaceSelection channel - face selection data
    • FaceEdgeVisibility channel - per-face edge visibility when saving a TriMesh
    • EdgeSharpness channel - per-edge sharpness used for Polygon meshes
    • VertexSharpness channel - per-vertex sharpness used for Polygon meshes

Autodesk Maya

  • XMesh MY can save and load arbitrary named data channels, however the current implementation of XMesh for Maya supports a hard-coded set of channels, as well as a specific subset of the channels saved by 3ds Max.
  • Only the Vertex and Face lists are madatory and always included, all other channels are optional and can be disabled when saving.
  • XMesh MY Saver supports:
    • Vertex list channel - always included
    • Face list channel - always included
    • EdgeSmoothing - automatically converted to 3ds Max Smoothing Groups when loading in 3ds Max
    • Per-vertex Normal channel - overriding Smoothing Groups if both are available when loading in 3ds Max
    • Maya Shading Groups saved as 3ds Max-compatible MaterialID channel - an additional MaterialID to Shading Group text file will be stored to control index to name mapping
    • Current UV set channel - saved as map1 channel
    • Custom UV sets channel - named from map2 to map99, if present. These are mainly for round-trip data transfer between 3ds Max and Maya.
    • Vertex Velocity channel - if available and explicitly requested by the user
  • When loading files saved from Autodesk 3ds Max, the XMesh MY Loader currently supports the following channels:
    • Vertex list channel
    • Face list channel
    • Per-face SmoothingGroup channel - automatically converted to Maya Edge Smoothing
    • Per-vertex Normal - overriding Edge Smoothing, if the SmoothingGroup channel is also available
    • Per-face MaterialID - converted to Shading Groups when loading in Maya
    • VertexColor channel - 3ds Max Mapping Channel 0, loaded as vertex color set.
    • TextureCoord channel - (3ds Max Mapping Channel 1), loaded as UV set named map1. Note that the 3rd (W) component of 3ds Max UVW channels will be lost.
    • MappingN channels (from Mapping2 to Mapping99), loaded as map2 to map99. Note that the 3rd (W) component of 3ds Max UVW channels will be lost.
    • Vertex Velocity - used by the Loader to extrapolate vertex positions on sub-frames, if requested via the Loader’s loading mode.

How Does The XMesh Loader Work?

Autodesk 3ds Max

  • The XMesh MX Loader is a geometry object and can be modified like any other geometry object in 3ds Max.
  • It can be freely translated, rotated or scaled to transform the loaded geometry, including animated transformations.
  • The XMesh MX Loader can be parented to other objects and be part of a hierarchy.
  • Topology-dependent modifications should not be used though since the mesh can change completely from one frame to the next.

Autodesk Maya

  • The XMesh MY Loader node is a geometry object and can be modified like any other geometry object in Maya.
  • It can be freely translated, rotated or scaled to transform the loaded geometry, including animated transformations.
  • The XMesh MY Loader can be parented to other objects and be part of a hierarchy.
  • Topology-dependent modifications should not be used though since the mesh can change completely from one frame to the next.

Two File Sequences

  • The XMesh Loader object can load one or two XMesh sequences from disk - a Render Sequence, and a Proxy Sequence.
    • The Render Sequence is used for the render-time representation of the object, and optionally for viewport display if no Proxy Sequence was saved. Even if a Proxy sequence is present, the viewpor display can be forced to show the Render Sequence when needed.
    • The optional Proxy Sequence, usually saved at a lower resolution, can be used as a viewport proxy for faster performance, but can be selected to be rendered too if needed (e.g. for fast previews).
  • The two sequences can be saved with different sampling steps and can include completely different channels (for example the Proxy sequence could exclude Velocities, Mapping, Materials or Smoothing data to reduce the data amount and improve playback speed). In addition, the Proxy sequence can be optionally saved with polygon reduction of a user-defined percentage.

The XMesh Loader lets you pick any XMesh file from the sequence and will replace the frame number with the current or custom time as needed. It can also optionally load only one frame from the sequence at every time, in which case the frame selected as representative of the whole sequence will be used without replacing its frame number.

Retiming

  • The XMesh Loader exposes controls to offset the playback in time, or to freely retime the sequence using a custom graph.
  • The XMesh Loader can also limit the loading of the sequence to a user-defined frame range to ensure that the retiming controls don’t request a non-existing frame.

Memory Caching

  • The XMesh Loader does not pre-load frames into memory, nor does it keep previously loaded frames in memory. This is done to avoid problems with very heavy meshes where a single frame can use up nearly all available RAM.
  • The XMesh Loader does support memory caching of individual channels to accelerate the loading of non-changing channel data though. For example, if the Face List is consistent between multiple frames, it will be loaded just once on the first frame and reused to build the mesh on the following frames without accessing the compressed data from disk.
  • When the XMesh Loader is set to use the same single frame on every frame, it will be cached and will not be reloaded on every frame. If the playback is limited to a custom range though where the first and last frame are the same, the specified frame will be reloaded from disk on each time change.

Multi-Threading

  • The XMesh Loader uses multi-threading to accelerate the loading process.
    • Since every data channel is stored in a separated compressed file, and decompression is rather CPU-intensive and single-threaded, a separate thread will be spawned to decompress each data channel.
    • This means that a multi-core machine with 8 threads could load up to 8 data channels at the same time.
../../_images/XMesh_Saving_And_Loading_License_Diagram_700p.png