Data Precision In The Sequoia Pipeline¶
Overview¶
Understanding how data is stored and processed outside and insde of Sequoia is highly important.
The following topic discusses point cloud storage and data processing in past and present versions of Sequoia.
The Basics¶
The Life Of LiDAR Data¶
Most point clouds will go through the following conversion steps during their lifetime:
Points start their lives as directions and distances, based on the scanner’s instantaneous mirror position, the orientation sensors (offset from vertical, e.g.), and the time of flight or phase offset of the return signal.
The points get converted to cartesian XYZ coordinates, in scanner space.
Multiple scans are registered together, in a local registration space. This is normally done with targets, spheres, and natural features in the scanned environment. Newer scanners use GPS or IMUs to produce fairly accurate initial estimates of the registration, then do cloud to cloud refinements.
The point cloud is georeferenced to a local surveying coordinate system or a global GIS coordinate system. This may be done by using GPS or total station measurements of target positions.
Point cloud coordinates need to maintain absolute precision to work well.
For example, scanner accuracy might be 2 mm, and storage at 0.1 mm would then represent the data well.
In the case of the Riegl file format for example, 0.25 mm precision is baked into the representation.
Where Does Data Live?¶
Point data usually exists in source point data files in one of the many industry formats.
Different file formats might have different data representation based on the nature of the data, the developer’s goals etc.
Once the data is bought into Sequoia, it is always converted into Sequoia’s SPRT file format.
The data is then loaded for display and meshing into various components of the Sequoia system.
The data can be re-exported to a supported output point data format after processing.
The resulting mesh is usually exported to one of the supported mesh file formats.
Single Precision In Sequoia v1.0¶
The initial Sequoia v1.0 release represented point position X, Y and Z components as 32 bit floats (also known as “Single precision” or simply “Singles”).
Due to the way the 32 bits are used to represent both the sign, exponent, and fraction of the resulting floating point number, the farther away from the world origin (coordinates 0,0,0) a point is located, the lower its decimal precision becomes.
As result, point clouds with a very large offset (millions of meters) from the origin resulted in data loss, and incorrect meshing and display results in Sequoia v1.0.
The most common data with large offsets are georeferenced LIDAR scans in GIS coordinate systems like the State Plane Coordinate Systems.
This problem affected both point data storage and calculations.
Double Precision - Pros And Cons¶
It would appear that switching Sequoia to use 64 bit floating point numbers (also known as “Doubles”) for both storage and calculations would the obvious solution.
However, representing all data as Doubles in every case could be wasteful:
It would significantly increase the storage requirements when saving SPRT files even if the data is close to the origin.
It would increase the memory usage of the application.
It could affect performance.
High Precision Concepts In Sequoia v1.1+¶
Sequoia v1.1 introduces new workflows that try to strike a balance between retaining high precision throughout the pipeline, while avoiding any drawbacks (storage requirements, memory usage, performance) when possible.
Data storage will adapt based on the source data precision and a user-specified target precision.
Meshing can employ a User Coordinate System to process the mesh relative to a local origin best suited for the particular point data distribution.
Point Data Conversion And Storage¶
The SPRT file format supports three different data representations:
Single Precision Storage¶
This is the same approach as in Sequoia v1.0 using 32 bit floats.
This method will be used when all point data is relatively close to the origin and the resulting floating point precision is enough to retain all the source data precision without any loss.
Single+Offset¶
The data in each Octree voxel will be represented as a 32 bit float relative to the origin of the voxel.
The voxel’s offset from the origin will be stored using 64 bit precision floats.
This combines the compactness of the Single Precision mode with the higher precision of Doubles.
This method can resolve 0.1 mm at distances up to one Astronomical Unit (the distance from the Sun to Earth), which is 149,597,870,700 meters or 92,955,807 miles.
Double Precision¶
Each point’s X, Y and Z coordinates will be represented as 64 bit Doubles.
This will result in much larger SPRT files.
On the plus side, this method can resolve 0.1 mm at distances roughly equal to the distance from Sun to Pluto.
Point Data Display¶
The underlying viewport technology of Sequoia (based on the Ogre3D game engine) only supports 32 bit floats for position information.
However, since the beginning Sequoia has been using a local origin approach which transforms all display data into a local coordinate system based on the viewer’s location.
These transforms have been enhanced in v1.1 to ensure all Double precision transforms necessary to retain the original precision of the point data are applied throughout the display pipeline.
As result, high-precision data will be displayed without visual artefacts in the Sequoia viewports.
Mesh Data Precision¶
Most mesh file formats do not support double precision.
However, as long as the mesh is located close to the origin, this limitation does not matter.
In Sequoia v1.0, point data meshed far away from the origina could produce noisy results due to the lack of floating point precision discussed above, but affecting the vertices of the mesh.
To solve this problem, Sequoia v1.1 introduces the User Coordinate System (UCS) object.
A UCS allows the Mesher to operate in a user-defined local space even if the point data has a very large offset from the world origin.
Both the Position and the Orientation of the UCS object are taken into account, allowing additional control over the mesh raster alignment in the final output.
A UCS can be
Created manually via the Create Objects toolbar or menu,
Added manually to a Mesher via the Mesher’s User Interface, or
Created and connected automatically when the source point cloud data precision requires it and a meshing or Hacksaw operation is requested via the Mesher object.
The following illustrations show a point cloud meshed at a distance of 10,000, 100,000 and 1,000,000 meters from the world origin - you can see the mesh quality deteriorating as the decimal precision of the vertex positions goes down with the distance:
Creating a UCS object in the middle of the point cloud solves this problem:
Data Already Containing Precision Loss¶
In some cases, the incoming data might suffer from precision loss even before entering Sequoia.
This can happen if a point cloud with a large offset from the origin was incorrectly saved using single precision to the source file.
Sequoia will check the incoming precision of source files and warn you if the storage precision has likely caused data loss.
In these cases, Sequoia cannot recover the missing data because the loss is already baked in the file.
You will have to resave the source data in the correct format if you have access to the original data, or
Contact the provider of the data to request a correctly saved file.
