Technical Architecture

This document outlines the architecture for the Omniverse Kit Extension (whoimpg.biologger.subscriber) that serves as the bridge between high-frequency biologger data and 3D rigged animal assets.

The core objective is to move data from a ZeroMQ (ZMQ) stream into the USD Fabric (runtime memory) at low latency, ensuring that the 3D model, sensor telemetry, and behavioral predictions remain in lockstep.

Architectural Overview

The extension operates as an asynchronous background service within Omniverse USD Composer. It decouples the simulation/processing logic (running on a separate machine or process) from the rendering viewport.

Data Flow:

  1. ZMQ Socket: Listens for PUB packets containing Quaternions and Physics telemetry.

  2. Logic Engine: A non-blocking Python thread parses JSON packets.

  3. Backend Dispatcher: Choose between cpu (standard Python) or warp (NVIDIA Warp) processing.

  4. Warp Kernel (Optional): If enabled, NED-to-USD coordinate transforms and slip angles are computed on the GPU.

  5. Fabric Writer: Injects pose data directly into the USD Fabric layer for sub-millisecond updates to the Prim (e.g., /World/Animal).

Implementation Details

ZMQ Subscription

To prevent UI “hitching,” the subscriber runs in a separate background thread.

  • Mechanism: ZMQ SUB socket in dedicated thread.

  • Port: Configurable (Default: 5555).

  • Topic: Subscribes to all topics (empty filter) for simplicity.

Dynamic Asset Loading

The extension supports dynamic loading of different animal assets based on command-line arguments. This allows the same extension code to drive a Shark, Swordfish, or other marine animals without code changes.

  • Base Scene: omniverse/assets/ocean_scene.usda (Empty environment).

  • Asset Injection: The extension reads the –/biologger/animal argument and references the corresponding USD/GLB file into the stage at runtime.

Supported Assets:

  • shark -> great_white_shark.glb

  • swordfish -> swordfish.usd

  • whaleshark -> whale_shark.usd

Payload Structure

The extension expects a ZMQ packet with the following structure:

{
  "timestamp": 1660358220.019,
  "rotation": {
    "euler_deg": [8.34, 35.24, -179.87],  // [Roll, Pitch, Heading] in degrees
    "order": "zyx"                         // Extrinsic rotation order
  },
  "physics": {
    "accel_dynamic": [-3.79, -3.84, -2.07],  // Body-frame acceleration [x, y, z]
    "vedba": 5.78,                           // Vectorial Dynamic Body Acceleration
    "odba": 9.70,                            // Overall Dynamic Body Acceleration
    "depth": 4.59,                           // Depth in meters
    "velocity": 0.0,                         // Speed (m/s)
    "vertical_velocity": 0.01,               // Vertical velocity (m/s)
    "pseudo_x": -0.062,                      // Dead-reckoned X position
    "pseudo_y": -0.0001                      // Dead-reckoned Y position
  }
}

Note on Rotation: Euler angles are expected in degrees with ZYX extrinsic order (standard aerospace convention). The extension converts to quaternions internally.

Deployment Considerations

Streaming (Virtual Ocean)

To support the “Virtual Ocean” use case, the extension is designed to be compatible with Omniverse Kit Streaming.

  • Headless Operation: The extension does not rely on local GUI windows for critical functionality.

  • WebRTC Support: UI overlays are rendered as part of the viewport stream.

Verification

To validate the pipeline:

  1. Launch the Omniverse App with the extension enabled.

  2. Run the simulation with a configuration file:

    python -m biologger_sim --config config/Swordfish-RED001_20220812_19A0564-causal.yaml

  3. Success Metric: The animal model in the viewport should move smoothly with slip angles <15° during straight swimming.

Slip Angle Diagnostics

The extension computes a slip angle metric to validate alignment between the animal’s heading direction and its actual velocity vector. This is critical for confirming that coordinate system conversions are correct.

Definition:

\[\theta_{slip} = \arccos(\hat{v} \cdot \hat{h})\]

Where:

  • \(\hat{v}\) = Normalized velocity vector (computed from consecutive position deltas)

  • \(\hat{h}\) = Normalized heading vector (forward direction of the mesh in world space)

Expected Values:

  • Swimming straight: Slip angle < 15° (typically 5-10°)

  • Turning maneuver: Slip angle may spike to 30-45°

  • Coordinate bug: Slip angle consistently > 80° indicates a sign error or axis mismatch

Implementation Details:

The heading vector is computed by transforming the mesh’s local forward direction (0, 0, -1) by the telemetry quaternion:

fwd = rot_quat.Transform(Gf.Vec3f(0, 0, -1))

This works because the USD xform op order ensures telemetry is applied in world space:

v_world = telemetry × spawn × v_local

Since spawn × (0, 1, 0) = (0, 0, -1), we have:

telemetry × spawn × (0, 1, 0) = telemetry × (0, 0, -1)

Diagnostic Logging:

The extension logs slip angle data to omniverse-logs/<session>/slip_log.csv with columns:

  • Timestamp: Simulation time

  • SlipAngle: Angle in degrees

  • Speed: Movement distance between frames

  • Vx, Vy, Vz: Normalized velocity vector components

  • Hx, Hy, Hz: Normalized heading vector components

  • NED_Heading: Raw NED compass heading (before negation)

GPU Acceleration (NVIDIA Warp)

To support high-performance scaling, heavy computational tasks are offloaded to the GPU using specialized Warp kernels.

NED to USD Coordinate Transform

The transform_ned_to_usd_kernel performs vectorized re-projection:

@wp.kernel
def transform_ned_to_usd_kernel(
    in_pos: wp.array(dtype=wp.vec3),
    in_quat: wp.array(dtype=wp.vec4),
    out_pos: wp.array(dtype=wp.vec3),
    out_quat: wp.array(dtype=wp.vec4),
    out_slip: wp.array(dtype=float)
):
    # Implementation of NED to USD Y-Up re-projection

This ensures that even with thousands of entities, coordinate conversion does not bottleneck the main application thread.

Coordinate System Reference

The following table summarizes the coordinate conventions used throughout the system:

Coordinate System Conventions

System

Forward

Up

Heading Convention

NED (Marine)

+X (North)

-Z (Up from Down)

Clockwise (0°=N, 90°=E)

USD Y-Up

-Z

+Y

Counter-Clockwise (Right-Hand Rule)

Shark GLB (Native)

+Y

+Z

N/A (requires spawn rotation)

Conversion Summary:

NED Heading → USD Heading:  h_usd = -h_ned
NED Depth   → USD Y:        y_usd = -depth_ned
NED X (North) → USD Z:      z_usd = -x_ned
NED Y (East)  → USD X:      x_usd = y_ned