Eye control example

Used Rerun types

EyeControls3D, Transform3D, Points3D, Pinhole, EncodedImage, Image, LineStrips3D, Scalars, TextDocument

Background

This example demonstrates how to programmatically configure and control the 3D view camera using the Rerun Blueprint API. By defining camera states in Python, you can precisely tailor your workspace to highlight the most relevant aspects of your data.

In this example, we define several specialized perspectives:

• Top-down overview: a global scene perspective for general spatial awareness.
• Comparative close-up: A focused view designed to analyze trajectory deviations between different localization methods.
• 3rd-person follow: dynamic camera that tracks the ego vehicle as it moves through the environment.

Finally, we demonstrate how to control the camera at runtime, enabling the creation of cinematic visualizations or automated data storytelling for presentations and datasets.

Useful resources

Below you will find a collection of useful Rerun resources for this example:

• Blueprints
• Building blueprints programmatically

Run the code

To run this example, make sure you have the Pixi package manager installed.

KTH RPL (indoor handheld dataset)

pixi run kth_rpl

You can type:

pixi run kth_rpl -h

to see all available commands. For example, you can set the voxel size used for downsampling, where the dataset is located, and for how long to sleep in-between frames.

NTU VIRAL (drone dataset)

pixi run ntu_viral

Visit NTU VIRAL dataset to see the scene names available.

Explaination

KTH RPL

Within eye_control/kth_rpl.py, the get_blueprint function serves as the orchestrator for our scene layout. This is where we define the specific 3D viewpoints using the Spatial3DView and EyeControls3D classes.

Dynamic camera movement: inside view

The inside view is designed to be dynamic. Rather than a static observation point, we calculate the camera's orientation and look-at target programmatically.

qw = orientation[0]
qx = orientation[1]
qy = orientation[2]
qz = orientation[3]

look_dir = np.array([
    1 - 2 * (qy * qy + qz * qz),
    2 * (qx * qy + qw * qz),
    2 * (qx * qz - qw * qy)
], dtype=np.float32)

look_target = position + look_dir
position -= look_dir

eye_control = rrb.EyeControls3D(
    position=position,
    look_target=look_target,
    eye_up=(0.0, 0.0, 1.0),
    spin_speed=0.0,
    kind=rrb.Eye3DKind.FirstPerson,
    speed=10.0,
)

inside_view = rrb.Spatial3DView(
    origin="/",
    name="Inside View",
    eye_controls=eye_control,
    ...
)

By updating the position and look_target parameters at runtime, you can create "cinematic" traversals through a dataset. In this example, the camera begins in a third-person perspective and then transitions into a continuous loop along the trajectory.

Tip: You can extend this logic by using splines or interpolation to create even smoother, multi-point camera paths for high-quality video exports.

Static perspectives: top-down and outside views

For the top-down and outside views, instead of calculating frames dynamically, we set the position and look_target values such that they best showcase the relevant spatial data.

top_down_eye_control = rrb.EyeControls3D(
    position=(0.02071, 0.17345, 5.0),
    look_target=(0.01474, 0.25178, 0.00327),
    eye_up=(0.0, 0.0, 1.0),
    spin_speed=0.0,
    kind=rrb.Eye3DKind.FirstPerson,
    speed=10.0,
)

outside_eye_control = rrb.EyeControls3D(
    position=(-3.1584, -0.06133, 0.41640),
    look_target=(-2.2398, -0.00435, 0.02432),
    eye_up=(0.0, 0.0, 1.0),
    spin_speed=0.0,
    kind=rrb.Eye3DKind.FirstPerson,
    speed=10.0,
)

top_down_view = rrb.Spatial3DView(
    origin="/",
    name="Top View",
    eye_controls=top_down_eye_control,
    ...
)

outside_view = rrb.Spatial3DView(
    origin="/",
    name="Outside View",
    eye_controls=outside_eye_control,
    ...
)

These views are essential for maintaining context. While the inside view follows the action, these views remain fixed, providing a consistent global frame of reference for the entire scene.

NTU VIRAL

The NTU VIRAL example uses the same structure as the KTH RPL dataset. The code is located in eye_control/ntu_viral.py, specifically within the get_blueprint function. To ensure optimal visibility for each unique environment, we define scene-specific camera positions and look_at targets.

Follow view

The follow view tracks the drone as it traverses the scene. By setting the origin to the BODY_FRAME, the camera becomes relative to the drone's coordinate system.

In this setup, the EyeControls3D position is locked 8 meters behind and 3 meters above the drone.

Note: The z-coordinate is set to -3 because the eye_up vector is defined as negative in the Z-axis.

follow_eye_control = rrb.EyeControls3D(
    position=(-8, 0, -3),
    look_target=(0.0, 0.0, 0),
    eye_up=(0.0, 0.0, -1.0),
    spin_speed=0.0,
    kind=rrb.Eye3DKind.FirstPerson,
    speed=20.0,
)

follow_view = rrb.Spatial3DView(
    name="Follow View",
    eye_controls=follow_eye_control,
    ...
)

Top-down view

The top-down view provides a global overview of the environment. Unlike static datasets, the NTU VIRAL scenes vary significantly in scale and layout. Consequently, the camera parameters are pulled from the TOP_DOWN_VIEW_POINT dictionary based on the active scene.

top_down_eye_control = rrb.EyeControls3D(
    position=TOP_DOWN_VIEW_POINT[scene][0],
    look_target=TOP_DOWN_VIEW_POINT[scene][1],
    eye_up=(0.0, 0.0, 1.0),
    spin_speed=0.0,
    kind=rrb.Eye3DKind.FirstPerson,
    speed=20.0,
)

top_down_view = rrb.Spatial3DView(
    name="Top Down View",
    eye_controls=top_down_eye_control,
    ...
)

Close-up view

The close-up view is designed for detailed analysis. It focuses on specific segments of the flight path, making it easier to compare the precision and drift of various localization methods. These points of interest are also scene-specific and defined in the CLOSE_UP_VIEW_POINT configuration.

close_up_eye_control = rrb.EyeControls3D(
    position=CLOSE_UP_VIEW_POINT[scene][0],
    look_target=CLOSE_UP_VIEW_POINT[scene][1],
    eye_up=(0.0, 0.0, 1.0),
    spin_speed=0.0,
    kind=rrb.Eye3DKind.FirstPerson,
    speed=20.0,
)

close_up_view = rrb.Spatial3DView(
    name="Close-Up View",
    eye_controls=close_up_eye_control,
    ...
)

Acknowledgements

NTU VIRAL: A Visual-Inertial-Ranging-Lidar Dataset for Autonomous Aerial Vehicle

@article{nguyen2022ntu,
  title     = {NTU VIRAL: A Visual-Inertial-Ranging-Lidar Dataset, From an Aerial Vehicle Viewpoint},
  author    = {Nguyen, Thien-Minh and Yuan, Shenghai and Cao, Muqing and Lyu, Yang and Nguyen, Thien Hoang and Xie, Lihua},
  journal   = {The International Journal of Robotics Research},
  volume    = {41},
  number    = {3},
  pages     = {270--280},
  year      = {2022},
  publisher = {SAGE Publications Sage UK: London, England}
}