Video recorder

Video recording allows you to store color, depth, skeleton, IMU, and floor data. You can play the captured data using the Video Player module. The data are stored in binary form. The video recorder is creating an internal queue where it’s processing the frames. The processing time may extend after a video recording session is stopped.

The LightBuzz SDK is recording lots of different data types (color, depth, skeleton). Typical MP4, MOV, or AVI files can only store color information. As a result, standard video formats are not enough when it comes to skeleton tracking. So, our team has implemented a simple alternative protocol to store all necessary data.

In terms of performance, the recording procedure is not impacting the live feed. The recorder uses parallel background threads, so your app keeps running smoothly.

Each video is stored in a separate system folder. All you need to do is provide a valid folder path where the recorder can store data. Here are the different file types that are stored internally.

Video settings

The video data are stored in a single folder on a per-frame basis. Every valid video folder includes two special files:

Never delete files! These configuration files are essential. If you delete them, the video player will not be able to play the videos.

Video data

The name of every single file is the unique timestamp of its corresponding frame. So, the color, depth, and body files of the same frame would have the exact same name. Each frame may include one or more of the following files:

Color

The software is encoding the one-dimensional RGBA array in JPEG. This way, we reduce the amount of space used, while keeping recording times short.

Depth

Depth data are stored in arrays of distance numbers (unsigned 16-bit integers). Each number represents the distance of a pixel in millimeters. The recorder is storing the depth values consecutively into binary files.

IMU

The IMU frame samples include two vectors and three rotation values.

• Acceleration vector (X, Y, Z)
• Gyroscope vector (X, Y, Z)
• Raw, pitch, yaw

The recorder is storing these values in plain text.

Body

The body data files include the ID of each skeleton and the joint information:

• Joint type
• Tracking confidence
• 3D position (X, Y, Z)
• Orientation (X, Y, Z, W)
• 2D position (X, Y)

The recorder is storing these values sequentially in plain text.

Floor

The floor data files include the two vectors which Mathematically describe a floor clip plane:

• Normal vector (X, Y, Z)
• Origin point (X, Y, Z)

The recorder is storing these values sequentially in plain text.

Example

Here is a basic C# example of recording a video:

using LightBuzz.BodyTracking.Video;
public class VideoRecordingDemo : MonoBehaviour
{
    private readonly VideoRecorder _recorder = new VideoRecorder();
    private readonly Sensor _sensor = Sensor.Create(new SensorConfiguration { SensorType = SensorType.Wecam });
    private void Start()
    {
        // Start the camera.
        _sensor.Start();
        // Initialize the recorder.
        _recorder.Settings = new VideoRecordingSettings
        {
            Path = "/Users/lightbuzz/videos/my_video",
            RecordColor = true,
            RecordDepth = true,
            RecordBody = true
            Smoothing = _sensor.Configuration.Smoothing,
            FrameRate = _sensor.FPS,
            ColorResolution = new Size(_sensor.Width, _sensor.Height),
            DepthResolution = new Size(_sensor.Width, _sensor.Height),
            ColorFormat = _sensor.ColorFormat
        };
        // Subscribe to the recorder events.
        _recorder.OnRecordingStarted += Recorder_OnRecordingStarted;
        _recorder.OnRecordingStopped += Recorder_OnRecordingStopped;
        _recorder.OnRecordingCanceled += Recorder_OnRecordingCanceled;
        _recorder.OnRecordingCompleted += Recorder_OnRecordingCompleted;
    }
    private void OnDestroy()
    {
        // Unsubscribe from the events and dispose the recorder.
        _recorder.OnRecordingStarted -= Recorder_OnRecordingStarted;
        _recorder.OnRecordingStopped -= Recorder_OnRecordingStopped;
        _recorder.OnRecordingCanceled -= Recorder_OnRecordingCanceled;
        _recorder.OnRecordingCompleted -= Recorder_OnRecordingCompleted;
        _recorder.Dispose();
        // Close and dispose the camera.
        _sensor.Close();
        _sensor.Dispose();
    }
    private void Update()
    {
        Frame = _sensor.Update();
        if (_recorder.IsRecording)
        {
            _recorder.Update(Frame);
        }
    }
    private void Recorder_OnRecordingStarted()
    {
        // Recording started.
    }
    private void Recorder_OnRecordingStopped()
    {
        // Recording stopped.
    }
    private void Recorder_OnRecordingCanceled()
    {
        // Recording canceled.
    }
    private void Recorder_OnRecordingCompleted()
    {
        // Recording completed.
        // IMPORTANT: recording has finished processing data.
    }
    // This is a custom action (e.g. a button click)
    // that starts and stops the recording.
    public void OnRecord_Click()
    {
        if (!_recorder.IsRecording)
        {
            _recorder.Start();
        }
        else
        {
            _recorder.Stop();
        }
    }
}

Methods

The video recorder includes four methods with straightforward functionality.

Start()

Starts recording frames from the camera.

Stop()

Stops recording camera frames. The video recorder may still be processing frames. Ensure you subscribe to the OnRecordingCompleted event to get notified when processing has finished.

Cancel()

Manually cancels the recording process. Does not delete any captured data.

Update(FrameData)

Feeds the video recorder with the specified camera frame data.

Dispose()

Clears unmanaged resources and closes any background threads. Call Dispose() when you no longer need the video recorder. In Unity, you should always call Dispose() when your video GameObject is disabled or removed from the hierarchy.

Properties

The video recorder provides the following properties:

IsRecording (read-only)

Checks whether the video recorder is currently recording frames.

IsSaving (read-only)

Checks whether the video recorder is currently saving frames to the disk.

Timestamps (read-only)

A list of every frame timestamp recorded. Each timestamp is a unique DateTime object.

FrameCount (read-only)

The number of captured frames.

Duration (read-only)

The duration of the video expressed as a TimeSpan structure.

Settings

The video recording options. Include:

• Path: the absolute path to the folder containing the video data files.
• RecordColor, Depth, Body, IMU, Floor: specifies whether the recorder should store color/depth/body/IMU/floor data, respectively.
• Smoothing: the smoothing value to apply.
• FrameRate: the frame rate of the camera.
• ColorResolution: the color resolution of the camera.
• DepthResolution: the depth resolution of the camera.
• ColorFormat: the color format of the camera.

Events

When recording a video, it’s recommended to subscribe to the following events:

OnRecordingStarted

Raised when the recorder starts recording data.

OnRecordingStopped

Raised when the recorder stops recording data. However, the recorder may still be processing data, even after the recording is stopped.

OnRecordingCompleted

Raised when the recorder finishes data processing. Subscribe to this event before starting video playback.

OnRecordingCanceled

Raised when the recording process is canceled manually.