Hiding Telemetry Inside a Video Stream

29 October 2026 · 3 min

The timestamp rides inside the video, not beside it.

A drone or robot camera rarely sends video alone. You also want to know when each frame was captured, where the vehicle was, what the sensors read. The usual approach is a second channel next to the video, and then a fight to line the two up at the other end.

The video-metadata repository from our lab at RBCCPS, IISc tries the other approach: put the data inside the video stream itself. Its README is empty, so this post walks through what the code (one main.cpp and one header, 2021, version 0.1) actually does.

The trick: SEI messages

An H.264 stream is a sequence of NAL units, each starting with the bytes 00 00 00 01. Most carry picture data, but type 6 is SEI, Supplemental Enhancement Information. Decoders that don't care about an SEI message simply skip it, so you can add one without breaking playback in any normal player.

One SEI flavour, user data unregistered (payload type 5), exists for exactly this: a 16-byte UUID followed by whatever bytes you like.

Step 1: Build the SEI unit

h264_sei_pack.h builds the NAL unit by hand:

the start code 00 00 00 01;
the NAL type byte 6;
the payload type, then the payload size, each written in H.264's variable-length style (a 255 byte for every full 255, then the remainder);
the payload bytes;
the trailing byte 0x80.

The payload is a fixed UUID plus a text string. Here is the detail worth knowing: the code formats that string to look like the banner the x264 encoder writes itself, x264 - core 148 - ... options: %s, and puts the telemetry where the options normally go. There is also a second mode with a self-defined payload type (100) and the bare string.

Step 2: Capture and encode

main.cpp ties it together with OpenCV and FFmpeg:

open the camera through V4L2 at 640 by 480;
convert each frame from OpenCV's BGR to YUV 4:2:0 with sws_scale;
encode with libx264 at 30 fps, a GOP of 10, roughly 2 Mbit/s, preset slow;
run for 91 frames, about three seconds.

Step 3: Stamp every packet

Each time the encoder hands back a packet, the code reads clock_gettime(CLOCK_REALTIME), turns it into microseconds since the epoch, packs that number into an SEI unit, and writes the SEI followed by the packet. So every compressed packet in the output is preceded by the wall-clock time it left the encoder.

timestamp = get_microseconds();
h264_sei_pack((uint8_t*)sei, &len, timestamp, true);
fwrite(sei, 1, len, stdout);
fwrite(pkt->data, 1, pkt->size, stdout);

On the receiving side you scan for NAL type 6, check the UUID, and read the number back. Compare it with your own clock and you have per-packet latency; replace the timestamp with GPS or IMU values and you have telemetry locked to the picture.

What it is, and what it isn't

This is a small, honest prototype, and it shows:

Only the sender exists. There is no decoder or parser in the repo; reading the SEI back is left to you (FFmpeg and most stream analysers will show the raw SEI).
It writes to stdout. It also opens test.h264, but only the end-of-sequence code ends up there. Pipe stdout to a file or a streamer to keep the video.
The stamp is encoder time, not capture time, and with B-frames enabled packets come out in a different order from capture, so treat it as a send timestamp.
Only a timestamp is sent, not position or sensor data yet.
Some settings need a tidy-up: the minimum rate (2.5 Mbit/s) is set above the maximum (2 Mbit/s), and it uses FFmpeg calls (av_register_all, avpicture_fill) that newer FFmpeg versions have removed.

The idea is still a good one to copy. Because the metadata is part of the stream, it survives whatever carries the stream, a file, RTP, a recorder, without a second channel to keep in sync. The full source is on GitHub.