Multi-stream video analytics on edge GPUs: frame budgets, zero-copy decode and fused TensorRT engines

Most edge video analytics demos show one camera, one model and a smooth overlay. Production systems are judged on a different number: how many streams a single device can sustain at a defined latency and accuracy, for weeks, inside a hot enclosure. That number is set less by the detector than by everything around it: decode, colour conversion, memory movement, batching, scheduling and the inference engine build. This post walks through how we design multi-stream pipelines on edge GPUs (Jetson-class modules and small discrete GPUs), where the time goes, what breaks, and how to benchmark them so the result means something.

Start from a frame budget, not a model

Before choosing a network, write down the arithmetic. Sixteen cameras at 15 fps is 240 frames per second; if every frame goes through the detector, the GPU has on average about 4.2 ms per frame for everything it touches. That budget is shared by preprocessing, inference, post-processing and any secondary models, and it shrinks when the device thermally throttles. Doing this sum first tells you immediately whether you need a smaller model, a lower inference rate, region-of-interest cropping or a bigger device. It also turns the vague goal of "real time" into testable service-level objectives: maximum end-to-end latency at p99, maximum frame-drop rate, and minimum accuracy relative to a reference.

Anatomy of the pipeline

A sustainable pipeline keeps pixels on the accelerator from the moment they are decoded until only metadata leaves. The stages are:

1. Ingest. RTSP/RTP sessions per camera, with a jitter buffer, timestamp normalisation and reconnect logic. This is CPU work and should never touch decoded pixels.
2. Hardware decode. H.264/H.265 bitstreams go to the dedicated video decoder (NVDEC on NVIDIA parts). Decoder sessions and output surfaces are finite resources; size the surface pool explicitly.
3. Convert and scale on the GPU. NV12 to planar RGB, resize/letterbox and normalisation run as GPU kernels (or on a dedicated scaler such as the VIC on Jetson). A CPU round trip here is the most common silent throughput killer.
4. Batch formation. Frames from many streams are gathered into a batch tensor for the inference engine.
5. Inference. A compiled engine (TensorRT, or ONNX Runtime with a TensorRT/CUDA execution provider) with fused layers and reduced precision.
6. Post-processing and tracking. Decoding boxes, NMS, multi-object tracking per stream, and any secondary classifiers on tracked objects.
7. Egress. Compact events and metadata over MQTT or gRPC; optional short clips encoded with the hardware encoder on trigger.

On Jetson-class devices the CPU and GPU share physical DRAM, so "zero-copy" means passing buffer handles (for example NVMM/dmabuf surfaces) rather than calling cudaMemcpy. On discrete GPUs the goal is the same: decode into device memory and keep it there. Every unnecessary copy costs memory bandwidth, which on embedded modules is usually the binding constraint before raw compute.

Batching across streams

Batching is where throughput is won and latency is lost. Larger batches improve GPU utilisation, but a frame waiting for a batch to fill is a frame getting older. We use a time-bounded batch former: emit a batch when it reaches the engine's optimal size or when the oldest frame has waited longer than a deadline, whichever comes first.

loop:
  wait until queue.size >= max_batch or now - queue.oldest_ts >= max_wait_ms
  batch = take_round_robin(queue, max_batch)   # fairness across streams
  if batch.size < engine.opt_batch: pad or use a smaller profile
  submit(batch, cuda_stream[next])             # double/triple buffered

Three details matter. First, fairness: pull round-robin across streams so a high-motion camera cannot starve others. Second, engine profiles: a dynamic-shape engine built with optimisation profiles for, say, batch 1, 4 and 8 avoids wasting work on padding when traffic is light. Third, overlap: use multiple CUDA streams so preprocessing of batch n+1 overlaps inference of batch n and post-processing of batch n-1.

The compiler stack and operator fusion

The inference engine build is where most of the per-frame speed comes from, and where many accuracy regressions are introduced.

Fusion

Engine builders fuse convolution, batch-norm and activation into single kernels, merge element-wise ops and choose tactics (kernel implementations) by timing them on the target GPU. We push this further by moving preprocessing into the graph: mean/std normalisation and channel reordering become a cheap first layer instead of a separate pass over memory. Box decoding and NMS run as an engine plugin (for example an efficient-NMS plugin) so candidate boxes never leave the GPU.

Precision

FP16 is usually close to free on accuracy. INT8 needs care. Post-training quantisation with a calibration set works when the calibration data matches deployment conditions: night scenes, rain, glare, the actual camera models and compression levels. When it does not, we use quantisation-aware training and export explicit Q/DQ nodes so the engine honours the scales learned during training rather than inventing its own. Sensitive layers, often the first convolution and detection heads, can be kept at FP16 in a mixed-precision engine.

Portability

Compiled engines are specific to the GPU architecture and the inference library version. They are build artefacts, not source. We treat the ONNX graph as the portable artefact and build engines per target in CI, recording the toolkit versions, the build flags and a hash of the calibration cache alongside the engine.

Doing less work per frame

The cheapest inference is the one you skip. Techniques that compound:

• Inference stride with tracking. Run the detector every k-th frame and let a tracker propagate boxes in between. The right k depends on object speed and the event you care about; measure missed events, not just mAP.
• Motion or change gating. A cheap frame-difference or encoder motion-vector check skips static scenes, which on many industrial cameras is most of the day.
• ROI cropping. Infer only on configured regions at a higher effective resolution, instead of downscaling a full 4K frame and losing small objects.
• Per-track secondary models. Run attribute or classification models once per new track (or on a schedule), not once per detection per frame.
• Cascades to the cloud. Escalate only low-confidence or high-value crops to a larger model off-device, with a strict bandwidth budget.

Failure modes we design for

• Latency creep. If any stage is slower than arrival rate, queues grow and latency rises without bound while throughput looks fine. Every queue needs a bound and an explicit drop policy (drop oldest, keep the newest frame per stream).
• Decoder surface exhaustion. Downstream stalls hold surfaces; the decoder blocks; RTSP buffers overflow; streams disconnect. Surface pools must be sized for worst-case in-flight frames.
• Reconnect storms. After a network blip, every camera reconnecting at once spikes CPU and decoder initialisation. Use jittered exponential backoff per stream.
• Silent preprocessing mismatch. BGR vs RGB, limited vs full-range YUV, or a letterbox that differs from training all reduce accuracy without any error. Only a parity test catches these.
• Thermal throttling. Sustained load in an enclosure triggers clock reduction hours into a run. A benchmark that lasts minutes will not see it.
• Timestamp drift. Camera clocks and RTP timestamps disagree; event correlation across cameras breaks. Normalise to a monotonic device clock at ingest and keep the source timestamp as metadata.
• Memory fragmentation and leaks. Long-running GStreamer or custom pipelines leak buffers slowly. Soak tests and per-process memory telemetry are mandatory.

Benchmarking methodology

A single "frames per second" figure hides almost everything that matters. The method we use:

1. Reproducible input. Record representative streams (real codecs, bitrates, GOP structures, resolutions) and replay them from a local RTSP server so every run sees identical input.
2. Fixed device state. Pin the power mode and clocks, record the firmware and library versions, and log temperatures throughout.
3. Warm-up, then steady state. Discard the warm-up window; measure over a long steady-state window, and separately run a multi-hour thermal soak inside the target enclosure or an equivalent ambient.
4. Per-stage timing. Instrument decode, preprocessing, queue wait, inference and post-processing with GPU events, not just wall-clock around the whole pipeline. Report p50, p95 and p99 per stage and end-to-end (glass-to-event).
5. Sweep stream count. Increase the number of streams until an SLO breaks. The headline result is "maximum streams at SLO", reported with the latency distribution and frame-drop rate at that point.
6. Accuracy parity. Dump a fixed set of decoded frames from the device pipeline and compare outputs of the deployed engine against an FP32 reference run on exactly those frames. This isolates quantisation and preprocessing error from dataset error.

How HOLORAI approaches this

Our engagements on multi-stream edge video follow a consistent engineering path:

• Capacity model first. We build the frame-budget spreadsheet from camera count, frame rates, resolutions and event requirements, and use it to shortlist hardware and model families before any training.
• GPU-resident pipeline. We design for hardware decode, on-GPU conversion and zero-copy buffer passing from the start, with bounded queues and explicit drop policies at every stage.
• Engines as CI artefacts. ONNX is the source of truth; TensorRT or ONNX Runtime engines are built per target device in CI, with calibration data versioned and quantisation choices (PTQ, QAT, mixed precision) decided by parity tests rather than by default.
• Golden-frame parity gates. Every release runs the deployed engine against a reference on fixed frames; a regression beyond an agreed tolerance blocks rollout.
• Soak before sign-off. We run multi-hour, full-load tests in realistic thermal conditions and report maximum streams at SLO, not peak throughput.
• Operable in the field. Devices publish per-stage latency, queue depth, drop counts, temperatures and clock states, so a fleet operator can tell a model problem from a network or thermal problem without a site visit.

The outcome we aim for is not the fastest demo; it is a pipeline whose capacity, latency and accuracy are known, measured and stable enough to plan a deployment around.