OTA model rollout for edge device fleets: signed artifacts, A/B slots, shadow evaluation and automatic rollback

Training a better model is half the job. The other half is getting it onto a fleet of cameras, gateways and embedded GPUs spread across sites with patchy connectivity, mixed hardware revisions and no one standing next to them. Once a model is on a device, it drives alarms, actuators and operator decisions. A bad update is not a failed deploy. It is a fleet-wide regression that may stay hidden for days because nothing crashes and the outputs just get quietly worse.

This post covers the engineering behind over-the-air (OTA) model rollout for edge fleets: what the release artifact should contain, how devices swap models atomically, how to stage a rollout when you have no ground-truth labels at inference time, and how to make rollback cheaper and more reliable than rollout.

Why a model update is not an app update

Classic firmware and app OTA has known answers: signed images, dual partitions, health checks. Models break those assumptions in three ways.

• Success is statistical, not binary. A new detector can boot, hit its latency budget and pass every health check while recall on night-time footage drops sharply. "Process is up" tells you almost nothing about whether the model is doing its job.
• Artifacts depend on the hardware. A TensorRT engine is tied to the GPU architecture and the TensorRT/driver versions it was built for. An INT8 TFLite model built for one NPU delegate can fall back to the CPU on another. So "the model" is really a matrix of compiled artifacts, one per hardware and runtime target.
• Behaviour depends on the environment. The same weights behave differently at a loading dock and in a cold store. Per-site differences in lighting, camera angle and sensor wear mean a rollout that looks fine in aggregate can be failing at specific sites.

The release artifact: what actually ships

We treat a model release as a signed, content-addressed bundle described by a manifest. The manifest is the single source of truth for what a device may load and under which conditions.

bundle: detector-v14
model_semver: 14.2.0
targets:
  - hw: orin-nx-16gb
    runtime: tensorrt-10.x
    artifact: sha256:9f1c...   # serialized engine
  - hw: x86-openvino
    runtime: openvino-2025.x
    artifact: sha256:41ab...
preprocess: { resize: letterbox-640, norm: imagenet, color: rgb, version: 3 }
postprocess: { nms_iou: 0.55, thresholds_ref: sha256:77de... }
golden_vectors: sha256:c0e2...
min_agent_version: 5.3.0
output_schema: detections.v4
anti_rollback_counter: 212
signatures: [release-key-1, release-key-2]

Several design decisions are packed into that file:

• Preprocessing is versioned and travels with the model. Mismatched resize, normalisation or channel order is one of the most common silent failures we see. Putting the preprocessing spec in the manifest stops a model trained on letterboxed RGB from being fed stretched BGR frames.
• Thresholds are a separate, addressable artifact. Per-site confidence thresholds change more often than weights. Separating them lets you re-tune a site without rebuilding engines, while keeping thresholds versioned and covered by the same signatures.
• Golden vectors. Each bundle carries a small set of fixed inputs plus reference outputs produced in CI on the same target build. After installing, the device runs them and compares outputs within a tolerance before it activates the model. This catches broken compilation, delegate fallback, quantisation drift and preprocessing errors on the actual device.
• The output schema is declared. Downstream services must know which output contract they are getting, because during a rollout the fleet is mixed by definition.

Full versus delta transfer

Binary diffs (bsdiff, or zstd with the previous artifact as a dictionary) work well for firmware but are less predictable for models. A recompiled engine can differ almost everywhere even when the weights barely changed. In practice, chunked, content-addressed transfer does better: split artifacts into fixed-size chunks named by hash, download only chunks the device doesn't already have, and resume after a dropped link without starting over. For bandwidth-constrained sites (cellular backhaul, satellite), schedule downloads into off-peak windows and pre-stage bundles well before the activation date.

On-device mechanics: A/B slots and atomic activation

The device agent keeps at least two model slots. The active slot is never modified in place. An update moves through explicit states:

1. Fetch and verify. Download into the inactive slot, then check chunk hashes, the manifest signature and the anti-rollback counter.
2. Prepare. Build or deserialise the engine for the local accelerator, warm up and allocate memory. If the engine is built on-device, cache it by hash so a reboot doesn't trigger a rebuild.
3. Self-test. Run the golden vectors, then a short soak on live frames with no outputs exposed: check latency percentiles, memory headroom and thermal behaviour.
4. Activate. Atomically switch a pointer (a symlink rename or a single-record write with fsync) and hand the inference process the new slot. Mark the activation as tentative.
5. Commit or revert. If the health and output checks hold for the bake window, mark the slot as committed. If the process crashes, a watchdog fires, or the device reboots while still tentative, revert to the previous slot automatically, with no call to the server.

The last step matters most. Rollback must work when the device can't reach the control plane, because a bad model that saturates the CPU or crashes the network stack may be exactly what cuts it off.

Staged rollout without ground truth

The control plane moves a release through rings: lab devices, then a shadow cohort, then canaries, then widening percentages. The key decision is how each ring is cut. Random percentages hide problems that only show up in specific places. We stratify cohorts by hardware SKU, runtime version, site and environment class, so every meaningful combination is in the early rings. Bake time is measured in inferences observed per stratum, not wall-clock hours. A quiet site at night takes longer to produce evidence than a busy one at noon.

Shadow mode

In shadow mode, the candidate runs alongside the incumbent on the same frames. Only the incumbent's outputs reach downstream systems. On compute-limited devices, the candidate runs on a sampled subset of frames at lower scheduling priority, so the production pipeline keeps its frame budget. Shadow gives you the most useful label-free signal available: paired disagreement between incumbent and candidate on the same inputs.

Gating signals

• System health: crash and restart rate, p50/p99 latency, dropped frames, memory high-water mark, accelerator utilisation, thermal throttling events and power draw.
• Output distribution: detections per frame, class frequencies and confidence histograms, compared with the incumbent's using population stability index or two-sample tests per stratum.
• Paired agreement: from shadow, the share of frames where the models agree within an IoU and class tolerance, broken down by class and condition. Disagreements are sampled and sent for human labelling, which turns the rollout into a targeted labelling campaign.
• Input health: brightness, blur, occlusion and embedding statistics on the inputs. This is how you tell "the camera was knocked out of alignment" apart from "the model regressed". Otherwise both look like an output shift.

Watching dashboards continuously and stopping when something looks off inflates false alarms. Treating a run as clean because the global average is flat hides regressions in one stratum. We use sequential tests with always-valid confidence bounds, set a minimum sample count per stratum, and evaluate each stratum on its own before aggregating, so a regression on one hardware SKU can't be cancelled out by an improvement on another.

Failure modes worth designing for

• Power loss mid-write. Solved by slot isolation plus atomic pointer swaps. Test it by pulling power during staging in the lab.
• Anti-rollback versus model rollback. Security anti-rollback stops attackers from downgrading to a vulnerable runtime. Model rollback needs to go backwards. Keep them as separate counters: revoke runtimes through security policy, and roll back models within the allowed window.
• Clock skew. Devices with dead RTC batteries can't be trusted to check signature expiry. Use monotonic version counters and server-issued freshness metadata rather than local wall time.
• Thundering herd. Thousands of devices fetching at once overloads site uplinks. Add jitter to fetch schedules and use a peer or gateway cache at large sites.
• Long-tail version skew. Some devices stay offline for weeks. Back-end consumers must accept at least the last two output schemas, and the control plane must be able to bring a stale device straight to the current release.
• Compatibility matrix explosion. Every new hardware SKU multiplies build and test cost. Keep the supported matrix explicit and fail closed: a device with no matching target keeps its current model and reports why.

Securing the update path

The update channel can push code-like artifacts to every device you own, so it is a high-value target. We follow The Update Framework (TUF) pattern of separated signing roles: an offline root key, threshold signatures on releases, and short-lived timestamp metadata to defend against freeze and replay attacks. Secure boot verifies the agent and runtime that verify the model. Where model IP matters, artifacts are encrypted at rest with device-bound keys, and devices report attested slot state so the control plane knows what is actually running, not just what it sent.

How HOLORAI approaches this

When we build an edge AI system, the release path is part of the system, not something added after the pilot. Our approach usually looks like this:

1. The manifest comes first. We define the bundle format, preprocessing spec, output schema and supported hardware matrix before the first model ships, so every release from day one is reproducible and verifiable.
2. CI produces per-target artifacts and golden vectors. Each target is compiled in a pinned toolchain container, golden outputs are generated on the matching hardware, and accuracy is re-checked after compilation and quantisation, not only on the training-framework model.
3. A lightweight device agent owns the state machine. Fetch, verify, prepare, self-test, activate, commit or revert. It is tested with fault injection: power cuts, network partitions, full disks and corrupted chunks.
4. Rollout policies are code. Strata, ring sizes, evidence thresholds and abort conditions live in version-controlled policy files that get reviewed like any other change, and every promotion decision is logged with the evidence behind it.
5. Disagreements feed the next model. Shadow disagreements and abort events go into a labelling queue, which connects the rollout to the data flywheel.

The goal is uneventful releases: a fleet that can take a new model every week, where any regression shows up in a small, well-instrumented cohort and gets reverted automatically before an operator notices.