From FP32 to INT8 on the Edge: A Production Pipeline for Quantization-Aware Vision Models

Almost every edge AI project hits the same wall. The model works beautifully on a workstation GPU in FP32, then gets exported to the target device and either runs too slowly or, after quantization, quietly loses accuracy on exactly the cases that matter. The gap between those two outcomes is not luck — it is a pipeline. This post walks through how we think about taking a vision model from FP32 training to a validated INT8 deployment on edge accelerators.

Why INT8, and what actually changes

Most edge NPUs, DSPs and edge GPUs deliver their peak throughput on integer arithmetic. Moving from 32-bit floats to 8-bit integers reduces weight memory roughly fourfold, cuts memory bandwidth (often the real bottleneck on edge SoCs) and unlocks dedicated integer matrix units. The cost is representational: every tensor is now mapped onto 256 levels.

Affine quantization maps a real value x to an integer q:

q = clamp(round(x / scale) + zero_point, qmin, qmax)
x_hat = scale * (q - zero_point)

The error x - x_hat is the quantization noise. The entire discipline is about choosing scale and zero_point — per tensor, per channel, per layer — so that this noise lands where the network is least sensitive.

Design decisions that matter

Symmetric vs asymmetric

Weights are usually roughly zero-centred, so symmetric quantization (zero_point = 0) is cheap and effective. Activations after ReLU are non-negative and often skewed, so asymmetric quantization uses the range better. Many accelerators constrain this choice, so the hardware's supported modes should be known before training starts — not discovered at export.

Per-tensor vs per-channel

Convolution filters can have wildly different weight ranges. A single per-tensor scale lets one large-range channel crush the resolution of all others. Per-channel weight quantization is the single highest-leverage default for CNN-style vision backbones, and is supported by mainstream runtimes.

Calibration strategy

Activation ranges must be estimated from data. Naive min/max calibration is sensitive to outliers; a single bright pixel region can waste most of the range. Alternatives include percentile clipping, entropy (KL-divergence) calibration and MSE-minimising range search. The calibration set should mirror deployment conditions — lighting, sensors, compression artefacts — not just a random slice of the training set.

PTQ first, QAT when it earns its cost

Post-training quantization (PTQ) is fast: freeze the model, calibrate, export. For many classification and detection backbones it is enough. When it is not, the failure is usually concentrated in a few layers: depthwise convolutions, the first and last layers, attention softmax inputs, or detection heads regressing fine-grained coordinates.

Quantization-aware training (QAT) inserts fake-quantization nodes during fine-tuning so the network learns weights that are robust to rounding. Gradients pass through the non-differentiable rounding via the straight-through estimator. A practical QAT recipe:

1. Start from the converged FP32 checkpoint — never QAT from scratch.
2. Fold batch-norm into preceding convolutions before inserting quantizers, mirroring what the deployment compiler will do.
3. Calibrate initial ranges with PTQ, then fine-tune for a small fraction of the original schedule at a reduced learning rate.
4. Freeze BN statistics and observer ranges for the final epochs to stabilise.
5. Keep known-sensitive layers in higher precision if the target supports mixed precision.

Finding the layers that hurt: sensitivity analysis

Rather than guessing, we quantize one layer at a time (everything else in FP32) and measure the task metric drop. The resulting sensitivity profile drives the mixed-precision plan: a handful of layers in FP16 frequently recovers most of the lost accuracy at a small latency cost. Complementary diagnostics include per-layer signal-to-quantization-noise ratio and comparing intermediate activations between FP32 and INT8 graphs to localise divergence.

The compiler is part of the model

The graph that runs on the device is not the graph you trained. Edge compilers (TensorRT, TVM, ONNX Runtime execution providers, TFLite delegates, vendor NPU SDKs) perform:

• Operator fusion — conv + BN + activation collapse into one kernel, changing where requantization happens.
• Layout transforms — NCHW to NHWC or vendor-blocked layouts.
• Kernel selection and fallback — unsupported ops silently fall back to CPU, creating costly device-host round trips.
• Precision promotion — some ops are quietly run in higher precision, which helps accuracy but hurts the latency you budgeted.

We treat the compiler report as a first-class artefact: which ops fused, which fell back, which precision each layer actually ran in. A model is only "INT8" once that report says so.

Validation on the device, not the laptop

Simulated INT8 accuracy in a training framework is an estimate. Final acceptance must be measured on the target hardware, with the real runtime and the real pre-processing path. Our acceptance harness checks:

• Task metrics on a held-out, deployment-representative set — sliced by condition (night, glare, occlusion, distance) because averages hide regressions.
• Numerical parity — output agreement between FP32 reference and on-device INT8 outputs, flagging per-sample outliers.
• Latency distribution — p50 and p99 under realistic concurrency and thermal steady state, not a cold single-shot benchmark.
• Power and thermals — sustained throughput after the device has heat-soaked, since throttling changes everything.
• Pre-processing equivalence — resize, colour conversion and normalisation must be bit-identical to training; this is one of the most common silent bugs.

Failure modes we watch for

1. Outlier activations blowing up ranges — addressed with clipping, cross-layer equalisation or architecture tweaks (e.g. bounded activations).
2. Detection-head degradation — box regression and small-object recall are often the first casualties; keep heads in higher precision if needed.
3. Calibration data drift — a model calibrated on daytime data that deploys at night.
4. Silent CPU fallbacks that wreck latency budgets.
5. Version skew between the compiler used in CI and the runtime on devices in the field.

How HOLORAI approaches this

We treat quantization as a reproducible pipeline, not a one-off export:

• Hardware-first constraints. We pin the target accelerator, runtime version and supported quantization modes before architecture selection, so the model is designed to quantize well.
• Automated PTQ → sensitivity → QAT ladder. CI runs PTQ on every candidate model, generates a per-layer sensitivity report and escalates to QAT or mixed precision only when acceptance criteria fail.
• Compiled-artefact versioning. The engine file, compiler report, calibration set hash and runtime version are versioned together and signed for OTA rollout.
• Hardware-in-the-loop gates. Device farms run the acceptance harness on every release candidate; nothing ships on simulated numbers.
• Field feedback. Low-confidence and disagreement samples from deployed devices flow back to recalibration and retraining sets.

An edge model is the combination of weights, quantization parameters, compiler version and pre-processing code. Version and validate them as one unit.

If your model looks great in FP32 but struggles on device, the answer is rarely a bigger accelerator — it is usually a better quantization pipeline.