inference latency

The real wall-clock time a model takes to run one forward pass on the hardware it will actually run on.

Inference latency is the real wall-clock time a model takes to run one forward pass on the hardware it will actually be deployed on. It is a genuinely different quantity from a model's FLOPs count (its number of floating-point operations, computed from the architecture alone), because latency also depends on memory-bandwidth bottlenecks and how well the target hardware's own kernels support each operation. Two models with identical FLOPs can measurably differ in real latency on the same device.

Examples

A model with a strict $15\text{ms}$ -per-frame budget is ruled out by any candidate backbone whose measured inference latency exceeds $15\text{ms}$ on the target device, regardless of its FLOPs count.