Inference Latency is the elapsed time between an input being submitted to a trained AI or machine learning model and the corresponding output being returned to the requesting application, measured end-to-end and including model computation time plus, for remotely served models, network transmission and queuing time.

Key Points

Inference latency measures a specific application of the broader latency concept — the delay between a model input and its output — and is composed of both compute time and, where relevant, network transit time.
For edge-inference deployments, inference latency is dominated by local compute time, since no network round trip is involved; for model-serving deployments, network round-trip time is added directly to total inference latency.
Inference latency is the primary metric used to determine whether a large language model or generative AI application can be centrally served or must be deployed at the edge in reduced form.
Acceptable inference latency thresholds vary sharply by application: a batch reporting tool can tolerate seconds of latency, while a real-time computer vision safety system requires latency in the low tens of milliseconds.
Inference latency should be measured end-to-end from the requesting application's perspective, not just at the model or serving infrastructure, since network conditions between the two can dominate the total figure for cloud-served models.

Concept

Inference latency is a composite figure. For a model running via edge inference, the dominant component is the local compute time required to process input data through the model's layers — there is no network component at all. For a model accessed through model serving, inference latency additionally includes the round-trip network time to reach the serving endpoint, any queuing delay if the serving infrastructure is under load, and the return transmission time for the result. This is why the same model can exhibit very different effective inference latency depending on deployment pattern and network path — a model served from a nearby edge cloud location will typically show lower inference latency than the same model served from a distant hyperscale region, independent of the model's own computational demands.

Explainer

For ConnectedEarth's audience, inference latency is the metric that operationalizes the edge-inference-versus-model-serving decision covered elsewhere in this knowledge base. A generative AI or large language model application intended for real-time interactive use in a remote or bandwidth-constrained environment will show unacceptable inference latency if served over a high-latency satellite backhaul link, making a smaller, locally-deployed model via edge inference the only viable option despite its reduced capability. Conversely, a batch-oriented application with looser latency tolerance can accept the round-trip cost of model serving in exchange for access to a larger, more capable centrally hosted model. Measuring and reporting inference latency in the context of the actual network path involved — not just the model's isolated compute benchmark — is essential to accurately assessing whether a given AI deployment will meet its operational requirement.