Model Serving is the practice of deploying a trained machine learning or AI model within a hosting infrastructure — typically cloud-based — and exposing it through a network-accessible interface, such as an API, that allows client applications to submit inference requests and receive results, usually accompanied by infrastructure for scaling, versioning, and monitoring the served model.

Key Points

Model serving hosts models centrally and exposes them via network-accessible interfaces, typically APIs, requiring a network round trip for every inference request.
Model serving infrastructure commonly includes auto-scaling to handle variable request load, allowing serving capacity to expand or contract with demand without manual intervention.
The network path between a client application and a model-serving endpoint is a direct component of the end-to-end inference latency experienced by the application, distinct from the model's own computation time.
Model serving is the default deployment pattern for large, resource-intensive models such as large language models, which are typically too large to run economically on edge or field hardware.
Model serving and edge inference are frequently combined in a single deployment: a large model is served centrally for less time-sensitive tasks, while a smaller, distilled version runs via edge inference for latency-critical tasks at the same site.

Concept

A model-serving deployment centralizes inference: rather than distributing a model to every device or site that needs its output, the model runs in one location (or a small number of regional locations) and is accessed remotely by every client. This centralization simplifies model updates and monitoring — a new model version can be deployed once, centrally, without redistributing it to every field device — but it makes the entire inference workflow dependent on network connectivity between the client and the serving infrastructure. Every inference request incurs a round trip, and the latency of that round trip is added directly to the model's own computation time to produce the total response time experienced by the requesting application.

Explainer

For ConnectedEarth's audience, model serving is the deployment pattern used whenever a model is too large, too frequently updated, or too centrally governed to distribute to the edge. A defense or maritime organization using a large language model to summarize incident reports typically relies on model serving in a secure cloud or on-premises data center, since running an LLM of that scale on field hardware is currently impractical. The connectivity implication is direct: the quality of that application's user experience is bound to the round-trip latency and reliability of the link between the field site and the serving infrastructure — a constraint that does not apply to a model deployed via edge inference. Understanding whether an AI application depends on model serving or edge inference is the first question ConnectedEarth asks when assessing its connectivity requirements.