LLM API Gateway
Quick Answer
An LLM API gateway is a request-routing layer between application code and one or more model backends, exposing a unified interface — typically OpenAI-compatible — and deferring concrete model selection to runtime. Downstream code targets an LLM capability rather than a specific vendor SDK; the gateway resolves each request against operational policies such as per-request selection, fallback, canary routing, and load balancing. LiteLLM is a common open-source implementation.
LLM API Gateway
An LLM API gateway is a request-routing service that sits between application code and one or more model backends — hosted APIs, self-hosted open-weight models, or both. Downstream code targets a stable, unified interface, typically an OpenAI-compatible endpoint, and the gateway resolves each call using a per-request selector plus policies: fallback on timeout, canary routing, and load balancing. Its unit of indirection is the LLM capability, not the network route, which distinguishes it from a generic API gateway or service mesh. LiteLLM is a common open-source implementation.
A gateway is one implementation of the model-agnostic pattern, not a prerequisite for it. It earns its cost — a service to run and keep in the hot path — when multiple services share backends and credentials, or when fallback policy must stay consistent across teams. Below that scale, an in-process abstraction gives the same isolation without the extra hop.
See also
- What building a model-agnostic research pipeline actually took — why an in-process abstraction beat a gateway at single-pipeline scale.