Most teams start with one model. They wire an SDK into a service, ship something useful, and move on. The trouble arrives later, and it arrives from three directions at once.
A better model is released, and switching means touching every call site. A client asks where their data goes when it is inferred, and the honest answer is “to a vendor in another jurisdiction”. The bill arrives, and nobody can say which feature spent the money. None of these is an AI problem. All three are integration problems, and integration problems are the ones that quietly decide whether a system is still workable in two years.
This is the gap LiteLLM fills.
What it actually is
LiteLLM is an open source project that puts a single, consistent interface in front of more than a hundred model providers. It comes in two shapes, and the difference matters.
The first is a Python library you call directly. It takes the request format popularised by OpenAI and translates it to whatever the target provider expects, so the same function call reaches Anthropic, Azure, Bedrock, Vertex, Cohere or a model running on your own hardware.
The second is a proxy server, usually called a gateway. You run it yourself. Your applications talk to it as though it were a single model provider, and it fans out to the real ones. Everything interesting lives here: which model serves which request, what happens when a provider is slow, who is allowed to spend how much, and what gets written down.
Most of what follows is about the gateway, because that is where a convenience library becomes infrastructure.
Model choice becomes a configuration decision
The first and most visible benefit is that swapping models stops being a code change. A routing rule points a workload at a different model, and the calling code never learns about it.
That sounds like a developer convenience. It is really a commercial position. The model layer is moving faster than any other part of the stack, prices fall in steps rather than curves, and the best model for a summarisation job in March is rarely the best one in September. A team that has to schedule engineering work to change model is a team that stays on the wrong model for a quarter at a time. A team that changes a config line will test three and keep the cheapest one that passes.
It also means a provider outage is survivable. Fallback chains are configuration: try the primary, and on a timeout or a rate limit, move to a named alternative. Anyone who has run a platform through a dependency’s bad afternoon recognises this pattern, because it is the same one we apply to payment gateways and stock feeds.
Routing by data classification
This is the part that matters most in our own work, and it is the reason a gateway is worth running at all.
Our delivery runs on a deliberate mix. Frontier models where the reasoning has to be deepest, and open weight models on hardware we own for everything that should not leave the building. Data classification decides which model sees which work: client intellectual property and regulated data infer locally, on our machines.
Stated as a policy, that is a paragraph in a document nobody reads during a deadline. Stated as a routing table in a gateway, it is a property of the system. The classification is attached to the request, the gateway sends it where the classification says it may go, and a developer in a hurry cannot accidentally send a client’s regulated data to a frontier API, because there is no path from their code to that API except through the gateway.
Speed from AI, confidentiality by architecture. A gateway is how the second half of that sentence stops being an aspiration.
The paper trail
A model without a paper trail is a liability. When something goes wrong six weeks later, the questions are specific: which model version answered, what was in the prompt, what did it cost, who called it, and did anything change between the run that worked and the run that did not.
A gateway is the natural place to answer all of that, because every request already passes through it. Logging callbacks push request and response metadata into whatever observability stack you already run. Spend is tracked per key, which means per team, per client or per feature, depending on how you cut the keys. Rate limits and budgets are enforced centrally rather than trusted to each caller.
The alternative is instrumenting every service that talks to a model, consistently, forever. That works until the third service, written by someone in a hurry.
Same model, same answer, everywhere
One of the quieter benefits is consistency across environments. Pin a model version in the gateway and every environment gets the same one. Development, staging and production stop drifting apart, and “it worked on my machine” stops being a plausible explanation for a difference in output.
This is the same discipline we apply to any runtime. Pin it, version it, and make the thing you tested the thing that runs.
The part that should worry you
Everything above describes concentration, and concentration cuts both ways.
We wrote about the other edge earlier this year, in Your agent is holding too many keys. That post opened with an IETF draft’s account of a March 2026 supply chain campaign against LiteLLM, in which a credential stealer was reportedly shipped through package infrastructure and harvested keys from machines running the affected versions. We have not independently verified those figures and would not repeat them as our own finding. The structural point needs no verification: a gateway’s entire job is to hold API keys for dozens of providers, which makes it one of the highest density credential targets in any estate.
So the honest position on LiteLLM is that the property making it valuable is the same property making it dangerous. One door to every model is also one door to every key.
That is an argument for engineering it properly rather than avoiding it. The disciplines are the ordinary ones, applied without exceptions.
Pin your dependencies and verify what you install, because the reported attack path was package infrastructure rather than the application itself. Give the gateway its own isolated host and its own blast radius, on the assumption that it will one day be the thing that is compromised. Keep provider keys in a secret manager with short lived credentials wherever the provider supports them, so that a snapshot of the gateway’s memory ages out. Put the admin interface behind your own authentication and never on the public internet. Monitor egress from the gateway host, because a credential stealer has to send its haul somewhere. And rehearse rotating every provider key at once, which is a fifteen minute exercise that tells you whether your recovery plan is real.
None of that is exotic. It is the same list we would write for any component holding credentials for an integration estate, which is most of what we do.
Why we think it is important
The interesting thing about LiteLLM is not the feature list. It is that it moves model access from being scattered through application code to being a piece of infrastructure with a name, an owner and a monitoring dashboard.
That change is what makes the hard questions answerable. Where does this client’s data infer? Which model version produced that output? What did last month cost, by feature? Can we move off this provider before renewal? Teams calling model SDKs directly from a dozen services cannot answer any of those without a project. Teams running a gateway answer them by looking.
It also makes the ambitious version possible. Running open weight models on your own hardware for regulated work only pays off if routing to them is trivial. Otherwise local inference becomes a special case that developers work around, and the exception quietly becomes the rule.
We implement AI the way we implement everything: built properly, then run properly. A gateway is a good example of what that means in practice, including the part where you plan for it being attacked.
If you are running models from several services and cannot say where a given client’s data is inferred, that is worth a conversation. It is the sort of problem we like.