Best LLM Gateway Platforms: Pick by Who Runs the Gateway

What is an LLM gateway?

An LLM gateway is a reverse proxy between your applications and one or more model providers. It exposes one API, usually OpenAI-compatible, and applies policy to every request: which model answers, what happens on failure, whether a cached answer is reused, who may spend how much, and what gets logged. Your code talks to the gateway; the gateway talks to OpenAI, Anthropic, Bedrock, or your own vLLM cluster.

Search “what is llm gateway” and you get vendor definitions that each end in a product. The neutral version is shorter: it’s the one place where model access becomes a managed, budgeted, observable resource instead of API keys scattered across environment variables. A team asking what is LLM gateway infrastructure for usually has several apps calling several providers and no shared view of spend.

Two neighbors get confused with it. An LLMOps platform covers prompts, evaluations, and tracing; the gateway is the runtime hop those tools observe. Teamvoy’s guide to LLMOps tools covers that category. An SDK gives you one interface inside one process, but it can’t enforce budgets or rotate keys across teams.

LLM API gateway vs classic API gateway

A classic API gateway terminates TLS, authenticates callers, rate-limits by request count, and routes by path. An LLM API gateway adds token quotas, semantic caching, provider-format translation, model fallback on 429 or 5xx errors, prompt guardrails, and per-token cost attribution. Kong, Azure API Management, and Apigee extend the classic gateway with these policies; LiteLLM, Portkey, and Bifrost start from the LLM side.

The practical test for an LLM API gateway candidate: can it reject a request because a team spent its token budget, and can it tell you which key, model, and prompt consumed it? A classic gateway answers neither without custom plugins.

How does an LLM gateway handle a request?

A request to an LLM gateway passes through authentication, policy, routing, provider translation, and logging before the response streams back. Each stage can short-circuit the path: an over-budget key stops at policy, a cache hit stops before routing, and a provider failure loops back for a fallback. The table maps the eight functions most products ship.

The path in order: client → virtual-key auth → policy checks → cache lookup → router → provider adapter → provider API → response guardrail → logging and cost attribution → client.

Latency labels are relative; Anthropic notes that actual latency depends on prompt length, output length, and thinking effort. For the provider-level choice, read Teamvoy’s Anthropic vs OpenAI comparison.

How do you choose an LLM gateway?

Choose an LLM gateway on eight criteria: hosting model, license, provider coverage, governance and RBAC, observability depth, cost model, lock-in surface, and ops burden. Weight them by your constraint: a regulated fintech weights hosting and governance first, a product team shipping a prototype weights time-to-live and cost model first. The table gives each criterion a question you can answer in a one-hour vendor call.

The gateway log is the audit trail your compliance team will ask for, and the retention tier decides whether it still exists in six months. Teamvoy’s piece on LLM observability and evals in production covers what auditors expect.

Which are the best LLM gateway platforms by operating role?

The best LLM gateway platforms fall into four operating roles, and the right answer depends on which role your organization can staff. Each group gets a best-fit and caveat table rather than a ranking, because a self-hosted MIT proxy and a credit-funded marketplace solve different procurement problems. Every vendor fact carries its retrieval date and official source.

1. Open-source self-hosted proxies: LiteLLM, Portkey Gateway, Bifrost, llmgateway.io, Helicone AI Gateway.
2. Managed LLM gateways and control planes: Cloudflare AI Gateway, Vercel AI Gateway, OpenRouter, Portkey Cloud, TrueFoundry.
3. API-management platforms extended for AI: Kong AI Gateway, Azure API Management, Google Apigee.
4. Cloud-provider patterns: AWS Guidance for Multi-Provider Generative AI Gateway.

Open source LLM gateway options: self-hosted proxies

An open source LLM gateway gives you the keys, the logs, and the patching duty. Five projects hold the category as of 2026-10-22, and their licenses are the first filter. LiteLLM is MIT with a separately licensed enterprise directory, Portkey Gateway is MIT, Bifrost and Helicone AI Gateway are Apache-2.0, and llmgateway.io is AGPLv3 with a commercial ee/ directory.

LiteLLM calls itself, in its README retrieved 2026-10-22, an “Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready.” Its LICENSE is MIT outside the enterprise/ directory. The enterprise docs, updated 2026-10-20, keep virtual keys, budgets, spend tracking, the admin UI, load balancing, guardrails, and an MCP gateway in open source. The license key gates SSO and SCIM beyond 5 users, audit logs, org and team admins, and secret-manager integrations. Enterprise pricing is “talk to sales.”

Portkey Gateway is MIT-licensed and, per its README on 2026-10-22, routes “to 250+ LLMs with 1 fast & friendly API.” The README announces a Gateway 2.0 pre-release that merges Portkey’s enterprise gateway into open source; the repository’s last push was 2026-05-25, so check which branch you deploy.

Bifrost, from Maxim, is Apache-2.0 and ships as a Go binary or Docker image. Its README on 2026-10-22 says it unifies “23+ providers” behind one OpenAI-compatible API with failover and semantic caching. Clustering, guardrails, and an MCP gateway sit in a commercial tier with no published price.

llmgateway.io appears in Google’s AI Overview for “llm gateway” as of 2026-10-22. Its LICENSE is AGPLv3 for the core, with the ee/ directory under a commercial license. AGPL matters if you modify the gateway and expose it as a service; have legal read it before the pilot.

Helicone AI Gateway is a Rust proxy under Apache-2.0 whose README on 2026-10-22 still carries a Public Beta badge; the repository’s last push was 2025-11-21. Verify maintenance before you shortlist it.

Managed LLM gateways and control planes

Managed LLM gateways remove the patching duty and replace it with a billing model you need to read closely. As of 2026-10-22 the five products below charge in four ways. Cloudflare keeps core features free and charges for logs. Vercel passes through provider list price with no gateway fee. OpenRouter takes a cut of purchased credits. Portkey Cloud prices by recorded logs, and TrueFoundry offers a free request allowance with custom tiers above it.

Cloudflare AI Gateway runs on all Cloudflare plans with logging, caching, rate limiting, retries, and model fallback, per docs updated 2026-04-20. Its pricing page, updated 2026-10-15, keeps analytics, caching, and rate limiting free. Log storage follows Workers Logs pricing for gateways created on or after 2026-09-24; older accounts get 100,000 logs on Workers Free and 10,000,000 per gateway on Workers Paid. Unified Billing adds a 5% fee to purchased credits, with no markup on inference.

Vercel AI Gateway, docs updated 2026-09-14, offers central credentials, request logs, budgets per team, project, key, or member, ordered fallbacks, and BYOK. Its pricing page, updated 2026-09-08, states the gateway “charges no markup and no platform fee on tokens” and that “you’re responsible for any payment processing fees that may apply.” Enterprise teams pay by invoice with no processing fees.

OpenRouter is a model marketplace rather than a proxy you host. Its FAQ, read 2026-10-22, states a 5.5% fee with an $0.80 minimum on credit purchases and provider pricing passed through without markup. BYOK on pay-as-you-go includes $25,000 per month of list-price inference with no BYOK fee; above that, OpenRouter charges 5% of the model’s price on its platform.

Portkey Cloud, priced on 2026-10-22, has a free Developer tier with 10,000 recorded logs per month and 3-day retention that Portkey labels “not suitable for production workloads.” Production is $49 per month for 100,000 recorded logs, plus $9 per additional 100,000 requests.

TrueFoundry AI Gateway lists VPC, on-prem, and air-gapped deployment plus MCP and Agent Gateway products as of 2026-10-22, with a free tier of 30,000 requests per month for 3 seats. The page states no license, so this article doesn’t call it open source. Paid tiers aren’t listed.

API-management platforms extended for AI

API-management vendors add LLM policies to a gateway your platform team may already run, which makes them the cheapest option to adopt and the hardest to leave. Kong, Azure API Management, and Google Apigee all documented semantic caching, prompt screening, and provider translation as of 2026-10-22. These features follow the vendor’s edition and tier structure, so the first question is which policies your contract includes.

Kong AI Gateway runs on the AI Proxy plugin. Per its docs on 2026-10-22, the plugin accepts OpenAI-format requests and translates them for 17 providers, including OpenAI, Azure OpenAI, Bedrock, Anthropic, Gemini, Vertex AI, Ollama, and vLLM. The docs hub names AI Semantic Cache, AI Semantic Prompt Guard, an AI Sanitizer that redacts PII, and governance for MCP and A2A traffic. Kong’s pricing page on 2026-10-22 lists AI Management Plus from $25 per month plus usage. The docs don’t state which AI plugins ship in open source versus Enterprise or Konnect, so check which plugins your Kong edition includes.

Azure API Management, per Microsoft’s page updated 2026-05-29, “extends API Management’s existing API gateway; it’s not a separate offering.” It applies to all tiers, with capabilities varying by tier, and handles OpenAI, Anthropic, and Vertex AI schemas plus Microsoft Foundry and Amazon Bedrock. Policies include token limits per consumer, token metrics, semantic caching, load balancing, Azure AI Content Safety moderation, and a credential manager.

Google Apigee documents a SemanticCacheLookup policy backed by Vertex AI embeddings and a SanitizeUserPrompt policy that screens prompts with Model Armor, both retrieved 2026-10-22. Google notes they “might have cost or utilization implications, depending on your Apigee license.”

Cloud-provider gateway patterns

AWS doesn’t sell a standalone LLM gateway; it publishes a reference pattern you deploy in your own account. The Guidance for Multi-Provider Generative AI Gateway on AWS, retrieved 2026-10-22, routes clients through CloudFront and WAF to a load balancer, then to a LiteLLM proxy on ECS Fargate or EKS, with ElastiCache, RDS, Secrets Manager, and S3.

The guidance states its middleware “integrates natively with Amazon Bedrock to enable features not supported by the LiteLLM opensource project.” You run LiteLLM, you pay AWS infrastructure, and you get Bedrock hooks the upstream project lacks. The page’s publication date wasn’t captured.

Pricing models and licenses compared

Gateway pricing follows five models, and the one you pick decides whether cost scales with traffic, logs, or seats. The table dates every number to 2026-10-22 and links the official page; expect drift within a quarter. For how gateway spend fits the wider budget, see Teamvoy’s breakdown of the cost of AI implementation.

Percentage fees on credits are painless at $2,000 a month and visible at $200,000. Per-log pricing means observability depth and bill move together. Infrastructure-only pricing looks free until you cost the engineer who runs it.

Leading AI gateway for LLM workload optimization: a decision framework

The phrase leading AI gateway for LLM workload optimization describes a capability set, not a product, and four levers deliver it: routing, caching, fallback, and budgets. Every gateway in this article ships some version of each. What separates them is how much policy you can express without code, and whether that policy lives in a format you could migrate. Judge a candidate by the four points below rather than by a crowned winner.

Routing. Can you send classification to a small model and reasoning to a large one by rule? Multi-provider routing also hedges the provider decision; Teamvoy’s Anthropic vs OpenAI comparison explains why that hedge is worth keeping open.

Caching. Exact-match caching is safe and narrow. Semantic caching widens the hit rate and adds the risk of answering a similar but different question. Set a similarity threshold per endpoint and exclude anything personalized.

Fallback. Ordered fallbacks keep the feature up when one provider rate-limits you. Each fallback needs its own cost ceiling, or an outage at your cheap provider becomes a bill from your expensive one.

Budgets. Budgets per key, team, and project with alerts before the hard limit are the control finance asks for first. LiteLLM’s open-source tier and Vercel’s gateway both document this as of 2026-10-22. A budget per gateway is a different control from a budget per product line.

Where LLM gateways fail

LLM gateways fail in four repeatable ways: an added network hop, a single point of failure, stale cached answers, and lock-in through proprietary policy formats. Each one is manageable, and each one costs engineering time the pricing page doesn’t show. Price them into the pilot before the gateway becomes load-bearing.

Hop latency. Every gateway adds a round trip and a translation step. Vendors publish their own overhead figures; this article omits them because the number you need is measured on your path. Compare p50 and p99 with and without the gateway on identical prompts.

Single point of failure. A gateway every application depends on is a new tier-one service. Self-hosted options need high availability and a runbook. Managed options move the risk to the vendor’s status page, a different risk rather than a smaller one. Keep a documented bypass path for applications that can’t tolerate gateway downtime.

Cache staleness. A cached answer about a price, a policy, or an account can be wrong the moment the source changes. Scope caching to prompts whose correct answer holds within the TTL, and log cache hits separately so support can trace a stale response.

Lock-in. Routing rules, guardrail configs, and virtual-key hierarchies in one vendor’s format are the migration cost you pay later. Prefer gateways whose config is plain YAML or JSON in your repo, and keep the OpenAI-compatible interface as the contract your apps depend on.

Build vs buy: when does Teamvoy fit?

Most mid-market teams should adopt an existing gateway rather than build one, and that includes teams that could hire Teamvoy to build it. LiteLLM, Cloudflare AI Gateway, or Vercel AI Gateway can run on shadow traffic within days using documentation alone. A bespoke gateway is justified only when policy requirements exceed every license tier you can afford: data residency that rules out SaaS, air-gapped deployment, or MCP governance across dozens of teams.

The honest version of this trade-off runs against our commercial interest. A two-person platform team with a clear provider list and a budget owner can finish the pilot protocol below without outside help. Teamvoy’s AI integration services earn their place in the work around the gateway: the evaluation set behind the routing rules, the observability pipeline into your SIEM, the guardrail policy mapped to your compliance scope, and the integration into the apps that call it. Our guide to enterprise AI integration covers how that work is scoped.

Teamvoy hasn’t deployed or benchmarked the products named here on client traffic. This comparison reports what each vendor documented on 2026-10-22, grouped so you can shortlist the best LLM gateway platforms by operating role.

How to pilot an LLM gateway in 7 steps

Pilot an LLM gateway on shadow traffic with written SLOs and a rollback plan before any production request depends on it. The seven steps take a platform team two to four weeks with any product here. The output is a decision memo with measured latency, cost per accepted request, and the policies you need, which is what procurement needs to approve a plan.

Decide and document rollback. Pick one gateway, write the bypass procedure for direct provider calls, and date the decision so pricing gets re-checked in a quarter.

Shortlist two gateways from different operating roles. One self-hosted and one managed, so the pilot compares cost models rather than feature lists.

Mirror production traffic. Send a copy of real requests through each gateway and discard the responses; shadow traffic exposes translation bugs that synthetic prompts miss.

Write SLOs before measuring. Set a p99 latency ceiling, an availability target, and a fallback success rate. A pilot without SLOs produces opinions.

Measure cost per accepted request. Divide total spend, including engineer-hours, by requests that produced a response your application used. Cache hits and fallbacks move this number more than token prices do.

Exercise failure. Revoke a provider key, force a 429, and fill a budget. Record what the gateway did and what your application saw.

Test the exit. Export the routing config, logs, and virtual-key hierarchy. If the export needs a vendor ticket, the lock-in surface is larger than the docs suggest.

Conclusion

The best LLM gateway platforms aren’t a ranked list; they’re four operating roles, each with a license and a billing model you should read before the feature list. Self-host when you must own the keys, buy managed when you need the layer this week, extend your API platform when you already run one, and use the AWS pattern when your account is the boundary. Pilot on shadow traffic, measure cost per accepted request, test the exit, and date the decision. Talk to Teamvoy About LLM Gateway Integration. Teamvoy builds the parts around the gateway that vendors don’t ship: the evaluation set behind your routing rules, the observability pipeline into your SIEM, the guardrail policy mapped to your compliance scope, and the application integration. If you want a second pair of eyes on the pilot plan, contact Teamvoy.

FAQ