One AI Gateway for Every Model

Most companies I work with end up in the same spot. You start with one model in the cloud. Then a second team wants a different one. Then a use case shows up where the data can't leave the building, so the model has to run on-premises. Then another provider turns out cheaper for a specific job. Before long you have models in Azure, models from other vendors, and models in your own data center — and no single place to control any of them.

That is exactly the problem an AI Gateway is meant to solve. The way I'd build it is with Azure API Management (APIM) and its Self-Hosted Gateway. Below I go a level deeper — the routing, the policies that run on every call, the protocols it speaks, and how security and cost are handled — but the idea stays simple: one governed layer in front of every model.

One front door for every model

Your applications shouldn't have to know which model they are talking to, or where it runs. When APIM sits in front, every model — Azure OpenAI, an external vendor like OpenAI, Anthropic or Google, or an open-source model you host yourself — lives behind one endpoint. The contract your SDKs already use (the OpenAI-style /chat/completions and /responses calls) stays the same, and APIM normalizes the differences behind it.

Routing is done with backend pools. You can load-balance across several deployments — round-robin, weighted, or priority-based — and a circuit breaker trips on repeated errors or HTTP 429s and fails over to the next backend. A common pattern is to send traffic to provisioned throughput (PTU) first and spill over to pay-as-you-go when the PTU is saturated. If you want to move traffic from one vendor to another, you change it in one place. The apps don't change.

Run the gateway where the model lives

This is the part that makes "cloud and on-prem" real. The Self-Hosted Gateway is the same APIM gateway, just shipped as a container. You run it wherever you need it — in your own data center, on Kubernetes or OpenShift, on plain Docker, or in another cloud such as GCP. It's deployed with a Helm chart and scales with HPA or KEDA like any other workload.

The split is what matters: the control plane stays in Azure, and the gateway only reaches out to it over 443 to pull configuration and push telemetry. The data plane — the actual prompts and responses — stays local and never leaves your network. The gateway keeps a local cache of its configuration, so if the link to Azure drops it keeps serving traffic. One control plane, many environments.

What runs on every call — the AI Gateway policies

APIM applies policies to each request and response. These are the GenAI-specific ones I rely on (each also has an azure-openai-* variant for Azure OpenAI backends):

• llm-token-limit — Enforces tokens-per-minute limits per app, team or key. Estimates prompt tokens before the call and returns standard rate-limit headers.
• llm-emit-token-metric — Emits prompt, completion and total token counts to Application Insights with your own dimensions — app, team, model, use case.
• llm-semantic-cache-lookup / -store — Caches responses by meaning, using embeddings and a vector store (Redis Enterprise), so similar prompts skip the model.
• llm-content-safety — Runs prompts through Azure AI Content Safety — Prompt Shields for jailbreak and injection, category filters, blocklists — before they reach the model.
• Backend pool + circuit breaker — Load-balances across model backends (weighted or priority) and trips a breaker on errors or 429s to fail over — e.g. provisioned throughput first, pay-as-you-go as spillover.
• validate-jwt — Validates Microsoft Entra ID (OAuth 2.0 / OpenID Connect) tokens and scopes on every request, before any model is touched.

The protocols and standards it speaks

• OpenAI Inference API — the /chat/completions and /responses contract, so existing clients and SDKs work unchanged across vendors.
• Streaming over SSE — token-by-token responses are streamed with Server-Sent Events (text/event-stream); APIM passes the stream through while still metering tokens.
• MCP (Model Context Protocol) — JSON-RPC 2.0 over SSE or the newer Streamable HTTP. APIM can expose an existing REST API as an MCP server and govern MCP tool calls.
• A2A (Agent2Agent) — agent-to-agent messaging over HTTPS (JSON-RPC, with SSE for streaming). APIM sits in the middle to enforce identity, policy and audit between agents.
• OAuth 2.0 / OpenID Connect / JWT — Entra ID tokens validated at the gateway, plus managed identity and on-behalf-of flows to reach backends without shipping keys to apps.
• OpenAPI, plus gRPC, WebSocket and GraphQL — APIs are imported from OpenAPI specs, and the gateway can also front gRPC, WebSocket and GraphQL backends with the same controls.

Cost, quotas and observability

Because the token-metric policy runs on every call, you get usage broken down by whatever dimensions you choose — app, team, model, use case — flowing into Application Insights and Log Analytics. Products and subscription keys give each consumer its own quota. For on-premises and dedicated-capacity models, where there is no per-token bill, you set your own internal token rates and do real charge-back against the same numbers. The same telemetry can be streamed to your SIEM, so cost and security see one source of truth.

Security in depth

• No keys in apps — the gateway authenticates to model backends with managed identity; certificates and secrets sit in Key Vault.
• Identity at the door — validate-jwt checks Entra ID tokens and scopes before a request reaches any model.
• Prompt-level protection — Azure AI Content Safety with Prompt Shields catches jailbreak and prompt-injection attempts, PII and blocked categories, on the way in and out.
• Traffic protection — mutual TLS to backends and client-certificate validation at the edge, so both ends of the connection are verified.

Governing agents: MCP and A2A

Agents change the traffic pattern — it's no longer just an app calling a model, it's tools calling tools and agents calling agents. The gateway treats those the same way. You expose MCP servers and tools through APIM and put authentication, rate and token limits, and telemetry on every tool call. For agent-to-agent (A2A) traffic you get a governance point in the middle: agent identity, authentication and authorization, policy enforcement, and a full audit trail of who called whom. That is exactly the kind of A2A gateway control a regulated platform needs.

Why this fits in every environment

You end up with one consistent way to govern AI, no matter where a model runs or who builds it. Security, cost and audit are handled at the gateway — not re-invented in every application. A new model, a new vendor, a new on-prem deployment simply plugs in behind the same door, inherits the same policies, and shows up in the same logs. And because the gateway can run inside your own network, the sensitive traffic stays inside your own network.

Bottom line

You don't have to pick one model, one vendor, or one place to run AI. With APIM and the Self-Hosted Gateway you put a single, governed layer — one contract, one set of policies, one audit trail — in front of all of them, and you keep control as you add more.