Enterprise LLM gateway architecture

An enterprise LLM gateway is a hop you operate, not a library you copy into every app. Identity comes in. Policy runs on the request. Logs and redaction happen here. A model provider, or several, sit on the other side. If each application still holds a vendor key, you drew a box on a slide. You did not build a gateway.

The why is in why enterprises need an LLM gateway. Putting models inside software you already run is LLM integration. This post is the shape we implement as AI governance: one governed entry for LLM usage, in your environment, with logging, PII redaction, and cost controls. No invented reference architecture. No SKU.

Put the hop in your network, not in a vendor's demo tenant

The gateway has to live where you can fail it closed. That usually means your VPC or equivalent, on identity you already use, writing logs to a store you can query without a vendor ticket.

What belongs inside the hop:

• Authentication of the calling app and, when you have it, the user.
• A declared data class or an inferred one from policy.
• Allow/deny for tool, model, region, and data class.
• Redaction or block before the payload leaves.
• Attribution: request id, app, model pin, cost fields.

What does not belong inside the hop:

• The product UI. Users stay in the ticket or the IDE.
• The system of record. The gateway is not your CRM.
• A second identity stack invented for AI.

If the only deploy is a cloud vendor's "gateway" you cannot log locally, you bought another dashboard. Treat it as a provider, not as the control plane.

Apps speak to the gateway. The gateway speaks to models.

Do not let application teams pick SDKs that bypass you. The contract is: one client, one base URL, one set of credentials that only the gateway holds for upstream providers.

On the inbound side:

• Service identity for each app. A shared "ai-user" is how you lose attribution.
• Optional user token so retrieval and tools can respect ACLs.
• Timeouts and retries that the gateway owns, so a loop is one place to kill.

On the outbound side:

• Provider credentials in a secret store you control.
• Region pins that match the vendor contract, not a UI checkbox.
• A route table: this workflow may use these models, these regions, this cost class.

When a team needs a new model, they get a route, not a new key. That is the difference between architecture and sprawl.

Policy is a request filter, not a handbook chapter

The useful policies are short and executable.

• Data class X cannot leave the tenancy, or must be redacted first.
• App Y cannot call a write tool.
• Model Z is only for low-risk internal drafts.
• This identity cannot exceed N tokens or N dollars an hour.

Map those to risk tiers you already use: low, medium, high. Low-risk should pass fast. High-risk should fail closed if the class is missing. A policy that requires a meeting for every completion will be routed around.

NIST MAP is the inventory row (app, data class, tier). MEASURE is the log line. MANAGE is the deny, the rate limit, and the incident when redaction fires on something that should never have been sent.

Logs are first-class, and they inherit the data class

If you "log everything" you may have built a second copy of the customer file. Architecture has to say what is stored, where, and who can read it.

Minimum fields:

• Request id, app, user or service, model pin, route.
• Declared data class and whether redaction or deny fired.
• Tokens and cost.
• Enough prompt/completion text for an incident, at the matching class, with admin-only access.

Retention matches your ticket or email schedule unless legal says otherwise. Export to the SIEM you already have. The gateway's own UI is not the system of record for audits.

Start with one app, then make bypass the hard path

Do not boil the ocean. Point the first production app at the hop. Prove you can pull a log, trip a deny, and cap a loop. Then move the second app. Personal keys and direct SDKs become the exception, time-boxed, on the inventory as unsanctioned or temporary.

DNS, egress, and package allow lists help, but the paved client is what makes developers stay. If the official path is slower and worse, they will not use your architecture.

We implement this in your environment, mapped to the same discover → risk-tier → guardrails → operationalize loop as the rest of governance. You own the deploy. We do not publish a package price.

If the box on the diagram still sits in a vendor console you cannot query, it is not your gateway. If you want the hop as engineering, talk to an engineer.