Why enterprises need an LLM gateway

A policy PDF cannot see a prompt. A laptop API key cannot enforce a data class. That is why enterprises need an LLM gateway: one path every sanctioned model call has to take, where logging, PII redaction, cost limits, and allow/deny actually run.

Without that path, you have shadow AI, five vendors' dashboards, and a security team that finds out after the paste. The hop itself is enterprise LLM gateway architecture. Putting models inside software you already run is LLM integration. Governance is still the operating model: a secure gateway, a living inventory, risk tiers, controls mapped to NIST AI RMF on AI governance. This post is why the gateway is the first control, not a nice-to-have after the chatbot ships.

Keys in apps are not a control plane

The default enterprise pattern is an API key in an environment file, or worse, in a developer laptop. Each app talks to a model vendor directly. Each vendor has a different log, a different retention story, and a different story about who on their staff can read a prompt.

What you cannot do in that pattern:

• Pull last Tuesday's traffic for one employee across every app.
• Block a data class for every caller at once.
• Stop a runaway agent before the invoice arrives.
• Prove to an auditor that restricted data did not leave.

A key is a credential. A gateway is a control plane. If the only "governance" is a wiki page and a key rotation policy, you are hoping. Hope is not GOVERN in the NIST AI RMF. It is a gap.

Shadow use will always be faster than an ungoverned official path

People paste customer records into public chatbots because the official tool is slow, missing, or forbidden without a quarter of review. Banning ChatGPT without a logged alternative moves the paste to a phone.

A gateway lets you pave the road:

• Approved models and tools on one path.
• Low-risk work logged and fast.
• High-risk classes redacted or blocked before they leave the tenancy.
• Cost visible so the official path does not become an unbudgeted surprise that pushes people back to free personal accounts.

The dirt path wins on Tuesday afternoon unless the official path is shorter. That is the whole argument for putting a gateway in front of the first production app, not after the tenth.

You cannot inventory what you cannot see

A living model inventory needs owner, data class, risk tier, and last eval. Those fields are fiction if the calls never pass a place you operate. Vendor invoices tell you spend. They do not tell you which workflow, which data class, or which prompt tried to extract a record.

With a gateway you can attach traffic to an app, an identity, and a declared data class. Without it, productionizing is a demo plus a prayer. Security asks "what did it say to this user at 14:12" and you open a ticket with the vendor.

Yes/no:

• You can name every production caller this week without pinging five teams.
• You can show whether redaction fired.
• You can show cost per workflow, not only per vendor contract.

If those are no, you do not have an estate. You have apps.

Cost, leakage, and multi-model are the same hole

Three later problems are the same missing hop.

Cost: token spend without a rate limit is how an agent in a retry loop becomes a finance event. A gateway is where you cap, route, and attribute. Spreadsheet archaeology after the bill is not management.

Leakage: PII and secrets in prompts are a data-class problem on the wire. Redaction and block lists belong in front of the model, not in a training about "be careful."

Multi-model: teams will add a second vendor. If every app owns its own client, you will never enforce "this class stays in this region" or "this workflow may not use the cheap model." Routing is a gateway job.

You do not need those three programs on day one. You need the hop they will all use. Architecture details belong in the next post. The reason to start is here: one place to say no.

A gateway is not a freeze, and it is not a chatbot

Done as paperwork, governance slows everything. Done as a path, low-risk work gets faster because the guardrails are already there. Risk-tier the workflow. Internal drafts of public copy should not wait a month. Customer files should not ride a personal key.

A gateway is also not the product. Users still live in the ticket, the claim, or the IDE. The gateway is the join between those apps and the model. Integration without that hop is still paste.

What we implement: logging, PII redaction, cost controls, policy on the request, in your environment. Mapped to NIST MEASURE (you can see the call) and MANAGE (you can block or roll back). We do not publish a package price. We do not invent a customer leak count. "What good looks like" on the governance page is directional: inventory you can produce, low-risk approvals in days, no sensitive-data leaks through governed channels. Ungoverned channels are why the gateway has to exist.

Put it in front of the first production call

Do not wait for a platform program. Discover the calls you already make, including SDKs and vendor features you forgot. Risk-tier them. Point the first sanctioned app at a path you operate. Then operationalize: review the logs, update the allow list, drill an incident.

If the first app still talks to the vendor with a laptop key, you have not started. If you want that path as engineering rather than a slide, talk to an engineer.