AI prompt and data leakage prevention
Prompt leakage is data leaving through the model hop: pasted secrets, unredacted PII, and logs that outlive the ticket. Stop it at the gateway, not in a policy PDF.
Prompt leakage is not a future risk. It is last week's ticket pasted into a public chat, a secret that now lives in a vendor log, and a debug bucket that still has last quarter's customer names. A policy PDF does not stop it. The hop does.
This is the control cut of enterprise AI security architecture. The gateway is where class is evaluated. Data governance is what the class means. No invented incident counts. No DLP product SKU.
If you cannot say what left, who saw it, and how you would stop the next one, you do not have leakage prevention. You have hope.
Leakage is a path, not a vibe
Treat leakage as data leaving a boundary you named. The usual paths:
- A person pastes a ticket, a contract, or a dump into a consumer model.
- An app sends restricted fields because the prompt template asked for "full context."
- A tool call returns more than the model needed, and the model echoes it.
- A prompt log, eval set, or Slack thread keeps the payload after the job is done.
- A vendor retention clause you never read keeps completions for training or support.
Yes/no:
- You can name the classes that may leave the VPC.
- You can name the classes that may never hit a consumer or personal account.
- You can pull last week's outbound prompts for one app without a forensic project.
If those are no, stop adding models. You are multiplying exits.
Do not call it "the model leaked" when a developer pasted an env file. That is egress you allowed. The model is a destination. You chose the route.
The paste problem is still the main hole
Most leakage is not a clever jailbreak. It is a person trying to finish work. The paved path is slow, the consumer path is fast, and the ticket has a customer email in it.
You will not get to zero paste. You can make the official path the easy one and make the unofficial path visible.
Hold:
- A paved client that already has identity, class, and logging, so the "just this once" paste is not faster.
- DNS and package allow lists so a personal key is not the default SDK path.
- A short, written rule: restricted data never goes to a personal account, even for a demo.
- An amnesty window to declare unofficial tools, then a kill or a pave. Shadow AI is the discovery cut.
Yes/no:
- A new hire can complete a normal AI task without a personal ChatGPT login.
- Security can see consumer-domain traffic from corp networks without reading message bodies.
- The joiner/leaver process includes prompt-log ACLs and unofficial tool access.
Training that says "don't paste secrets" and nothing else is a slide. People paste secrets when the official tool cannot see the ticket.
Redact before the hop, not after the headline
Redaction is a control on the payload, not a promise in the system card. It runs before the bytes leave.
Decide what you strip versus what you block. Emails, account numbers, and obvious secrets can be replaced with tokens the app still understands. A full health record, a raw dump of credentials, or an entire contract marked restricted should deny, not "best effort mask."
Hold:
- Detectors for secrets, payment-shaped numbers, and your own customer-id patterns.
- A map from class to action: allow, redact, or deny.
- A way to put a field on the deny list without a two-week change board.
- Tests. If redaction only exists in a README, it does not exist.
Fail closed on high-risk routes. Missing class is deny. Unknown app identity is deny. A detector timeout on a restricted route is deny, not "send it, we will look later."
Do not pretend regex is a privacy program. It is a speed bump. The real control is class plus route. Redaction catches the field someone forgot to drop. It is not a license to send the rest of the row.
Class decides the route
Public copy can use a cheap, fast model. Confidential business data stays on a contracted tenant in a pinned region. Restricted data (PII, health, payment, credentials) does not go to a consumer model, a personal account, or an unlogged SDK.
The architecture is class to route to redaction or block. That lives on the gateway, not in each app's good intentions.
Yes/no:
- Two apps cannot pick different models for the same restricted class.
- A region pin is enforced, not documented.
- You can change the map without editing twelve clients.
Vendor evaluation still applies: residency, retention, who at the vendor can read prompts, subprocessors, exit. Their SOC report is input. It is not your redaction.
If a developer can still hit the provider with a laptop key, class is advisory. You will not get to zero bypass. You can make bypass obvious in logs.
Logs, eval, and Slack are stores
The completion is not the only copy. Prompt logs inherit the class. So do eval fixtures, fun completions in Slack, and the support export the vendor keeps.
Hold:
- Retention that matches the class, not a forever debug bucket.
- ACL on raw traces. Break-glass is gated and logged.
- Eval sets that use synthetic or tokenized data unless the eval owner accepted the real class.
- A ban on pasting production prompts into chat for "does this look right?"
Yes/no:
- You can delete or expire a customer's prompts when legal asks.
- A contractor cannot browse last month's restricted traces from a shared bucket.
- You know the vendor's retention in days, not "industry standard."
A debug store that holds raw customer prompts is a store you have to defend. If you would not put that file on a laptop, do not put it in an unscoped object store.
Tools are an exit too
The model is one destination. A tool call is another. A lookup that returns a full customer row, then a completion that quotes the row, is leakage with extra steps.
Keep tool results minimal. Pass identifiers, not dumps. If the tool can write, treat the write as a separate class of incident. MCP sits behind the same identity and class story as the model hop. This post is not a tools primer. The rule is the same: if it can leave, it is on the map.
Yes/no:
- Each tool has an allow list and a data class.
- A tool not on the list is deny.
- You can turn a write tool off without redeploying the app.
If the only off switch is "email the vendor," you do not own the exit.
Prove it with one route, then expand
Do not write a leakage program for twelve models. Pick one production app. Put it on the hop. Prove:
- Restricted payload is denied or redacted, and the app degrades.
- You can pull the outbound prompt and the class decision in one query.
- A personal-key bypass shows up as an exception, not silence.
- An incident note can expire the log and update the deny list the same day.
Then the second app. High-risk writes stay gated. Governance is the operating model. This post is the engineering cut.
We implement the hop in your environment. You own the deploy. We do not sell a leakage appliance.
If the next paste still has no class and no log, you do not have prevention. If you want the controls designed as engineering, talk to an engineer.
