AI governance for software developers

A policy PDF does not sit in the pipeline. It does not block a bad prompt from reaching production. It does not keep last week's model pin. Developers already know this. They ship around documents that never touch git.

AI governance for software teams is the discipline you already use for services: change control, tests before promote, observability in prod, a rollback that has been run. The artifacts are different — prompts, model IDs, retrieval configs, tool permissions, eval sets — but the failure mode is familiar. If it is not in the release path, it is not governed.

This is not a freeze on feature work. Treat the AI pieces as code. The enterprise AI governance fields still apply: inventory, owner, data class, risk tier. This post is how those fields show up in an SDLC. The method matches the rest of AI governance: discover what is calling a model, risk-tier the write path, implement guardrails in the environment, operationalize so the next release does not throw the controls away.

If you still lack a source of record or an owner, the readiness gaps come first. Do not add an eval gate to a workflow nobody owns.

Prompts and models are change-controlled artifacts

The prompt that ran in the demo is not the prompt in prod two months later. Someone tweaked a system message. Someone bumped a model because it was cheaper. Retrieval started pointing at a share that includes customer exports. None of that showed up in the PR list because none of it was treated as a change.

Put the movable pieces in the repo or in a config service the repo references:

• System and developer prompts, versioned, not edited only in a vendor UI.
• Model name and pin, or a documented floating alias with an owner who accepts the drift.
• Retrieval sources, chunking rules, and which collections a given app may search.
• Tool permissions: what the agent may call, with which credentials.
• Decoding settings that change behavior enough to matter.

A prompt change goes through the same review as a code change for that service. Production cannot take a prompt that is not on the promoted revision. The inventory row records the current prompt version, model pin, and last eval. Rolling back the service rolls back the prompt and the pin together.

If product wants to A/B a prompt, that is a flagged experiment with a ticket, a traffic split, and a stop condition — not a live edit. NIST MAP is the inventory of what you shipped. If you cannot name the revision, you cannot map it.

Eval is a gate, not a demo script

"It looked good on a few tickets" is how a demo leaves the room. In the SDLC, eval is the test suite for behavior you cannot assert with a unit test alone.

Build a set that matches the risk:

• Golden tasks from real, redacted production examples, including ones that already failed.
• Policy cases: PII in the input, conflicting sources, a user who asks to ignore the system prompt, a retrieval miss.
• Regression cases that lock yesterday's fixes.
• A written pass bar: accuracy, refusal, citation, or schema validity — numbers, not "feels better."

Before promote: the set lives in version control next to the prompt; CI or a release job runs it on the candidate; fail blocks prod. High-risk services cannot skip the job with a comment. A break-glass path exists, is rare, and writes a ticket. Medium-risk gets a smaller set plus a named reviewer on the release notes. Low-risk internal drafts can ship with a smoke set. They still have a set. The inventory stores the score and the date.

MEASURE in the NIST AI RMF is this job. If you only measure in a slide after launch, you are writing a postmortem in advance. A prompt drafted in a public model still has to pass the gate once it lands in the repo.

Human-in-the-loop sits on write-backs, not on every token

A rule that a human must approve every completion will be ignored. It should. Most completions are drafts. The dangerous step is the write: creating a ticket, sending a customer email, changing a claim, merging a record, deploying a config.

Split the path:

• Read and draft: the model may propose. Log it. A human or downstream system decides whether to use it.
• Suggest-and-apply on low-risk internal objects: allowed if logged and reversible.
• Write to a system of record, a customer channel, or a payment or identity path: human confirm, or a deterministic policy that is not the model.

For high-risk write-backs: the agent cannot obtain the production credential without a confirmation step you own; the UI shows the proposed diff or message, not only "approve"; the person who clicks understands the record; a dry-run and idempotency exist so a double click does not create two customer letters; the inventory tier is high if a write is possible, even if the happy path is read-only. Permissions drift.

This is MANAGE at the moment of action. A policy that says "be careful with customer data" does not intercept the API call. A confirmation step does. If you human-gate a Slack emoji summary, people will turn the gate off.

Production has logs you can actually query

Once it is live, demand what you already demand of an HTTP service: who called, what went in, what came out, what it did next, how much it cost, and whether a guardrail fired.

Minimum log for a governed AI service:

• Request id, service version, prompt version, model pin.
• Caller identity and tenant.
• Declared data class and whether redaction or a block fired.
• Tool calls and whether a write was proposed or committed.
• Latency and cost fields so a loop is an ops event.
• Retention that matches your incident and audit window.

Logs should land with the rest of the service's telemetry, or in a gateway you control, not only in a vendor dashboard. PII in prompts is redacted at the gateway or before the log line is stored. "We log everything" is not a virtue if the log is a second copy of the customer file. On-call can answer what it said to this user at 14:12 without a vendor feature request. Cost alerts exist. An agent in a retry loop is a page, not a month-end surprise.

A secure AI gateway is the usual way to get logging, redaction, and cost controls on every model call without inventing it per app. You can start with one service. You cannot claim the estate is governed until the calls you care about pass a path you operate.

Rollback is a rehearsed action, not a slide

When eval is green and prod is not, you need to go back. AI rollbacks fail when the model pin lives in a vendor console, the prompt lives in a database row someone edited, and the retrieval index was rebuilt in place.

Design the release so rollback is boring:

• Prompt, model pin, retrieval pointer, and tool-permission file move as one release unit.
• Indexes are versioned or alias-switched so you can point at last week's corpus.
• Feature flags can disable write tools without disabling the whole app.
• The previous eval score stays attached to the previous revision.

You have run a rollback in staging in the last quarter. On-call docs name the flag, the pin, and who can approve a high-risk rollback after hours. Rolling back does not require the original developer. After rollback, the inventory row is updated so "current" is not a lie.

MANAGE includes the un-ship. If the only path is "call the vendor and hope they still have the old prompt," you do not have a rollback.

The pipeline is the policy, or the policy will lose

Policy-PDF governance is a long document and a committee. Developers read it once, or never. Prompts change in a UI. Models float. Eval is a meeting. An incident becomes a search through Slack and a vendor export you may not get. Low-risk work waits months. High-risk work happens on a laptop because the official path is the slow one.

Pipeline governance encodes the same principles as inventory fields, risk tiers, CI gates, gateway rules, and a write confirmation. A prompt change is a PR. A failed eval is a red job. A write is a gated tool. Security can pull logs. Product can see cost. Legal can see the data class on the row. Low-risk ships in days because the guardrails are already in the path.

You still need short written rules: approved tools, banned data classes, who can approve a tier change. Those rules are useless if they only live in a handbook. They become useful when the handbook and the pipeline agree.

For a team that already has one AI feature in prod: discover the actual model calls, including SDKs and vendor features you forgot; risk-tier by data class and write-back, not by how proud the demo was; put prompt and pin in git; add a small eval job; put calls through a logged path; gate the write; re-run eval when the corpus or the model alias changes.

That is GOVERN as engineering, not as a slide. MAP the artifacts, MEASURE in CI and in prod, MANAGE with confirmation and rollback. Do not wait for a perfect program to put the first pin in the repo.

If the AI work is already in your services and the controls are still in a PDF, put governance on the path those services already take. See how Mufasa builds that as engineering — gateway, living inventory, risk tiers, and the operational loop — in your environment.