RAG development for enterprise

Retrieval-augmented generation is not a chatbot with a folder attached. It is a pipeline that finds the right passages from systems you already trust, respects who may see them, and lets a model write an answer you can check. If you cannot name the document, the permission, and the last eval score, you do not have RAG. You have a demo.

We build these as custom AI: retrieval over proprietary data, evaluation from day one, in your cloud and your repos. This post is the production cut. No vendor SKUs. No invented customer win rates.

If the source of record is still a zip of exports, stop. Fix data governance first.

Retrieval is the product. The model is a writer.

The failure mode is always the same. Someone dumps PDFs into a vector store, points a public model at it, and calls it "our knowledge assistant." The answers sound fluent. They cite the wrong policy, or a draft, or a file the user could not have opened in the source system.

A knowledge system has to answer:

• Which system is canonical for this object (policy CMS, wiki, ticket, contract store).
• How fresh the index is when the source changes.
• Which identity the retrieval uses. A shared service account that can see everything is a bypass.
• What the model is allowed to do with a miss. "I don't know" is a feature.

The model can be an API. The retrieval, the ACL, and the eval should be yours. If you vanished, the index and the filters should still run.

Permissions cannot get looser in the index

If a user cannot open the file in the source, the assistant must not return it. That single rule is most of enterprise RAG.

Hold the line:

• Filter on the user (or a narrower role), not on a god-mode indexer identity, for anything above public or internal non-sensitive.
• Rebuild or refresh enough that a revoked share actually revokes.
• Treat prompt logs as the data class of their contents.
• Do not sync answers into Slack or a ticket with a weaker retention policy.

Yes/no:

• A revoked user loses model access to that corpus the same day they lose source access.
• Security can say who may read raw prompts.
• Restricted classes never go to a personal chatbot "to draft the answer."

Do not invent a new ACL for AI. Reuse the one you already have.

Eval belongs in the first slice, not after the launch party

RAG without eval is a vibe. You need cases that look like your documents: stale policy, conflicting versions, a miss, a user who pastes PII, a question the corpus should refuse.

Minimum:

• A held-out set next to the index config.
• A pass bar: citation present, right document, refusal on a miss — numbers, not "feels grounded."
• A re-run when you change chunking, the model, or the corpus.
• Failures block promote for customer-facing or high-risk internal use.

"It looked good on three questions in the demo" is how POCs stall. Productionizing a knowledge demo is usually eval, permissions, and freshness, not a larger model.

Citations and freshness are how you defend an answer

An answer without a source is a guess with better grammar. Show the passage, the document, and the date. If the index is a week behind the CMS, say so, or do not ship.

Operational rules:

• Indexes are versioned or alias-switched so you can roll back last week's corpus.
• A source change rebuilds or queues the affected objects. Orphan chunks are incidents.
• Chat history is not allowed to become a second, unofficial source of record.
• Cost and rate limits exist. A retrieval loop is an ops event.

If you cannot reconstruct which chunks were used for an answer at 14:12, you cannot do an incident review. Logging is not optional.

When RAG is the wrong shape

RAG is for questions over documents and records you already have. It is the wrong first build when:

• The work is a multi-step write (refund, ticket, CRM update). That is an agent.
• The work is a score or a forecast in a screen people already use. That is a decision model, still custom, not a chat.
• The corpus is small enough that a search box and a human are cheaper. Count hours.
• You do not own the data path. A vendor knowledge base you cannot export is a tenant, not your system.

Buy a generic search copilot when the corpus is clean and not your edge. Build RAG when the documents are how you stay legal or make money, and the permissions are specific.

Scope a thin slice: one corpus, one audience, eval in the first 30-day staging cut if the discovery says it is worth building. You own the code. Fixed scope after discovery. No lock-in.

If the knowledge is yours and a vendor UI cannot express the permissions, the next step is a scoped retrieval slice. See custom AI development.