Retrieval-augmented generation in regulated industries is not primarily a technology problem. It is a data governance, access control, and audit trail problem that happens to use a language model. The architecture must answer compliance questions before it answers user questions.
The compliance questions first
Before designing a RAG system for a regulated environment, answer four questions: What data can be indexed, and who approved that list? Who is allowed to query which documents, and how is that enforced at retrieval time, not just at the application layer? Is every query and its retrieved sources logged with enough detail to satisfy an audit? And where does inference happen -- on your infrastructure or through a third-party API?
Reference architecture
The pattern that satisfies the most regulated environments: a self-hosted vector store (Qdrant or pgvector on your infrastructure), a local embedding model (not an API), document-level access control enforced via metadata filters in the retrieval query, and an audit log capturing query text, retrieved document IDs, and inference output -- written to an append-only store that your compliance team can review.
Inference should stay local for environments where the document content is the regulated asset. A self-hosted Ollama or vLLM instance running a quantized open-weight model keeps the entire pipeline on your infrastructure. For environments where data classification allows external inference, query routing (small model for simple queries, large model for complex) reduces API exposure and cost.
What to test before go-live
Access control at retrieval time, not just the application layer. A query made by a user without access to a document should never return that document in the retrieved context, even if the application layer has additional guards. Test this with a document set that has mixed access levels before any user sees the system.