+1 (201) 771-1665 Sales Support 24/7 Engagements active in 5 regions
Field Notes

AI in Production -- RAG, Agents & Cost | Cobham Consulting Group

The gap between a demo and a production AI system is measured in observability, failure handling, cost controls, and the people who have to keep it running at 2am. Most AI projects fail not because the model is wrong but because the infrastructure around it was not built for production from the start.

What production means in practice

A production AI system has four properties a demo lacks: it handles the inputs it was not designed for without crashing, it logs enough information to debug failures after the fact, it has a circuit breaker that prevents a degraded model from affecting your users indefinitely, and someone knows what to do when it breaks.

RAG systems fail in ways that are hard to detect without measurement. Retrieval quality degrades as documents change. Embedding models drift relative to query patterns. Chunk sizes that were optimal at deployment become suboptimal as the document corpus grows. Without automated evaluation running continuously, you will not know retrieval quality is degrading until users tell you.

Cost management

API-based inference costs are a function of token count multiplied by call volume. Systems that looked cheap in testing become expensive at production traffic levels. Cost per query should be a tracked metric from day one, with alerts when it exceeds a defined threshold. The cheapest path is usually a smaller model for the majority of queries with a larger model for fallback -- not the largest model for everything.

Every AI system we deploy includes a cost dashboard and an alert threshold. The team operating it should know cost per query before they know anything else.

Common questions

Frequently asked.

What metrics should we track for a production RAG system?
Time-to-first-token, retrieval precision (are the retrieved chunks relevant?), retrieval recall (are relevant chunks being retrieved?), cost per query, and user feedback rate. Start with the first two -- if retrieval is wrong, model quality is irrelevant.
How do we detect when model quality degrades over time?
Run a fixed evaluation set on a schedule and alert when metrics fall below a threshold. The evaluation set should include questions with known correct answers drawn from your actual document corpus.
Should we use GPT-4 or a smaller model for production?
Start with the smallest model that meets your quality threshold. A 7B model handles most document extraction and summarization tasks adequately at 1 to 5 percent of the cost of GPT-4. Use the larger model as a fallback for cases the smaller model cannot handle.
All field notes
Questions on this?

Talk to a principal.

If this essay described a problem you are working on, schedule a 30-minute call. We map your specific situation and tell you what we would do first.

Schedule a call
Direct to a principal · Response under four hours