Evaluating LLMs in Production: A Pragmatic Measurement Framework

Executive Summary

"It looks pretty good to me." That sentence is the single biggest blind spot in enterprise AI today. Most Generative AI applications are still evaluated by developers manually running a few dozen queries and eyeballing the outputs — a "vibe check." It works fine for a demo of ten questions. It fails completely at the scale of a production system fielding thousands of complex, unpredictable user interactions a day, where it silently hides hallucinations, data leakage, and slow model drift until a customer notices first.

This paper lays out a pragmatic, programmatic alternative: a mathematical evaluation framework built on three pillars — the RAG Triad for measuring retrieval-and-generation quality, LLM-as-a-judge for scoring it at scale, and continuous production telemetry for catching drift after deployment.

The reframe that matters: evaluation is not a pre-launch checkbox. It is a rigorous, continuous testing and telemetry discipline — the same discipline traditional software engineering already applies to deterministic code, adapted for a probabilistic system that can produce a different answer to the same question twice.

The RAG Triad: Three Axes, Not One Score

Originated as part of the open-source TruLens project and now a widely adopted evaluation pattern, the RAG Triad decomposes "is this answer good?" into three independently measurable questions [1][2]:

Each axis catches a different failure. A system can retrieve perfectly relevant context and still fail Groundedness by hallucinating extra detail the context never supported. It can be perfectly grounded in the retrieved text and still fail Answer Relevance by technically-correct rambling that never answers what was asked. Measuring only one axis — as most "does the answer look right" manual review implicitly does — misses the other two entirely.

Why this decomposition matters operationally: when a RAG Triad score drops, the axis that dropped tells you where to look. A Context Relevance regression points at the retrieval pipeline (see our companion paper, Beyond Naive RAG). A Groundedness regression points at the generation prompt or the model itself. Conflating them into one blended "quality score" turns every regression into a full-system fire drill instead of a scoped fix.

LLM-as-a-Judge: Scoring at Scale

Manually grading the Triad across thousands of production interactions is not feasible. The practical alternative, now well studied in the research literature, is LLM-as-a-judge: using a strong, carefully-prompted LLM with an explicit grading rubric to score outputs automatically.

The foundational study on this technique — Zheng et al.'s "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," published at NeurIPS — found that frontier judge models can match human preference judgments with over 80% agreement on open-ended evaluation tasks, while also documenting real limitations: position bias (favoring whichever answer is shown first), verbosity bias (favoring longer answers), and self-enhancement bias (a model favoring outputs similar to its own style) [3][4]. Open-source evaluation frameworks built on this pattern — RAGAS being one of the most widely adopted — operationalize the RAG Triad's metrics (faithfulness, answer relevance, context precision, context recall) as automated, LLM-judged scores rather than manual review [5][6].

Operationalizing it

At Vibodh AI, we implement LLM-as-a-judge evaluation as a CI/CD gate, not a one-time report:

1. Maintain a golden dataset of curated queries (a few hundred is a realistic enterprise starting point) representing real, previously-seen query patterns and known-correct answers.
2. Run the full RAG Triad against that dataset automatically on every meaningful change — a new system prompt, a swapped embedding model, an updated reranker.
3. Fail the pipeline (or flag for review) if any axis regresses below a defined threshold, exactly as a unit test suite blocks a regression in deterministic code.

This converts "did that prompt change break anything?" from a question someone answers by vibes to a question a CI pipeline answers automatically, every time.

Continuous Telemetry: Evaluation Doesn't Stop at Launch

A golden dataset tells you how the system performs on queries you anticipated. Production tells you how it performs on the queries you didn't. Deep observability closes that gap.

Production LLM telemetry, at minimum, should capture:

• Latency — time-to-first-token (TTFT) and full generation speed, since perceived responsiveness drives usage and abandonment independently of answer quality.
• Rejection rates — how often deterministic guardrails (content filters, PII checks, off-topic detectors) block or intervene on a request, which surfaces both attempted misuse and legitimate queries the guardrails are too aggressive about.
• Implicit and explicit user feedback — copy/paste rates and session continuation as implicit signals of a useful answer; thumbs up/down and correction submissions as explicit ones.
• Drift indicators — periodic re-scoring of the golden dataset against the live production pipeline (not just at deploy time) to catch silent degradation from an upstream model version change or a vendor-side update no one on the team triggered.

The uncomfortable truth about drift: an LLM-powered system can degrade without a single line of your code changing. A model provider ships a quiet update, an embedding model gets deprecated in favor of a new default, a data source's formatting shifts upstream. Continuous telemetry is the only way to notice before a customer does.

Conclusion

Confidence in AI output should come from measurement, not intuition. The RAG Triad gives engineering teams three independent, diagnosable axes instead of one vague "quality" impression. LLM-as-a-judge makes measuring those axes tractable at production scale, with well-documented biases that can be engineered around rather than ignored. Continuous telemetry extends evaluation past the deploy button, into the only environment that actually matters: real usage, over time. Organizations that treat AI evaluation as a rigorous testing and observability discipline — not a pre-launch checklist — are the ones that can deploy Generative AI with the same operational confidence traditionally reserved for deterministic software.

References

1. TruLens (TruEra), "The RAG Triad," open-source project documentation.
2. TruEra, "What is the RAG Triad?," AI quality education resource.
3. L. Zheng, W.-L. Chiang, Y. Sheng, et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," Advances in Neural Information Processing Systems (NeurIPS) 36, 2023. arXiv:2306.05685.
4. Snowflake Engineering, "Benchmarking LLM-as-a-Judge for the RAG Triad Metrics" and "Eval-Guided Optimization of LLM Judges for the RAG Triad," engineering blog, 2026.
5. Explosion / IPSR, "RAGAS: An Open-Source Framework for Smarter LLM Evaluation," 2026.
6. Confident AI, "RAG Evaluation Metrics: Assessing Answer Relevancy, Faithfulness, Contextual Relevancy, and More," 2026.

Sources referenced in this paper include the peer-reviewed NeurIPS literature on LLM-as-a-judge, open-source evaluation framework documentation (TruLens, RAGAS), and current engineering writeups on production LLM observability. All figures are paraphrased from public reporting; readers should consult the original publications for full methodology.

About Vibodh AI

Vibodh AI builds the measurement layer enterprises need to trust their AI systems in production — RAG Triad instrumentation, LLM-as-a-judge evaluation pipelines wired into CI/CD, and the telemetry to catch drift before it reaches a customer.

From evaluation-framework design to full observability rollouts across existing AI products, we partner with engineering and product teams as a long-term, responsible AI partner. Think AI. Build beyond.