AI Observability and LLM Evaluation for Production GenAI Systems
AI observability and evaluation is the practice of tracing every step of a GenAI request, such as retrieval, prompts, model calls and tool use, and continuously scoring outputs for correctness, groundedness, safety, latency and cost, both before release and in production. NTSPL builds this telemetry and evaluation layer so teams can detect regressions, explain failures and prove quality to risk owners.
Delivered on
In scope
What to Observe in a GenAI System Beyond Uptime and Latency
Traditional monitoring says the API returned 200 in 1.8 seconds. It cannot say the answer was wrong. GenAI systems fail quietly: a model version update shifts tone, a new document batch pushes stale passages up the ranking, an agent starts taking an extra tool call per case, and token costs creep up 30% with no alert. Teams discover problems from user complaints weeks later, then cannot reproduce them because nobody stored the retrieved context or prompt version that produced the bad output.
NTSPL instruments GenAI applications end to end using OpenTelemetry-compatible traces that capture inputs, retrieved chunks, prompt versions, model parameters, tool calls, outputs, tokens and latency per span. On top, we build three evaluation loops: an offline golden-set suite that gates every release, online scoring of sampled production traffic with calibrated LLM judges, and a human review queue fed by low scores and user feedback. Dashboards and alerts are set on quality, not just uptime.
Signs Your GenAI Application Is Running Blind
Signals you need AI Observability & Evaluation
- A GenAI application is live with more than 500 daily users but quality is judged only by complaints or thumbs-down counts.
- A model or prompt change in the last quarter caused a regression that took more than a week to detect.
- Risk, compliance or clinical owners ask for accuracy evidence and the team can only show demo examples.
- Token spend per request has risen more than 20% without anyone being able to attribute the cause.
- Agents or RAG pipelines fail intermittently and engineers cannot reproduce failures from logs.
When to choose something else
- If the application is still a prototype with fewer than 50 users, a simple golden-set script and manual review are enough; full observability can follow.
- If the main concern is adversarial attacks rather than quality, start with AI security and red-teaming, then add evaluation.
- If the need is infrastructure monitoring for GPUs and clusters, use platform monitoring; this page addresses application-level quality.
Tracing, Evaluation and Review Components We Implement
Tracing instrumentation
OpenTelemetry-based spans for retrieval, prompt assembly, model calls, tools and post-processing, with prompt and model versions as attributes.
Golden datasets
Curated test sets of 200-500 items with references, covering normal, edge and adversarial cases, versioned like code.
Offline evaluation gate
Automated scoring of each release candidate on correctness, groundedness, format and safety, blocking promotion on regression.
Online evaluation
Sampled scoring of 2-10% of production traces with calibrated judges, plus heuristics for refusals, length and tool-call counts.
Human review queue
Low-score traces and negative feedback routed to domain reviewers, whose labels become new golden items.
Dashboards and alerts
Quality, latency, cost per request and error dashboards per application and version, with alerts on drift and spikes.
Three-Loop Evaluation Architecture: Offline, Online and Human
Three-Loop LLM Evaluation and Tracing Architecture
Applications emit traces for every request into a trace store; an offline loop scores release candidates against golden sets, an online loop scores sampled production traces with calibrated judges, and a human loop reviews the worst cases. Scores, costs and latencies feed dashboards, alerts and the release gate.
Select any component
Three-Loop LLM Evaluation and Tracing Architecture: click any of its 9 components to see its role and the design choice NTSPL makes for it, or trace the 7-step primary path.
CloudWatch, Foundry Observability and Google's Gen AI Evaluation Service
Bedrock invocation logging and CloudWatch generative AI observability capture model and agent traces, Bedrock evaluations run RAG and model evaluation jobs, and S3 holds golden sets and labelled traces.
Foundry tracing builds on OpenTelemetry and Application Insights, Foundry evaluators score groundedness, relevance and safety, and evaluations run in CI pipelines as release gates.
The Gen AI evaluation service provides model-based and computation-based metrics, Cloud Trace captures OpenTelemetry spans, and BigQuery holds trace and score history for analysis.
Metric Choice, Judge Calibration, Sampling and Release Gates
Choosing metrics per application
Pick 3-5 metrics that map to business harm. RAG assistants: faithfulness, citation precision, answer relevance and refusal correctness. Extraction: field-level precision and recall. Agents: task completion, tool-call count and override rate. Classification: per-class recall, especially for high-risk classes. Avoid generic scores like overall quality 1-10; they are noisy and hard to act on.
Calibrating LLM judges
An uncalibrated judge is an opinion. Calibrate each judge prompt against 100-200 human-labelled items and require Cohen's kappa of about 0.6 or more, or 85% binary agreement, before using it for gates. Use a judge from a different model family where possible, require reasoning before scores, and recalibrate after any judge model change.
Sampling production traffic
Scoring every trace is usually unaffordable. Sample 2-10% stratified by intent, language and user segment, plus 100% of traces with negative feedback or safety flags. For low-volume high-risk intents, such as complaints or medical questions, score all traffic. Budget online evaluation at roughly 10-25% of the application's own model spend, and revisit the sampling rate after the first month of data.
Release gate tolerances
Set tolerances per metric, not one overall threshold: for example, no more than a 1-point drop in faithfulness, zero new failures on the safety set and no more than 10% p95 latency increase. Small drops within noise should not block releases; run the golden set three times when results are near the threshold to separate noise from regression.
Instrument, Baseline, Gate, Monitor: Rollout Steps
Instrumentation
Add tracing spans, version attributes and feedback capture to the application and route traces to an in-region store.
Golden set and baseline
Build or extend golden sets with domain experts and measure the current production version as the baseline.
Judge calibration
Write judge prompts per metric and calibrate them against human labels until agreement thresholds are met.
Release gate in CI
Wire offline evaluation into the deployment pipeline with agreed tolerances and reporting.
Online scoring and alerts
Enable sampled online evaluation, dashboards and drift alerts, and start the human review queue.
What you receive
Quality, Reliability and Cost Signals We Track
| KPI | Typical target |
|---|---|
| Time to detect a quality regression in production | Under 24 hours |
| Releases passing through the evaluation gate | 100% |
| Judge agreement with human labels | Kappa 0.6 or higher per metric |
| Traces reproducible from stored context and versions | Above 99% |
| Online evaluation cost as share of app model spend | 10-25% |
Evaluation Traps That Give False Confidence
| Risk | How NTSPL prevents it |
|---|---|
| Golden sets become stale and stop reflecting real usage | Monthly additions from human review of production failures and quarterly retirement of obsolete items. |
| LLM judges reward fluent but wrong answers | Calibration on human labels, reference-based scoring where possible and blind human sampling of judge-approved outputs. |
| Traces store sensitive data longer than allowed | Masking at ingestion, chunk IDs instead of text for restricted content and retention aligned to data class. |
| Alert fatigue from noisy quality metrics | Alerts on sustained shifts over rolling windows, minimum sample sizes and ownership per application. |
LLM evaluation vs traditional software testing: what is different?
Traditional tests assert exact outputs; GenAI outputs vary between runs and many correct answers exist. Evaluation therefore uses scored metrics over datasets rather than pass-or-fail assertions, and judges, human or model-based, to grade open-ended outputs. It also never ends at release: models, content and user behaviour change, so production traffic must be sampled and scored continuously. Keep conventional unit and integration tests for code, schemas and tool contracts, and add golden-set evaluation, calibrated judges and online monitoring on top. Teams that rely only on manual demo checks discover regressions from users, usually weeks late.
Key facts: AI Observability & Evaluation
AI Observability & Evaluation in Practice
2 AI Observability & Evaluation scenarios, from Pharmaceuticals (pharmacovigilance case processing) and Higher education (university admissions), plus related platform case studies.
Pharmacovigilance hub catches a seriousness-classification regression within a day
University admissions chatbot stops quoting last year's fees after evaluation drift alerts
Telecom VAS provider cuts daily pages from 2,300 alerts to 9 actionable incidents
IT services firm brings 43 Azure subscriptions under one policy hierarchy
AI Observability & Evaluation: Frequently Asked Questions
What is LLM-as-a-judge and can we trust it?
LLM-as-a-judge uses a language model with a scoring rubric to grade another model's outputs. It is trustworthy only after calibration: we compare its scores with human expert labels on 100-200 items and use it for gates only when agreement is high. Humans still review a sample of judged outputs.
What does setting up AI observability and evaluation cost?
For one or two production applications, a typical engagement takes six to eight weeks with indicative NTSPL fees of ₹10-24 lakh. Ongoing costs include trace storage and online judge calls, usually 10-25% of the application's own model spend, plus a few hours a week of expert review.
Which metrics matter for a RAG application?
Faithfulness, meaning claims are supported by retrieved context; citation precision; answer relevance; retrieval recall on known questions; and correct refusal for out-of-scope questions. Latency and cost per answer complete the picture. We avoid single overall quality scores because they hide which component failed.
Can traces be stored in India?
Yes. Trace stores, logs and evaluation results can sit in AWS Mumbai, Azure Central India or Google Cloud Mumbai or Delhi. CERT-In's directions require ICT logs to be retained for 180 days within India, and DPDP Act 2023 obligations apply to personal data in prompts and responses, so we mask identifiers at ingestion.
Do we need a third-party observability tool?
Not necessarily. The native tracing and evaluation services on each cloud cover most needs, especially when combined with OpenTelemetry. Specialist open-source or commercial LLM observability tools add richer trace views and dataset management; we recommend them when several applications span multiple clouds.
How do we evaluate agents rather than single answers?
We score whole tasks, not just final text: task completion against expected outcome, number and correctness of tool calls, policy compliance at each step, human override rate and cost per task. Replaying historic cases through a new agent version is the most reliable pre-release test.