Evaluating LLM Applications in Production
Offline test sets tell you whether a change is safe to ship; production signals tell you whether it keeps working. How to combine both into an evaluation practice for LLM features.
- Published
- Reading time
- 2 min read
- Categories
- LLM Evaluation
On this page
Evaluating an LLM feature once, before launch, is not enough. Inputs drift, users find new ways to phrase things, and models are updated. Evaluation has to become a continuous practice with two halves: offline evaluation before a change ships, and online evaluation while it runs.
Start from the behaviour you expect
Before choosing metrics, write down what "good" means for this feature. For a summarisation feature that might be: faithful to the source, covers the key points, stays within a length limit, and never invents names or numbers. Each of these statements becomes something you can check.
Offline evaluation: a regression suite for behaviour
An offline suite is a fixed dataset of inputs with expectations, executed against a candidate configuration (model, prompt, retrieval settings).
| Check type | Example | Strength |
|---|---|---|
| Deterministic | Output parses against a schema; length below a limit | Cheap, exact, reproducible |
| Reference-based | Overlap with or similarity to a reference answer | Useful when a good reference exists |
| Model-graded | A grader model scores faithfulness against a rubric | Scales to open-ended output |
| Human review | Expert labels on a sample | Ground truth for calibrating the others |
A few practices make the suite trustworthy:
- Keep the dataset representative, including edge cases and known past failures — not only the happy path.
- Pin everything that affects the result: model version, prompt version, temperature, retrieval index.
- Calibrate model-graded checks against a set of human labels, and re-check that agreement when the grader changes.
- Compare against a baseline, not against an absolute threshold alone. The question is usually "is this better or worse than what we ship today?"
Online evaluation: signals from real usage
Production traffic reveals what an offline set cannot. Useful signals include:
- Validation failures and fallbacks triggered by the application.
- Explicit feedback (ratings, corrections) and implicit feedback (regenerations, abandonment).
- Sampled outputs scored asynchronously by the same model-graded checks used offline.
- Latency and cost per request, which are part of quality from a user's perspective.
Be careful with privacy: decide up front which data may be logged, for how long, and who can see it.
Close the loop
The most valuable step is turning production observations into offline test cases. When a failure is found, add a minimal, anonymised example to the dataset with the expected behaviour. Over time the offline suite becomes a record of everything the system has learned not to do.
A minimal workflow
- Define expected behaviour as checkable statements.
- Build a small, representative dataset — dozens of well-chosen examples beat thousands of random ones.
- Run the suite in CI for every prompt, model, or retrieval change and compare to the baseline.
- Monitor production signals and sample outputs for grading.
- Feed failures back into the dataset.
Evaluation does not make an LLM deterministic. It makes changes to an LLM system observable and comparable, which is what engineering decisions need.