Write an evaluation contract
Define the task, users, allowed outcomes, severe failures, data, slices, labels, metrics, thresholds, owners, exceptions, and review cadence before running candidates. Include security and human-impact outcomes, not only average answer quality.
Version the evaluation set and preserve ambiguity decisions. Synthetic data is useful for privacy and attacks, but validate whether it represents the language, workload, and consequences of the intended environment.
Use measures with known limitations
Pair deterministic checks, expert rubrics, user signals, and calibrated model-based judges. Measure judge agreement and drift when its model or prompt changes. Do not let a candidate grade itself as the sole release authority.
Report distributions and confidence, not only one score. Analyze severe failures separately and avoid overinterpreting tiny slices.
Observe semantic health
Infrastructure telemetry says whether requests ran; evaluation says whether the result was useful, supported, authorized, and safe. Combine scheduled probes and governed samples with correlated stage traces and release metadata.
Minimize raw content. Prefer identifiers, versions, counts, durations, safe error categories, and sampled scores. If content inspection is justified, use restricted access, redaction, short retention, and auditable purpose.
Operate the feedback loop
Define quality SLOs, alert windows, minimum samples, owners, and runbooks. Canary candidates, contain severe failures, roll back complete system configurations, verify recovery, and add incidents to the regression set.
Decision framework
| Area | Guidance |
|---|---|
| Signal | What it answers |
| Trace | Where time, errors, tools, and policy decisions occurred |
| Metric | How health changes across time and releases |
| Evaluation | Whether task, evidence, safety, and quality outcomes are acceptable |
| Decision record | Why a release was promoted, changed, rejected, or rolled back |
Practical checklist
- Freeze evaluation definitions before comparison
- Track complete version lineage
- Calibrate automated judges
- Minimize telemetry content
- Define quality SLO owners and runbooks
- Test rollback and recovery
First-party sources
Source status last checked 2026-09-11. Links can change after publication.