Evaluate retrieval before generation
Create labeled queries with relevant passages, authority, freshness, permission scope, and expected no-answer behavior. Measure recall at k, ranking quality, duplicate rate, filter correctness, and source-version selection. A model cannot cite evidence it never receives.
Include exact identifiers, paraphrases, multi-hop questions, conflicting sources, stale content, denied content, empty results, and hostile documents. Analyze by language, product, tenant, source type, and query difficulty.
Measure claim support and citation validity
Break answers into material claims and check whether the cited passages support them. Score correctness, completeness, citation precision, citation recall, abstention, and conflict handling. Fluency and similarity are not substitutes for evidence.
Validate citation identifiers against the authorized retrieved set after generation. Removed, denied, stale, or unrelated sources must never appear merely because the model emitted a plausible ID.
Connect quality to operations
Trace ingestion, embedding, query rewriting, filtering, lexical and vector retrieval, fusion, reranking, context assembly, generation, and citation validation. Record versions, counts, durations, status, tokens, and safe error classes.
Report stage p50/p95/p99 latency, error and timeout rates, index freshness, deletion lag, tokens, and cost per supported answer. Evaluate fallback behavior under search, model, and identity failures.
Gate releases
Compare baseline and candidate on the same versioned set. Use noncompensable authorization and severe unsupported-claim gates alongside quality, latency, and cost. Canary with fixed probes and representative traffic where permitted, then preserve the decision and rollback manifest.
Decision framework
| Area | Guidance |
|---|---|
| Layer | Evidence |
| Ingestion | Source ownership, version, ACL, deletion, and chunk lineage |
| Retrieval | Recall, ranking, filtering, freshness, and duplicates |
| Generation | Correctness, support, citations, abstention, and safety |
| Operations | Latency, errors, drift, freshness SLO, and cost |
Practical checklist
- Version corpus, chunker, embeddings, index, prompts, and models
- Label expected evidence and abstention
- Test authorization before retrieval
- Validate claim-level citations
- Run injection and stale-source cases
- Compare cost per supported answer
First-party sources
Source status last checked 2026-09-11. Links can change after publication.