When a retrieval-augmented system gives a bad answer, the instinct is to change the prompt or the model. Usually the right passage was never in the context.
A retrieval-augmented pipeline has two places to fail. Either the right information never reached the model, or it did and the model handled it badly. These need completely different fixes, and they are easy to confuse because both surface as a wrong answer.
Separate the two before touching anything
Build a small set of questions with the passage that should answer each one labelled by hand. Fifty is enough to be useful. Then measure one thing: how often the labelled passage appears in the retrieved context at all. That single number tells you which half of the system to work on, and it is usually the disappointing half.
- Low retrieval hit rate — chunking, embeddings, and query handling are the problem. Prompt work will not help.
- High hit rate, bad answers — now generation is worth attention: prompt structure, citation enforcement, model choice.
What usually fixes retrieval
- Chunk on document structure — headings, sections, function boundaries — rather than a fixed character count that splits sentences and separates a claim from its qualifier.
- Run hybrid search. Dense embeddings miss exact identifiers, error codes, and product names; BM25 catches them. Fuse the two result sets.
- Add a cross-encoder reranker over a wider candidate set. Retrieve 50, rerank, keep 5. This is often the single largest improvement available.
- Keep enough metadata on each chunk to cite it precisely. An answer you cannot trace back to a source is not verifiable, and users learn quickly not to trust it.
Then keep measuring it
Retrieval quality drifts as the corpus grows — a chunking strategy tuned on 200 documents behaves differently at 20,000. Put the eval in CI so a regression shows up in a pull request rather than in a support conversation.