What is RAG?

The original formulation#

Lewis et al. (2020) defined RAG as marginalising the generator over retrieved documents:

p(y | x)  ≈  ∑  p_η(z | x) · p_θ(y | x, z)
            z ∈ top-k(x)

x is the query, y the output sequence, z a retrieved passage, η the retriever parameters and θ the generator parameters. Two variants were proposed. RAG-Sequence conditions the whole output on one z. RAG-Token marginalises per generated token, allowing different passages to support different parts of one answer.

Both were trained end to end, backpropagating into the query encoder while the document index stayed fixed. Almost no production system does this today. The industry standard is a frozen retriever and a frozen generator joined by prompt concatenation, which is strictly weaker than the paper and vastly easier to operate.

Fusion-in-Decoder (Izacard and Grave, 2020) is the important middle ground: encode each passage independently, concatenate in the decoder. Cost grows linearly in k rather than quadratically, which is why it scales to large k.

Retrieval, three families#

Sparse lexical. BM25 remains a genuinely strong baseline:

score(q,d) = ∑  IDF(t) · ( f(t,d) · (k1 + 1) )
             t∈q          ---------------------------------------
                          f(t,d) + k1 · (1 − b + b · |d| / avgdl)

IDF(t) = ln( 1 + (N − n_t + 0.5) / (n_t + 0.5) )

f(t,d) is the frequency of term t in document d. |d| is document length and avgdl the mean length over the corpus. N is the number of documents and n_t the number containing t. k1 controls term-frequency saturation, usually 1.2 to 2.0. b controls length normalisation, usually 0.75.

Dense bi-encoder. DPR (Karpukhin et al., 2020) trains separate query and passage encoders with an inner-product objective and in-batch negatives. Passages are embedded once, offline. Query cost is one forward pass plus an approximate nearest-neighbour lookup.

Late interaction. ColBERT (Khattab and Zaharia, 2020) keeps per-token embeddings and scores by summing per-query-token maxima over document tokens. Higher quality than a single-vector bi-encoder, at a substantially larger index.

Hybrid retrieval#

Sparse and dense fail on different queries. Sparse handles exact identifiers, product codes and rare proper nouns. Dense handles paraphrase and synonymy. Reciprocal rank fusion (Cormack et al., 2009) combines ranked lists without needing calibrated scores:

RRF(d)  =  ∑  1 / ( k + r_i(d) )
           i

r_i(d) is the rank of document d in list i, and k is a smoothing constant conventionally set to 60. The absence of score normalisation is the point: raw BM25 scores and cosine similarities are not on comparable scales, and fusing ranks sidesteps that entirely.

Reranking#

A cross-encoder scores the query and passage jointly in a single forward pass, which is far more accurate than a bi-encoder and far too slow to run over a corpus. The standard cascade retrieves k ≈ 100 cheaply, then reranks to the top 5 with O(k) cross-encoder passes. This is usually the highest quality-per-engineering-hour change available to a struggling RAG system.

Position effects#

Liu et al. (2023) documented a U-shaped accuracy curve over the position of relevant information within a long context: strong at the beginning, strong at the end, measurably weaker in the middle. The practical consequences are concrete. Order retrieved passages so the most relevant sit at the edges, and treat "increase k" as a change that can reduce accuracy.

Evaluation#

Separate the stages or you will chase the wrong bug.

• Retrieval: recall@k, MRR, nDCG@10 against labelled query-passage pairs.
• Grounding: is every claim in the answer entailed by the retrieved context? Natural-language-inference models or an LLM judge with a strict rubric both work; both need a human-labelled calibration set.
• Answer quality: exact match or F1 for extractive questions, pairwise preference for open-ended ones.

RAGAS (Es et al., 2023) packages faithfulness, answer relevance and context relevance as reference-free metrics. Useful for regression detection, weak as ground truth. Do not ship on it alone.

Active directions#

Self-RAG (Asai et al., 2023) trains the model to emit reflection tokens deciding when to retrieve and whether the retrieved evidence supports its own draft. HyDE (Gao et al., 2022) embeds a hypothetical answer generated from the query rather than the query itself, closing the vocabulary gap between short questions and long passages at the cost of one extra generation.

The open problem remains upstream of all of it. Retrieval failure dominates the error budget in almost every deployed system, and no amount of generator quality compensates for a missing passage.

Papers#

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020 — arxiv.org/abs/2005.11401
• Karpukhin et al., Dense Passage Retrieval, 2020 — arxiv.org/abs/2004.04906
• Khattab and Zaharia, ColBERT, 2020 — arxiv.org/abs/2004.12832
• Izacard and Grave, Leveraging Passage Retrieval with Generative Models, 2020 — arxiv.org/abs/2007.01282
• Gao et al., Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE), 2022 — arxiv.org/abs/2212.10496
• Liu et al., Lost in the Middle, 2023 — arxiv.org/abs/2307.03172
• Asai et al., Self-RAG, 2023 — arxiv.org/abs/2310.11511
• Cormack et al., Reciprocal Rank Fusion, SIGIR 2009

What to learn next#

• Vector databases — approximate nearest neighbour search in practice.
• Embeddings — bi-encoder training and representation quality.
• Fine-tuning — when to adapt weights instead of retrieving.