Chapter 99 — Advanced RAG & Retrieval at Scale

📖 Definition

Retrieval-Augmented Generation (RAG) retrieves relevant information and gives it to the LLM as context. The basics (chunk → embed → search → prompt) are covered in the AI / LLM Engineering: RAG chapter. This chapter covers what senior interviews probe: why RAG fails, how to measure it, and how to scale it.

Most RAG quality problems are retrieval problems. If the right passage isn't in the context, no prompt will fix it.

🔍 The Full Production Pipeline

Offline (ingestion)

1. Load sources (PDF, HTML, Confluence, tickets, DB rows) with metadata (source, owner, date, permissions).
2. Parse and clean (tables, headers, OCR for scans).
3. Chunk with structure awareness.
4. Enrich (titles, summaries, keywords, contextual headers).
5. Embed + build vector and keyword indexes.
6. Keep it fresh: incremental updates, deletions, re-embedding when the model changes.

Online (query)

1. Understand the query (rewrite, decompose, route).
2. Retrieve from several retrievers (vector + BM25 + filters).
3. Fuse and rerank.
4. Assemble context within a token budget, ordered and deduplicated.
5. Generate with citations; verify; refuse if evidence is missing.
6. Log everything for evaluation.

📊 Chunking Strategies

Too small → missing context. Too large → diluted embeddings and wasted tokens. Tune with your eval set.

🔍 Hybrid Search

• Dense (vector) search understands meaning ("car" ≈ "automobile") but misses exact terms: product codes, error IDs, names.
• Sparse (BM25 / keyword) nails exact terms but misses paraphrases.
• Hybrid runs both and merges results — almost always better in enterprise data.

💻 Code Example — Reciprocal Rank Fusion (RRF)

from collections import defaultdict

def reciprocal_rank_fusion(result_lists: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """Each list is doc ids ordered best-first. RRF needs only ranks, not comparable scores."""
    scores: dict[str, float] = defaultdict(float)
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)

fused = reciprocal_rank_fusion([vector_hits, bm25_hits])[:50]
reranked = cross_encoder_rerank(query, fused, top_n=8)      # precision stage

🔍 Query-Side Techniques

🔍 Context Assembly

• Rerank top 50–100 down to 5–10 with a cross-encoder (or LLM reranker).
• Deduplicate near-identical chunks; group chunks from the same document.
• Order: models attend best to the beginning and end ("lost in the middle") — put the strongest evidence first.
• Budget tokens: long context is not free (cost, latency, distraction).
• Include source IDs so the answer can cite them.

🔍 Advanced Patterns

🔍 Vector Indexes (ANN) at Scale

Exact nearest-neighbour search over millions of vectors is too slow, so we use approximate nearest neighbour (ANN).

Key knobs: recall@k vs latency vs memory. Always measure recall against brute force on a sample.

🔍 Permissions & Multi-Tenancy

• Store ACLs (tenant, groups, document permissions) as metadata and filter at query time — never rely on the LLM to hide data.
• Prefer pre-filtering (filter then search) for strict security; post-filtering can return too few results.
• Separate indexes/namespaces per tenant when isolation requirements are strict.
• Propagate deletions quickly (right-to-be-forgotten).

📊 Evaluating RAG

Evaluate retrieval and generation separately.

Build a golden set: real user questions + the documents that answer them + reference answers. Slice results by query type (lookup, multi-hop, numeric, follow-up).

💻 Code Example — Retrieval Metrics

import math

def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    return len(set(retrieved[:k]) & relevant) / max(len(relevant), 1)

def mrr(retrieved: list[str], relevant: set[str]) -> float:
    for rank, doc in enumerate(retrieved, start=1):
        if doc in relevant:
            return 1.0 / rank
    return 0.0

def ndcg_at_k(retrieved: list[str], gains: dict[str, float], k: int) -> float:
    dcg = sum(gains.get(d, 0) / math.log2(i + 2) for i, d in enumerate(retrieved[:k]))
    ideal = sorted(gains.values(), reverse=True)[:k]
    idcg = sum(g / math.log2(i + 2) for i, g in enumerate(ideal))
    return dcg / idcg if idcg else 0.0

⚠️ RAG Failure Modes & Fixes

🧠 What a Senior Answer Adds

• Separates retrieval metrics from generation metrics and fixes retrieval first.
• Designs for freshness, permissions, cost and latency, not just accuracy.
• Knows when not to use vector RAG (numeric questions → SQL; small static corpus → long context + caching).

🎯 Likely Interview Questions

1. Your RAG system gives wrong answers. How do you debug it? — Check if the right chunk was retrieved (recall@k); if not fix chunking/hybrid/query rewriting; if yes fix reranking, prompt, faithfulness checks.
2. Why hybrid search? — Dense handles meaning, sparse handles exact terms/IDs; fusion beats either.
3. What is RRF? — Rank fusion summing 1/(k + rank) across lists; no score normalisation needed.
4. Why rerank? — Cross-encoders are more accurate; apply them to a small candidate set for precision.
5. How do you choose chunk size? — Based on content structure and eval results; parent–child to balance precision and context.
6. What is HyDE? — Embed a hypothetical answer instead of the short question to improve matching.
7. HNSW vs IVF-PQ? — Graph, high recall, memory-heavy vs clustered + compressed, memory-efficient, lower recall.
8. How do you handle permissions in RAG? — ACLs as metadata, filter at retrieval time, never trust the LLM for access control.
9. How do you evaluate RAG? — Retrieval (recall@k, MRR, NDCG) and generation (faithfulness, correctness, citations) on a golden set.
10. RAG vs long context? — Long context for small static corpora with caching; RAG for large, changing, permissioned data and cost control.
11. What is GraphRAG good for? — Multi-hop and global "summarise themes" questions over entity relationships.
12. "Lost in the middle" — what is it? — Models use information at the start/end of long contexts better; order the best evidence first.

← Embeddings, Multimodal & Diffusion | Index | Next: Evaluation & Responsible AI →