Chapter 108 — GenAI Interview: RAG & Vector Database Questions

📖 How to Use This Chapter

RAG is the most common GenAI project type, so expect many questions on it. Answers here are practical; for deeper material see AI / LLM Engineering: RAG and GenAI & ML (Senior): Advanced RAG & Retrieval.

1. What is RAG and why use it?

Retrieval-Augmented Generation: before the LLM answers, retrieve relevant information from your data and include it in the prompt. Benefits: answers use private and up-to-date data, fewer hallucinations, citations, no retraining when data changes, and permission control.

2. Explain the RAG pipeline.

Ingestion (offline):

Load documents (PDF, Word, web pages, Confluence, database).
Extract and clean text (OCR for scans, handle tables).
Split into chunks.
Create embeddings for each chunk.
Store vectors + text + metadata in a vector database.

Query (online):

Embed the user's question.
Search for the most similar chunks (often hybrid with keyword search).
Optionally rerank.
Build a prompt with instructions + chunks + question.
LLM generates the answer with citations.

3. How do you choose chunk size?

Depends on the content. Typical start: 300–800 tokens with 10–15% overlap. Small chunks = precise but may lose context; large chunks = more context but less precise matching and more tokens. Better: split by structure (headings, paragraphs), keep tables intact, and test different sizes on your evaluation set.

4. What is chunk overlap and why use it?

Repeating some text between neighbouring chunks so a sentence or idea at a boundary isn't cut in half and lost.

5. Which vector databases have you used? How do you choose?

Options: Pinecone, Weaviate, Qdrant, Milvus, Chroma, pgvector (PostgreSQL), MongoDB Atlas Vector Search, Elasticsearch/OpenSearch, Azure AI Search.

Choose by: existing stack (already on MongoDB/Postgres → use its vector search), scale, metadata filtering needs, hybrid search support, managed vs self-hosted, cost, and data residency.

6. What is cosine similarity?

A measure of how similar two vectors' directions are, from −1 to 1. Commonly used to compare embeddings — higher means more similar meaning.

7. What is hybrid search and why use it?

Combining vector (semantic) search with keyword (BM25) search. Vectors find similar meanings ("cancel subscription" ≈ "terminate plan"); keywords find exact terms (product codes, error numbers, names). Merge the results (e.g. reciprocal rank fusion). Almost always better for business documents.

8. What is reranking?

After retrieving, say, the top 30 chunks, a reranker model (cross-encoder) scores each chunk against the question more accurately and keeps the best 5–8. Improves answer quality and reduces tokens.

9. How do you handle metadata filtering?

Store metadata with each chunk (department, product, date, language, access groups) and filter during search — e.g. only the user's permitted documents, or only the latest version. Filtering is also key for security.

10. How do you ensure users only see documents they're allowed to?

Store allowed groups/users as metadata and filter in the retrieval query based on the logged-in user. Never rely on the prompt ("don't show confidential info") for access control.

11. How do you keep the vector database up to date?

Incremental ingestion: detect new/changed/deleted documents (timestamps, hashes, webhooks), re-chunk and re-embed only those, delete old chunks by document ID, and store version metadata. Re-embed everything when changing the embedding model.

12. The RAG system gives wrong answers. How do you debug?

Check retrieval first: were the right chunks retrieved? (log retrieved chunks).
If not: improve chunking, add hybrid search, query rewriting, metadata filters, better embedding model.
If retrieval is right but the answer is wrong: improve the prompt, reranking/ordering, reduce noise, use a stronger model.
Add the failing case to the evaluation set.

13. How do you evaluate a RAG system?

Retrieval: hit rate / recall@k (is the right chunk in the top k?), MRR.
Answer: correctness, faithfulness (supported by context?), relevance, citation accuracy.
Behaviour: says "I don't know" when the answer isn't available.
Use a test set of real questions with expected answers and source documents; score with code checks, an LLM judge, and human review. Tools: Ragas, DeepEval, LangSmith, custom scripts.

14. How do you handle follow-up questions in a chat?

Rewrite the follow-up into a standalone question using chat history before retrieval: "What about its waiting period?" → "What is the waiting period for the Gold Health policy?". Then retrieve with the rewritten query.

15. How do you handle tables and scanned PDFs?

Use a parser that preserves table structure (convert to markdown/HTML), OCR or a vision-capable model for scans, keep each table as one chunk with its title, and test questions that depend on tables.

16. What is query rewriting / multi-query / HyDE?

Query rewriting: make the query clearer or standalone.
Multi-query: generate several versions of the question, search all, merge results.
HyDE: let the LLM write a hypothetical answer and use its embedding to search — helps when questions are short.

17. What if the answer needs live data (order status, account balance)?

Don't put live data in a vector database. Use tool/function calling to call your API or database, and combine with RAG for documents.

18. RAG vs fine-tuning vs long context?

RAG: large or changing knowledge, citations, permissions.
Fine-tuning: behaviour, style, format, narrow tasks.
Long context: small document sets that fit in the context window — simple, but costs more per request (prompt caching helps).

19. How do you reduce RAG latency?

Parallel retrieval (vector + keyword together), smaller k, fast reranker, caching embeddings and frequent answers, streaming the response, smaller model for query rewriting, keeping the vector index close to the app (same region).

20. Code — a simple RAG endpoint in Node.js (TypeScript)

import express from "express";
import Anthropic from "@anthropic-ai/sdk";
import { embed } from "./embeddings";          // your embedding provider wrapper
import { Chunk } from "./models/chunk";        // Mongoose model with an Atlas vector index

const anthropic = new Anthropic();
const app = express();
app.use(express.json());

app.post("/api/ask", requireAuth, async (req, res, next) => {
  try {
    const question = String(req.body.question ?? "").slice(0, 2000);
    if (!question) return res.status(400).json({ message: "Question is required" });

    const queryVector = await embed(question);
    const chunks = await Chunk.aggregate([
      {
        $vectorSearch: {
          index: "chunk_vector_index",
          path: "embedding",
          queryVector,
          numCandidates: 200,
          limit: 6,
          filter: { allowedGroups: { $in: req.user.groups } },   // permission filter
        },
      },
      { $project: { text: 1, docId: 1, title: 1, score: { $meta: "vectorSearchScore" } } },
    ]);

    const context = chunks
      .map((c) => `<doc id="${c.docId}" title="${c.title}">\n${c.text}\n</doc>`)
      .join("\n");

    const response = await anthropic.messages.create({
      model: "claude-sonnet-5-5",
      max_tokens: 800,
      system:
        "Answer using ONLY the documents provided. Cite sources as [doc id]. " +
        "If the documents don't contain the answer, say you couldn't find it.",
      messages: [{ role: "user", content: `<documents>\n${context}\n</documents>\n\nQuestion: ${question}` }],
    });

    const answer = response.content.find((b) => b.type === "text")?.text ?? "";
    res.json({ answer, sources: chunks.map((c) => ({ docId: c.docId, title: c.title })) });
  } catch (err) {
    next(err);
  }
});

21. What embedding model would you use?

Depends on language, domain and budget: provider embedding models (OpenAI, Cohere, Voyage, Google) or open-source ones (bge, e5, gte). Compare on your own retrieval test set; check multilingual support if users ask in Hindi or other languages; note the vector dimension (storage cost).

22. What are common RAG failure reasons?

Bad text extraction, poor chunking, missing exact keyword matches, wrong document version, too many irrelevant chunks, missing permission filters, the model ignoring context, and no evaluation set to catch regressions.

← LLM & Prompt Engineering Questions | Index | Next: Agents & Tool Calling Questions →