Chapter 102 — GenAI System Design Interviews

📖 What This Round Tests

"Design an AI assistant for X" is now a standard senior round. The interviewer checks that you can turn a vague product idea into a reliable, safe, measurable, affordable system — not just "call an LLM".

🔍 A Framework You Can Reuse (45–60 minutes)

Start simple (prompt + RAG + evals), then add agents, fine-tuning or self-hosting only with a reason.

🔍 Back-of-the-Envelope Estimates

Users: 50,000 daily active employees, 8 questions/day        → 400,000 requests/day
Peak QPS ≈ 400,000 / (8 work hours × 3600) × 3 (peak factor)  ≈ 42 requests/s

Tokens per request: 3,000 input (system + context) + 400 output
Daily tokens: 400k × 3.4k ≈ 1.36 B tokens/day

Cost (illustrative prices): input $3 / 1M, output $15 / 1M
  input  : 400k × 3,000 = 1.2 B → $3,600/day
  output : 400k × 400   = 160 M → $2,400/day          ≈ $6,000/day ≈ $180k/month
With 70% prompt-cache hits on the static 1,500-token prefix and routing 60% of queries to a model 5× cheaper,
the bill drops by well over half → this is why cost design is part of the interview.

Always state that prices are assumptions and show which levers change the number.

🏗️ Design 1 — Enterprise Knowledge Assistant ("Chat with company docs")

Requirements: 50k employees; Confluence, Google Drive, PDFs, tickets; answers with citations; respects document permissions; p95 TTFT < 2 s; English + Hindi.

Request flow

1. User asks in Slack/web → API gateway (auth via SSO, rate limit).
2. Conversation service loads the last turns; query rewriter (small model) makes a standalone query and extracts filters (team, date).
3. Retriever runs hybrid search (vector + BM25) filtered by the user's ACL groups, returns top 50.
4. Reranker (cross-encoder) keeps top 8; dedupe; build context with source IDs.
5. Generator (strong LLM) answers using only context, with citations; says "not found" when unsupported.
6. Post-checks: citation validation, PII/output filter; stream to the user.
7. Log trace (query, doc IDs, tokens, latency, feedback).

Ingestion pipeline

• Connectors poll/webhook for changes → parse → structure-aware chunking with title/section headers → embeddings → upsert into vector + keyword index with ACL metadata.
• Deletions and permission changes propagate within minutes (security requirement).
• Re-embed everything in the background when changing embedding models (dual-index cutover).

Key decisions & trade-offs

• Hybrid + reranker over vector-only (acronyms, ticket IDs).
• Permission pre-filtering in the retriever, never in the prompt.
• Small model for rewriting/routing; strong model for answers; prompt caching of the system prompt.
• Multilingual embedding model; answer in the user's language.

Evaluation: golden set of 300 real questions per department with source docs; recall@10, faithfulness, citation accuracy, refusal correctness; weekly human review; thumbs-up rate and "ticket deflection" online.

Failure handling: if retrieval returns nothing relevant → say so and suggest a contact/channel; LLM timeout → fallback model; index lag → show "last updated" dates.

🏗️ Design 2 — Customer Support Agent with Actions

Requirements: answer order questions, process refunds under ₹5,000 automatically, escalate otherwise; 1M chats/month; must never leak another customer's data.

Components

1. Intent router (small model or classifier): FAQ, order status, refund, complaint, other.
2. FAQ path: RAG over help-center articles.
3. Action path: an agent with tools — get_order(order_id), get_refund_eligibility(order_id), create_refund(order_id, amount), escalate(reason).
4. Policy layer outside the LLM: tools check that the order belongs to the authenticated customer; create_refund enforces the ₹5,000 limit and eligibility in code; amounts above require human approval.
5. Human handoff with conversation summary for the agent.
6. Conversation memory: session state in Redis; summary of long chats.

Safety: authorization in tool handlers (not prompts), idempotency keys on refunds, rate limits, audit log of every action, prompt-injection defences (customer text is data, not instructions), PII redaction in logs.

Metrics: resolution rate without human, CSAT, wrong-action rate (must be ~0), escalation quality, cost per conversation, average handle time saved.

Rollout: start read-only (answers + order status) → add refunds behind human approval → automate small refunds after measuring accuracy.

🏗️ Design 3 — AI Code Review Assistant

Requirements: comment on pull requests in a monorepo; low false-positive rate; runs within 5 minutes; code must not leave the company's cloud.

Flow

1. Webhook on PR → queue (handles bursts).
2. Context builder: diff + changed files + related files (imports, callers via a code index/LSP) + repo guidelines.
3. Static checks first (linters, type-check, tests) — cheap and certain.
4. LLM review per file/chunk with a focused rubric (bugs, security, performance); ask for line-anchored findings with confidence.
5. Verifier pass: a second prompt (or tests) checks each finding; drop low-confidence ones.
6. Post top-N comments; collect 👍/👎 and "resolved" signals.

Decisions: self-hosted or VPC-hosted model for data residency; cache embeddings of the repo; limit to N comments to avoid noise; measure precision (accepted comments / posted) as the north-star.

🏗️ Design 4 — Document Extraction Pipeline (Invoices / KYC)

Requirements: 200k documents/day, extract 30 fields into JSON, 99% field accuracy for amounts, human review for low confidence.

Flow: upload → OCR / VLM page reading → LLM extraction with structured output schema → validation rules (totals add up, GST format, dates valid) → confidence scoring → auto-accept or human review queue → corrected examples feed back into prompts/fine-tuning.

Cost: batch API for non-urgent docs; small model for easy layouts, big model for hard ones; caching per template/vendor.

📊 Common Building Blocks Cheat Sheet

🧠 What a Senior Answer Adds

• Clarifies success metrics and constraints before drawing anything.
• Puts security and correctness in code, not in prompts.
• Gives numbers (QPS, tokens, cost) and names the levers.
• Proposes a phased rollout with evaluation gates.

🎯 Likely Interview Questions

1. Design a chatbot over company documents. — Design 1: ingestion with ACLs, hybrid retrieval + rerank, cited generation, evals, permissions.
2. How do you stop an agent from taking harmful actions? — Authorization and limits in tool code, human approval, idempotency, audit logs, least privilege.
3. Agent or workflow? — Workflow when steps are known (cheaper, predictable); agent when the path depends on intermediate results.
4. How do you estimate cost for an LLM feature? — Requests × (input tokens × price + output tokens × price), adjusted for caching and routing.
5. How do you reduce latency? — Streaming, smaller models for sub-steps, parallel calls, prefix caching, shorter context.
6. How do you handle model provider outages? — Fallback models/providers, timeouts, circuit breakers, graceful degradation.
7. How would you roll out an AI feature safely? — Internal dogfood → shadow → limited beta → canary → GA, with eval gates and kill switch.
8. How do you keep the knowledge base fresh? — Change-data connectors, incremental indexing, deletion propagation, freshness metadata.
9. How do you measure success of a support bot? — Resolution rate, CSAT, wrong-action rate, escalation quality, cost per conversation.
10. What would you build first? — The simplest end-to-end path with an eval set and logging; iterate from measured failures.

← MLOps & LLMOps | Index | Next: Classic ML System Design →