Chapter 110 — GenAI Interview: Evaluation, Security, Cost & Production Questions

📖 Why These Questions Matter

Building a demo is easy; running GenAI reliably in production is what companies pay for. These questions check whether you can measure quality, protect data, control cost and operate an LLM feature.

📏 Evaluation

1. How do you evaluate an LLM application?

1. Build a test set of real inputs with expected outputs or rules (50–300 to start).
2. Score with:
   • Code checks where possible (valid JSON, correct fields, exact values, SQL result matches).
   • LLM-as-judge with a clear rubric for open-ended answers (correctness, faithfulness, tone).
   • Human review for a sample and high-risk cases.
3. Run the test set on every prompt/model change and compare with the previous version.
4. In production: user feedback, sampled reviews, task success metrics.

2. What is LLM-as-a-judge? Any problems?

Using an LLM to grade outputs against a rubric. Problems: prefers longer answers, prefers the first answer in comparisons, may favour its own model's style, can be inconsistent. Fixes: specific rubric, swap answer order, compare with human labels on a sample, use a strong judge model.

3. Which metrics do you track?

4. How do you know a new prompt is better than the old one?

Run both on the same test set, compare scores (and pairwise judge preferences), check important slices (e.g. Hindi questions, long documents), and only ship if it improves without breaking other cases. Then roll out gradually and watch production metrics.

5. Tools for evaluation and tracing?

LangSmith, Langfuse, Ragas, DeepEval, Promptfoo, Arize Phoenix, or custom scripts + a spreadsheet/dashboard. Mention what you actually used.

🔐 Security & Privacy

6. What are the main security risks in GenAI apps?

• Prompt injection (direct and through documents/web pages/emails).
• Data leakage — users seeing other users' data, system prompt leaks, PII in logs.
• Excessive agency — agent performs harmful actions.
• Insecure output handling — rendering model output as HTML (XSS), running generated SQL/code.
• Denial of wallet — abuse causing huge token bills.
• Reference: OWASP Top 10 for LLM Applications.

7. How do you prevent prompt injection?

You can't fully prevent it, so limit damage:

• Separate instructions from data with tags; tell the model data is not instructions.
• Authorization in code, least-privilege tools, human approval for risky actions.
• Don't render model output as raw HTML; validate structured outputs.
• Input/output filters and monitoring; test with known attack prompts.

8. How do you handle PII?

Collect only what's needed, mask/redact PII before sending to the LLM where possible, mask in logs, use enterprise API settings (no training on data, limited retention), keep data in the required region, restrict log access, and follow DPDP/GDPR rules for deletion.

9. Is it safe to send company data to OpenAI/Anthropic/Gemini APIs?

With enterprise/API terms, providers typically don't train on API data and offer data-retention controls, regional processing and compliance certifications — but the client's policy decides. Options for stricter needs: cloud-hosted models in the client's own account (Azure OpenAI, AWS Bedrock, Google Vertex AI) or self-hosted open-source models.

10. How do you stop the chatbot from answering off-topic or harmful questions?

System prompt with clear scope, a classifier/moderation check for unsafe inputs, polite refusal templates, output moderation, and tests for off-topic and harmful prompts.

💰 Cost & Performance

11. How do you estimate LLM cost?

Monthly cost = requests per month × (input tokens × input price + output tokens × output price)

Example: 100,000 questions/month, 2,500 input tokens + 300 output tokens each,
input $3 per 1M tokens, output $15 per 1M tokens (illustrative prices)
Input:  100,000 × 2,500 = 250M tokens → $750
Output: 100,000 × 300   = 30M tokens  → $450
Total ≈ $1,200/month before caching and routing optimisations

12. How do you reduce cost?

• Smaller/cheaper models for simple tasks (routing).
• Prompt caching for repeated system prompts and documents.
• Fewer, better chunks in RAG.
• Shorter outputs (max_tokens, "be concise").
• Cache answers to repeated questions (exact or semantic cache).
• Batch API for non-urgent jobs (often ~50% cheaper).
• Per-user rate limits and budgets.

13. How do you reduce latency?

Streaming responses, smaller models for intermediate steps, parallel calls, prompt caching, shorter prompts, keeping services in the same region, and precomputing where possible.

14. How do you handle API rate limits and outages?

Retries with exponential backoff and jitter for 429/5xx, request queues for spikes, fallback to another model/provider/region, timeouts, circuit breakers, and a graceful message to users ("Assistant is busy, please try again").

🛠️ Production & Operations

15. What do you log for each LLM request?

Request ID, user/tenant (hashed if needed), feature, model and prompt version, retrieved document IDs, tool calls, tokens in/out, latency, cost, errors, user feedback — with PII masked.

16. How do you deploy a prompt or model change safely?

Version prompts in Git → run the evaluation suite in CI → deploy behind a feature flag → roll out to a small % of users → monitor quality, cost and errors → full rollout or rollback.

17. What happens when the provider releases a new model version?

Pin model versions in config; when a new version comes, run the evaluation suite, compare quality/cost/latency, then roll out gradually. Plan for model retirement dates.

18. How do you collect and use user feedback?

Thumbs up/down with optional comment, track edits users make to AI drafts, review negative feedback weekly, add failing cases to the evaluation set, and fix prompts/retrieval/data.

19. How do you stream LLM responses in a MERN app?

Backend: Server-Sent Events (SSE) from Express — set Content-Type: text/event-stream, forward tokens as they arrive from the provider's streaming API, handle client disconnects. Frontend: read the stream with fetch + ReadableStream (or EventSource for GET), append text to state as it arrives.

20. How would you build a GenAI feature that must be 99.9% available?

Multi-region or multi-provider fallback, timeouts and retries, queue-based processing for non-interactive tasks, health checks, graceful degradation (show search results without AI summary), monitoring and alerting, and load testing at expected peaks.

← Agents & Tool Calling Questions | Index | Next: Scenario & Design Questions →