Chapter 100 — Evaluation, Experimentation & Responsible AI

📖 Definition

Evaluation is how you know a model or prompt change made things better. For GenAI it is the hardest part of the job, because outputs are open-ended, non-deterministic, and "good" depends on the task. Responsible AI covers safety, fairness, privacy, transparency and governance.

Senior mantra: "No eval, no ship." Every prompt, model or retrieval change runs through the same eval suite before it reaches users.

The AI / LLM Engineering: Evaluation & Hallucination chapter shows a minimal harness. This chapter is about designing an evaluation strategy for a team.

🔍 The Evaluation Stack

🔍 Building a Golden Dataset

1. Start from real usage (logs, support tickets) — synthetic data only fills gaps.
2. Cover slices: intents, languages, difficulty, edge cases, adversarial inputs, "should refuse" cases.
3. Write reference answers or rubrics with domain experts.
4. Size: 50–100 examples to start catching regressions; hundreds to thousands for fine-grained decisions.
5. Version the dataset; never train on it; add every production failure as a new test case.

📊 Metric Types

Prefer verifiable checks whenever possible (run the code, execute the SQL, check the citation exists).

🔍 LLM-as-Judge Done Right

Known biases and fixes:

Calibrate the judge: have humans label ~100–200 examples, measure agreement (accuracy, Cohen's kappa) with the judge, and iterate on the rubric until agreement is acceptable.

💻 Code Example — Pairwise Judge with Position Swap

import json

JUDGE_PROMPT = """You compare two answers to a user question.
Rubric: correctness (most important), completeness, follows instructions, concise.
Question: {q}
Reference (may be partial): {ref}
Answer A: {a}
Answer B: {b}
Reply with JSON: {{"winner": "A" | "B" | "tie", "reason": "<one sentence>"}}"""

def judge_once(llm, q, ref, a, b) -> str:
    out = llm(JUDGE_PROMPT.format(q=q, ref=ref, a=a, b=b), temperature=0)
    return json.loads(out)["winner"]

def pairwise(llm, q, ref, candidate, baseline) -> str:
    first = judge_once(llm, q, ref, candidate, baseline)    # candidate as A
    second = judge_once(llm, q, ref, baseline, candidate)   # swapped
    second = {"A": "B", "B": "A"}.get(second, "tie")        # map back to candidate = A
    return first if first == second else "tie"             # disagreement = position bias → tie

def win_rate(results: list[str]) -> float:
    wins = results.count("A") + 0.5 * results.count("tie")
    return wins / len(results)

🔍 Statistics for Evals (Often Skipped, Senior Signal)

• LLM outputs vary run to run → evaluate with multiple samples or temperature 0, and report confidence intervals (bootstrap).
• A 2% win-rate difference on 100 examples is usually noise.
• Use paired comparisons (same inputs for both systems) for more statistical power.
• Watch for Goodhart's law: once the eval becomes the target, people overfit to it — refresh and keep a hidden holdout.

🔍 Online Experimentation

Guardrail metrics stop you from "winning" on engagement while increasing cost, latency or unsafe outputs.

🔍 Hallucination — Senior View

Measure the hallucination rate on your own data; reduce by grounding, verification steps (generate → check → revise), lower temperature for factual tasks, and allowing "I don't know".

🔍 Safety & Security (Quick Recap)

Covered in depth in AI / LLM Engineering: AI Security. Senior summary:

• Prompt injection (direct and indirect via documents/web pages) — treat model output as untrusted; least-privilege tools; human approval for risky actions.
• Jailbreaks — layered defences: model-level safety training, input/output classifiers, policy prompts, monitoring.
• Data leakage — PII redaction, tenant isolation, no secrets in prompts, training-data opt-outs.
• Red teaming — internal + external, automated attack generation, track attack success rate over time.

🔍 Fairness & Bias

• Bias comes from data (who is represented), labels (annotator bias), and deployment (who uses it how).
• Measure performance per group/slice (language, region, dialect, gender where lawful and appropriate).
• Fairness definitions conflict (demographic parity vs equal opportunity vs calibration) — choose with stakeholders based on harm.
• Mitigations: data rebalancing, targeted examples, post-processing thresholds, human review for high-stakes decisions.

🔍 Governance & Compliance

🧠 What a Senior Answer Adds

• Designs an eval pipeline (golden set → CI → human calibration → canary → monitoring), not a one-off score.
• Knows judge biases and statistical noise; uses paired comparisons and confidence intervals.
• Balances product metrics with guardrail metrics and responsible-AI requirements.

🎯 Likely Interview Questions

1. How would you evaluate an LLM feature before launch? — Golden set with slices, programmatic checks, calibrated LLM judge, human review on risky slices, red teaming, then shadow/canary/A-B.
2. Biases of LLM-as-judge and fixes? — Position, verbosity, self-preference; swap order, rubrics, different judge, calibrate against humans.
3. Why are BLEU/ROUGE weak for LLM evaluation? — They measure word overlap, not correctness or helpfulness.
4. How do you know an improvement is real? — Paired evaluation, enough samples, confidence intervals, consistent across slices.
5. How do you measure hallucination? — Faithfulness checks against sources on a labelled set; claim-level verification; production sampling.
6. Offline vs online evaluation? — Offline catches regressions cheaply; online measures real impact with users.
7. What are guardrail metrics? — Metrics that must not get worse (latency, cost, safety incidents) while optimising the primary metric.
8. How do you build an eval dataset? — From real usage, cover slices and edge cases, expert references/rubrics, version it, keep it out of training.
9. What is red teaming? — Adversarially probing for harmful outputs, jailbreaks, data leaks, measuring attack success.
10. How do you handle fairness in an LLM product? — Slice-level evaluation, define harm-based fairness criteria, mitigate in data/prompts/policies, human oversight.
11. What goes in a model card? — Intended use, out-of-scope uses, training data summary, eval results per slice, limitations, safety considerations.

← Advanced RAG & Retrieval | Index | Next: MLOps & LLMOps →