Chapter 101 — MLOps & LLMOps in Production

📖 Definition

MLOps is the engineering discipline of taking ML models from notebooks to reliable production systems: reproducible data and training, automated deployment, monitoring and retraining. LLMOps applies the same ideas to LLM applications, where the "model" is often a prompt + retrieval + tools + a hosted or self-served LLM.

🔍 The ML Lifecycle

🔍 Training–Serving Skew (Classic Senior Question)

The model sees features computed one way in training (SQL over the warehouse) and another way in production (Python in the API) → silent accuracy drop.

Fixes:

• Single feature definition used for both (feature store, shared library).
• Log features at serving time and train on those logs.
• Compare training and serving feature distributions continuously.
• Point-in-time correct joins for training data (only use values that existed at that timestamp).

💻 Code Example — Experiment Tracking with MLflow

import mlflow
import mlflow.sklearn
from sklearn.metrics import average_precision_score

mlflow.set_experiment("churn-model")

with mlflow.start_run(run_name="lgbm-v3"):
    params = {"num_leaves": 63, "learning_rate": 0.05, "n_estimators": 800}
    mlflow.log_params(params)
    mlflow.log_param("data_version", "s3://ml-data/churn/2026-10-01")   # data lineage

    model = train_lgbm(X_train, y_train, **params)
    pr_auc = average_precision_score(y_val, model.predict_proba(X_val)[:, 1])
    mlflow.log_metric("val_pr_auc", pr_auc)

    mlflow.sklearn.log_model(model, "model", registered_model_name="churn")   # → model registry

Reproducibility = code commit + data version + config + environment (container) + random seed.

📊 Deployment Patterns

Release strategies: shadow → canary → A/B → full rollout, with automatic rollback on guardrail breaches. Keep the previous model "warm" for instant rollback.

🔍 Monitoring: What to Watch

import numpy as np

def psi(expected: np.ndarray, actual: np.ndarray, bins: int = 10) -> float:
    """Population Stability Index. < 0.1 stable, 0.1–0.25 moderate shift, > 0.25 significant."""
    cuts = np.quantile(expected, np.linspace(0, 1, bins + 1))
    cuts[0], cuts[-1] = -np.inf, np.inf
    e = np.histogram(expected, cuts)[0] / len(expected)
    a = np.histogram(actual, cuts)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return float(np.sum((a - e) * np.log(a / e)))

🔍 Retraining Strategy

• Scheduled (weekly/monthly) — simple, predictable.
• Triggered — on drift alerts or performance drop.
• Continuous / online learning — for fast-changing domains (ads, fraud), with strong safeguards.
• Always: retrain → evaluate vs current champion → promote only if better (champion/challenger).

🔍 What Changes with LLMOps

💻 Code Example — Prompt as a Versioned Artifact

# prompts/support_answer/v7.yaml is checked into git and reviewed like code
PROMPT_REGISTRY = {
    "support_answer": {
        "version": "v7",
        "model": "claude-sonnet-5-5",
        "temperature": 0.2,
        "max_tokens": 600,
        "template_path": "prompts/support_answer/v7.yaml",
        "eval_suite": "evals/support_answer.jsonl",
        "min_scores": {"faithfulness": 0.95, "resolution_judge": 0.80},
    }
}

def run(name: str, variables: dict, trace) -> str:
    cfg = PROMPT_REGISTRY[name]
    prompt = render(cfg["template_path"], variables)
    with trace.span("llm", prompt_name=name, prompt_version=cfg["version"], model=cfg["model"]) as s:
        out = llm_call(cfg["model"], prompt, temperature=cfg["temperature"], max_tokens=cfg["max_tokens"])
        s.set(tokens_in=out.usage.input_tokens, tokens_out=out.usage.output_tokens)
    return out.text

CI runs the eval suite when the prompt or model changes and blocks the merge if scores fall below min_scores.

🔍 LLM Observability: What to Log per Request

• Request ID, user/tenant (hashed), timestamp, feature/route.
• Model, prompt version, parameters.
• Retrieved document IDs and scores; tool calls and results.
• Input/output tokens, latency per step, cache hits, cost.
• User feedback (thumbs, edits, escalation), safety-filter results.
• PII policy: redact or don't store raw content where not allowed; retention limits.

🔍 Cost Management

• Track cost per request / per feature / per tenant; alert on spikes.
• Model routing (small model first, escalate when needed).
• Prompt caching, response caching, batch APIs for offline work.
• Budgets and rate limits per user/tenant.
• Self-hosting break-even: compare API cost vs GPU cost at your utilisation (idle GPUs are expensive).

🔍 Build vs Buy (Hosted API vs Self-Hosted Model)

🧠 What a Senior Answer Adds

• Prevents training–serving skew by design and logs features at serving time.
• Defines ownership: alerts, on-call, rollback, retraining triggers.
• For LLMs, versions everything that changes behaviour (prompt, model, retrieval, tools) and gates releases on evals.

🎯 Likely Interview Questions

1. What is MLOps and why is it needed? — Practices to deploy, monitor and maintain ML reliably; models decay and need reproducibility and automation.
2. What is training–serving skew and how do you prevent it? — Different feature computation offline vs online; shared feature definitions, feature store, logging serving features.
3. Data drift vs concept drift? — Inputs change vs the input–output relationship changes.
4. How do you detect drift? — PSI/KS/KL on features and predictions, performance once labels arrive, alerts with thresholds.
5. How do you deploy a new model safely? — Registry approval → shadow → canary → A/B → rollout with automatic rollback.
6. What do you version in an ML project? — Code, data, features, configs, models, environment; for LLMs also prompts, retrieval config and tools.
7. Batch vs real-time inference? — Precomputed periodic scores vs on-request predictions; choose by latency need and freshness.
8. What is a feature store? — Central place for feature definitions with offline (training) and online (serving) stores and point-in-time correctness.
9. What would you monitor for an LLM app? — Quality evals/sampled grading, user feedback, hallucination/safety flags, latency, tokens/cost, tool errors, retrieval quality.
10. When would you self-host an open model instead of using an API? — Data control, high steady volume, deep customisation, latency/edge constraints — weigh against ops burden.
11. What is champion/challenger? — Current production model vs candidate evaluated on the same data/traffic; promote only if better.

← Evaluation & Responsible AI | Index | Next: GenAI System Design →