Chapter 96 — Fine-Tuning, LoRA & Parameter-Efficient Methods

📖 Definition

Fine-tuning continues training a pretrained model on your own data. Full fine-tuning updates every weight; parameter-efficient fine-tuning (PEFT) freezes the base model and trains a small number of extra parameters — most often LoRA.

🔍 Should You Fine-Tune at All? (Decision Order)

Senior interviewers want to hear that fine-tuning is not the first tool.

Fine-tuning teaches behaviour and style well; it teaches new facts poorly (and can increase hallucination). Facts belong in RAG.

🔍 LoRA — How It Works

Instead of learning a full update ΔW (size d × k) for a weight matrix, learn two small matrices:

W' = W + ΔW,    ΔW = (α / r) · B · A
W: frozen (d × k)     A: (r × k)     B: (d × r)     r ≪ d, k  (e.g. r = 8 or 16)

Trainable params per matrix: r · (d + k)  instead of  d · k
Example: d = k = 4096, r = 16 → 131,072 vs 16,777,216  (≈ 0.8%)

• B is initialised to zero, so training starts exactly from the base model.
• α (alpha) scales the update; common setting α = 2r or α = r.
• Apply LoRA to attention projections (q, k, v, o) and often the MLP layers — targeting all linear layers usually works best.
• After training, merge W + BA into the weights → zero extra inference latency. Or keep adapters separate and hot-swap many adapters on one base model (one adapter per customer/task).

💻 Code Example — A LoRA Linear Layer from Scratch

import math
import torch
from torch import nn

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r: int = 16, alpha: int = 32, dropout: float = 0.05):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False                         # freeze original weights
        self.A = nn.Parameter(torch.empty(r, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))   # zero → ΔW starts at 0
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.scale = alpha / r
        self.drop = nn.Dropout(dropout)

    def forward(self, x):
        return self.base(x) + (self.drop(x) @ self.A.T @ self.B.T) * self.scale

    @torch.no_grad()
    def merge(self):
        self.base.weight += (self.B @ self.A) * self.scale   # fold into base for inference

💻 Code Example — QLoRA with Hugging Face PEFT

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",            # 4-bit NormalFloat — suits normally-distributed weights
    bnb_4bit_use_double_quant=True,       # quantise the quantisation constants too
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model_id = "meta-llama/Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb, device_map="auto")
model = prepare_model_for_kbit_training(model)      # enables gradient checkpointing, casts norms

lora = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora)
model.print_trainable_parameters()   # e.g. trainable: ~42M of 8B (~0.5%)

QLoRA = base model frozen in 4-bit, LoRA adapters trained in bf16 → fine-tune an 8B model on a single 24 GB GPU, a 70B model on one 80 GB GPU (approximately).

📊 Fine-Tuning Methods Compared

🔍 Building the Dataset (Where Most Effort Goes)

1. Define the task precisely and write an eval set first (held out, never trained on).
2. Collect examples from real traffic, experts, or a strong model (distillation) — then review.
3. Use the model's exact chat template.
4. Balance: cover edge cases, refusals, multi-turn, different lengths.
5. Deduplicate and remove anything that overlaps with the eval set.
6. Mix in some general instruction data to reduce catastrophic forgetting.

{"messages": [
  {"role": "system", "content": "You are a support agent for Acme. Reply in under 80 words."},
  {"role": "user", "content": "My invoice shows the wrong GST number."},
  {"role": "assistant", "content": "Sorry about that! You can update your GST number under Settings → Billing → Tax details. New invoices will use it automatically; for past invoices, reply with the invoice IDs and we'll reissue them within 24 hours."}
]}

🔍 Hyperparameters That Matter

⚠️ Common Failure Modes

🔍 Evaluating a Fine-Tuned Model

• Task metrics on the held-out set (accuracy, exact match, F1, pass@k for code).
• Side-by-side pairwise comparison vs the base model + prompt (human or LLM judge).
• Regression suite on general abilities and safety.
• Online A/B test: task success, user ratings, escalations, cost and latency.

🧠 What a Senior Answer Adds

• Explains why fine-tuning (cost, latency, consistency) with numbers, and why not (facts → RAG).
• Treats the eval set and data quality as the main work.
• Plans serving: merged weights vs multi-adapter serving (e.g. vLLM/LoRAX with many LoRAs on one base).

🎯 Likely Interview Questions

1. When would you fine-tune instead of using RAG or prompting? — Style/format consistency, narrow tasks, smaller cheaper model, latency; not for injecting fresh facts.
2. Explain LoRA. — Freeze W, learn a low-rank update BA (rank r), scale by α/r; B initialised to zero; merge after training.
3. Why does low rank work? — Weight updates for adapting a pretrained model have low "intrinsic dimension".
4. What is QLoRA? — LoRA on top of a 4-bit (NF4) quantised frozen base with double quantisation and paged optimizers.
5. Does LoRA add inference latency? — Not if merged; separate adapters add a small overhead but allow hot-swapping.
6. How do you avoid catastrophic forgetting? — PEFT, low LR, few epochs, mix general data, regression evals.
7. How much data do you need? — Often hundreds to a few thousand high-quality examples for style/format; more for hard tasks.
8. How do you choose the LoRA rank? — Start 8–16; increase for complex tasks or lots of data; validate.
9. How do you evaluate a fine-tuned model? — Held-out task metrics, pairwise vs baseline, general/safety regression, online A/B.
10. Full fine-tuning vs LoRA memory? — ~16 bytes/param for full AdamW vs frozen weights (2 bytes, or 0.5 with 4-bit) + tiny adapter states.

← Pretraining & Alignment | Index | Next: Inference & Serving →