Chapter 93 — Deep Learning & Training Dynamics

📖 Definition

A neural network is a stack of layers; each layer does a linear transform (W·x + b) followed by a non-linear activation. Training repeats four steps:

1. Forward pass — compute predictions.
2. Loss — measure the error (cross-entropy for classification/next-token, MSE for regression).
3. Backward pass (backpropagation) — use the chain rule to compute the gradient of the loss for every parameter.
4. Optimizer step — move parameters a little against the gradient.

🔍 Backpropagation in One Paragraph

The loss depends on the output, which depends on the last layer, which depends on the previous layer, and so on. The chain rule lets us compute ∂loss/∂weight for every weight by passing gradients backwards layer by layer, reusing intermediate results from the forward pass. That's why training needs memory for activations — they are stored during the forward pass for use in the backward pass.

💻 Code Example — A Training Loop in PyTorch

import torch
from torch import nn

model = nn.Sequential(
    nn.Linear(784, 256), nn.GELU(), nn.Dropout(0.1),
    nn.Linear(256, 10),
)
loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=3e-4, total_steps=10_000)

for step, (x, y) in enumerate(train_loader):
    model.train()
    logits = model(x)                    # 1. forward
    loss = loss_fn(logits, y)            # 2. loss
    opt.zero_grad(set_to_none=True)
    loss.backward()                      # 3. backward
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)   # stability
    opt.step()                           # 4. update
    sched.step()

    if step % 500 == 0:
        model.eval()
        with torch.no_grad():
            val_loss = evaluate(model, val_loader)

model.train() / model.eval() matter: dropout and batch norm behave differently in each mode.

📊 Activation Functions

📊 Optimizers

Adam vs AdamW (common senior question): in Adam, L2 regularisation added to the loss gets rescaled by the adaptive learning rate, so it doesn't behave like real weight decay. AdamW applies weight decay directly to the weights, separately from the gradient step — better generalisation.

Adam keeps two extra values per parameter (first and second moment), which is why optimizer memory is large for LLMs.

🔍 Learning Rate & Schedules

The learning rate is the most important hyperparameter.

• Too high → loss explodes or oscillates. Too low → slow, stuck.
• Warmup: start small and ramp up over the first ~1–5% of steps (Adam's estimates are noisy at the start).
• Cosine decay (or linear decay) down to ~10% of peak afterwards.
• Bigger batch → can usually use a bigger learning rate.
• Fine-tuning uses a much smaller LR than pretraining (e.g. 1e-5 to 2e-4).

🔍 Normalisation

Pre-LN vs Post-LN: modern transformers put normalisation before attention/FFN (pre-LN), which makes deep models train stably without careful warmup.

🔍 Vanishing & Exploding Gradients

Residual connections (x + layer(x)) give gradients a "highway" straight back through the network — the reason 100-layer transformers train at all.

Initialisation: Xavier/Glorot (tanh/sigmoid), He/Kaiming (ReLU) keep activation variance stable across layers.

🔍 Mixed Precision Training

scaler = torch.amp.GradScaler()          # only needed for fp16, not bf16
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
    loss = loss_fn(model(x), y)
loss.backward()

🔍 Memory Tricks for Big Models

🔍 Reading Loss Curves

Debugging tip: "Can the model overfit 10 examples to ~0 loss?" If not, there's a bug in data, labels or code — not a modelling problem.

🔍 CNNs and RNNs (Short, for Context)

• CNN: shares small filters across an image → translation-invariant features; still used for vision (ResNet) though Vision Transformers compete at scale.
• RNN / LSTM / GRU: process sequences step by step with a hidden state; hard to parallelise and weak on long-range dependencies → replaced by transformers for language.

🧠 What a Senior Answer Adds

• Explains why an optimizer or norm is chosen (stability, memory), not only what it is.
• Knows memory budgets: weights + gradients + optimizer states + activations.
• Debugs systematically: overfit a small batch, check data, then tune.

🎯 Likely Interview Questions

1. Explain backpropagation. — Chain rule applied backwards through layers to get each parameter's gradient, reusing forward activations.
2. Why do we need non-linear activations? — Without them, stacked linear layers collapse into one linear function.
3. Adam vs AdamW? — AdamW decouples weight decay from the adaptive gradient update — correct regularisation, standard for transformers.
4. Why use learning-rate warmup? — Early Adam statistics are noisy and big early updates destabilise training.
5. Batch norm vs layer norm? — Across the batch vs across features of one example; transformers use layer norm/RMSNorm.
6. What causes vanishing gradients and how do you fix them? — Repeated small derivatives; ReLU-family, residuals, normalisation, init.
7. What does gradient clipping do? — Caps the gradient norm to prevent exploding updates.
8. BF16 vs FP16? — Same 16 bits; BF16 keeps FP32's exponent range so no loss scaling is needed.
9. What is gradient checkpointing? — Don't store all activations; recompute them in backward — trades compute for memory.
10. How does dropout work at inference? — It's turned off; activations are scaled during training so expected values match.
11. Your loss became NaN — what do you check? — LR, precision overflow, bad inputs/labels, division by zero, clip gradients.
12. Why residual connections? — Gradient highway; lets very deep networks train.

← ML Fundamentals | Index | Next: Transformer Architecture →