Chapter 98 — Embeddings, Multimodal & Diffusion Models

📖 Definitions

• Embedding model — maps text (or images/audio) to a vector so that similar meanings are close together.
• Multimodal model — understands and/or generates more than one modality (text + images, audio, video).
• Diffusion model — generates images (or audio/video) by learning to remove noise step by step.

Senior GenAI roles increasingly include vision and audio, so expect at least a few questions here.

🔍 How Embedding Models Are Trained (Contrastive Learning)

1. Build pairs that should be similar: (question, answer), (query, clicked document), (caption, image), two paraphrases.
2. Encode both sides into vectors.
3. Pull matching pairs together and push non-matching ones apart using in-batch negatives — every other item in the batch acts as a negative.

InfoNCE loss for one query q with positive d⁺ and batch items d₁…d_N:
loss = −log(  exp(sim(q, d⁺)/τ)  /  Σ_j exp(sim(q, d_j)/τ)  )
sim = cosine similarity,  τ = temperature (small, e.g. 0.05)

• Hard negatives (documents that look relevant but aren't) make embeddings much sharper.
• Bigger batches = more negatives = better embeddings (why contrastive training uses huge batches).

💻 Code Example — Contrastive (InfoNCE) Loss in PyTorch

import torch
import torch.nn.functional as F

def info_nce(query_emb, doc_emb, temperature=0.05):
    """query_emb, doc_emb: (B, D). Row i of docs is the positive for row i of queries."""
    q = F.normalize(query_emb, dim=-1)
    d = F.normalize(doc_emb, dim=-1)
    logits = q @ d.T / temperature               # (B, B) cosine similarities
    labels = torch.arange(q.size(0), device=q.device)   # diagonal = positives
    return F.cross_entropy(logits, labels)       # other columns are in-batch negatives

📊 Bi-Encoder vs Cross-Encoder

Matryoshka embeddings: trained so the first 256 dimensions of a 1024-dim vector still work well → cheaper storage and search, choose size per use case.

🔍 Vision Transformers & CLIP

• ViT: split an image into patches (e.g. 16×16 pixels), flatten each patch into a vector ("visual token"), add positions, run a transformer.
• CLIP: an image encoder and a text encoder trained contrastively on hundreds of millions of (image, caption) pairs → images and text share one embedding space.
  • Zero-shot classification: compare an image embedding with "a photo of a {cat, dog, car}" text embeddings.
  • Image search by text, and the text encoder conditions many diffusion models.

🔍 How Vision-Language Models (VLMs) Work

1. A vision encoder (ViT/CLIP-style) turns the image into patch embeddings.
2. A projector/adapter (MLP or cross-attention) maps them into the LLM's token embedding space.
3. The LLM reads image tokens + text tokens together and generates text.
4. Training: first align (train projector on image–caption pairs), then instruction-tune on visual Q&A.

Cost note: one image can be hundreds to thousands of tokens — resize/crop images to control cost and latency.

🔍 Audio

• Speech-to-text (ASR): e.g. Whisper — encoder–decoder transformer on log-mel spectrograms; robust multilingual transcription.
• Text-to-speech (TTS): neural acoustic models + vocoders, or token-based audio LMs.
• Voice agents: ASR → LLM → TTS pipeline (watch total latency: each stage adds delay; stream everything) or native speech-to-speech models.

🔍 Diffusion Models in Simple Steps

Training

1. Take a real image x₀.
2. Pick a random timestep t and add the matching amount of Gaussian noise → x_t.
3. Train a network to predict the noise that was added (given x_t, t, and the text prompt).
4. Loss = mean squared error between predicted noise and real noise.

Generation (sampling)

1. Start from pure random noise.
2. Repeatedly ask the network "what noise is in this?" and remove a bit of it.
3. After N steps (20–50 with modern samplers like DDIM/DPM-Solver), you get an image.

# Training step (simplified DDPM)
t = torch.randint(0, T, (batch,))
noise = torch.randn_like(x0)
x_t = sqrt_alpha_bar[t] * x0 + sqrt_one_minus_alpha_bar[t] * noise   # forward noising, closed form
pred = unet(x_t, t, text_emb)
loss = F.mse_loss(pred, noise)

🔍 Latent Diffusion & Guidance

• Latent diffusion (Stable Diffusion): a VAE compresses a 512×512 image into a small latent (e.g. 64×64×4); diffusion runs in the latent space → ~50× cheaper; the VAE decoder turns the result back into pixels.
• Text conditioning: text embeddings (CLIP/T5) injected via cross-attention.
• Classifier-free guidance (CFG): run the model with and without the prompt and push the result toward the prompt: noise = uncond + s × (cond − uncond) — higher s = closer to prompt, less diversity (typical 5–8).
• Newer models use diffusion transformers (DiT) instead of U-Nets, and flow matching objectives; distillation (consistency / turbo models) cuts sampling to 1–4 steps.
• Control: ControlNet (edges, pose, depth), LoRA for styles/subjects, inpainting (regenerate a masked region).

📊 Generative Model Families

🔍 Evaluating Generative Media

• FID (Fréchet Inception Distance): distance between feature distributions of real and generated images — lower is better.
• CLIP score: how well an image matches its prompt.
• Human preference studies remain the gold standard.
• Safety: NSFW filters, watermarking / content credentials (C2PA), likeness and copyright policies.

🧠 What a Senior Answer Adds

• Uses bi-encoder for recall, cross-encoder for precision, and knows why.
• Can explain diffusion training as "predict the noise" and sampling as "iterative denoising" in under a minute.
• Thinks about cost per image/minute of audio and latency budgets in multimodal pipelines.

🎯 Likely Interview Questions

1. How are embedding models trained? — Contrastive learning on positive pairs with in-batch and hard negatives (InfoNCE).
2. Bi-encoder vs cross-encoder? — Separate encoding (fast, scalable retrieval) vs joint encoding (accurate reranking).
3. What are hard negatives and why do they matter? — Similar-looking but wrong examples; they teach fine distinctions.
4. How does CLIP work and what is it used for? — Contrastive image–text training into one space; zero-shot classification, search, diffusion conditioning.
5. How does a VLM process an image? — Vision encoder → projector into token space → LLM attends to image tokens with text.
6. Explain diffusion training and sampling. — Add noise, learn to predict it; generate by iteratively denoising from noise.
7. Why latent diffusion? — Diffuse in a compressed VAE latent → far cheaper than pixel space.
8. What is classifier-free guidance? — Combine conditional and unconditional predictions to strengthen prompt adherence.
9. GAN vs diffusion? — GANs fast but unstable/mode collapse; diffusion stable and diverse but needs many steps.
10. How do you evaluate image generation? — FID, CLIP score, human preference, safety checks.
11. What are Matryoshka embeddings? — Vectors whose prefixes are usable embeddings → trade dimension for cost.

← Inference & Serving | Index | Next: Advanced RAG & Retrieval →