Appendix: Interview Question Bank

Fundamentals

Tokenization

  1. Why do modern LLMs use subword tokenization instead of word-level or character-level tokenization?
  2. Compare BPE, WordPiece, and Unigram LM tokenizers—which should you choose in which setting?
  3. What is byte-level BPE (BBPE)? Why is it better suited to multilingual models?
  4. How do you trade off vocabulary size against compute and quality?
  5. Your biomedical domain terms are being split into 8+ subword pieces—what would you do? What are the risks?
  6. What is a tokenizer’s “fragmentation rate”? Why does it matter for cost and latency?
  7. A teammate swapped the tokenizer between training and inference—what breaks?
  8. Why can’t you directly compare perplexity across models that use different tokenizers?
  9. How does tokenization affect arithmetic and numerical reasoning in LLMs?
  10. What are the risks of extending a tokenizer midway through training? How do you mitigate them?
  11. What is the relationship between token count and API cost? How do you optimize it in cost-sensitive deployments?
  12. How do SentencePiece and HuggingFace Tokenizers differ in how they handle whitespace and unknown characters?
  13. You are building a multilingual model (English, Chinese, Arabic, code). Walk through your tokenizer design decisions—vocabulary size, algorithm choice, byte fallback, special tokens.
  14. A model that is good at code suddenly fails when users paste markdown fenced with backticks—what tokenization issue might be responsible?

Embeddings

  1. Why can’t a model use token IDs directly as numbers—why does it need embeddings?
  2. Explain embedding lookup—how is it equivalent to one-hot vectors × matrix multiplication?
  3. What is “weight tying” (input embeddings = output softmax weights)? Why do it?
  4. What is the difference between word/token embeddings and sentence embeddings?
  5. Compare CBOW and Skip-gram—when is Skip-gram better?
  6. How does FastText handle out-of-vocabulary words?
  7. What is negative sampling? Why do you need it to train large-scale word2vec?
  8. How would you evaluate the quality of learned embeddings?
  9. Explain contextual embeddings. How do BERT-style models differ from static word2vec/GloVe embeddings?
  10. What is the anisotropy problem in embedding spaces—why do modern LLM embeddings cluster into a narrow cone? How do you fix it?
  11. How do cross-modal embeddings (such as CLIP) align text and images in a shared space? What is contrastive loss?
  12. You are choosing an embedding model for a production RAG system—what dimensions do you evaluate (dimension size, latency, domain coverage, MTEB score)?

Attention

  1. Explain scaled dot-product attention step by step. Why divide by √d_k?
  2. What is a causal mask? Why is it necessary for decoder-only LLMs?
  3. Why is vanilla self-attention O(L²) in sequence length? What are the practical consequences?
  4. Explain multi-head attention (MHA). Why use multiple heads instead of one big head?
  5. What is cross-attention? Where is it used?
  6. Compare MHA, MQA, and GQA—how does each affect the KV cache and quality?
  7. Explain the KV cache. Why do you need it in autoregressive decoding?
  8. How does FlashAttention improve performance without changing the mathematics of attention?
  9. What is sparse attention (for example, sliding window attention)? What do you gain, and what do you lose?
  10. How do you turn vanilla softmax attention into “linear attention”? What are the tradeoffs?
  11. Why does long-context prefill dominate time to first token (TTFT)?
  12. You increase the context window from 4K to 128K—what breaks? What becomes expensive?
  13. Derive the memory cost of the KV cache for a single Transformer layer as a function of batch size, sequence length, number of KV heads, and head dimension.
  14. What is multi-head latent attention (MLA), as used in DeepSeek V2? How does it compress the KV cache even more aggressively than GQA?
  15. Explain Ring Attention / Striped Attention. How does it distribute long sequences across multiple GPUs during training?

FFN, Activations, Residual Connections, Normalization

  1. If attention already mixes information across tokens, why does a Transformer still need an FFN/MLP?
  2. Compare ReLU, GELU, Swish, and SwiGLU—why do modern LLMs prefer gated activations?
  3. What is the “dying ReLU” problem?
  4. Why do residual (skip) connections help train deep networks?
  5. Compare pre-norm and post-norm Transformers—which is more stable at depth, and why?
  6. Compare LayerNorm and RMSNorm. Why do modern LLMs prefer RMSNorm?
  7. What is DeepNorm? When do you need it?
  8. What would happen if you removed all residual connections from a 70-layer Transformer?
  9. FFNs typically use a hidden dimension that is 4× the model dimension (or 8/3× in SwiGLU). Why that ratio? What are the memory and compute implications?
  10. What is the “residual stream” view of a Transformer? How does it help you reason about feature circuits and superposition?

Positional Encoding

  1. Why do Transformers need positional encoding? What happens without it?
  2. Explain sinusoidal positional encoding. Why use sin/cos at different frequencies?
  3. What is RoPE (rotary position embedding)? Give the intuition in one sentence.
  4. Compare absolute position, relative position, RoPE, and ALiBi—state the difference in one sentence.
  5. What is length extrapolation? Why do some positional schemes degrade on longer contexts?
  6. How do NTK-aware scaling and YaRN extend RoPE to longer contexts?
  7. Your model was trained on 8K context, but users need 128K—what options do you have?
  8. Why do some models (such as ALiBi) claim they do not need positional encoding at all? What are they actually doing?

Decoding Strategies

  1. Compare greedy search, beam search, top-k, top-p (nucleus sampling), and temperature sampling.
  2. What does temperature do mathematically? What goes wrong at very low temperature and very high temperature?
  3. Why can beam search reduce diversity and produce bland outputs?
  4. Top-k vs top-p—which is more robust across prompts, and why?
  5. Explain best-of-N sampling. When would you use it instead of single-shot generation?
  6. What is self-consistency / majority voting? When is it useful?
  7. What is a stop sequence? Why does it matter for tool calling and structured outputs?
  8. How does repetition penalty work? What can it accidentally break?
  9. Users complain that the model is “too repetitive”—which decoding parameters do you check first?
  10. Explain min-p sampling. How is it different from top-p? Why is it becoming more popular in open-source inference?
  11. How does guided/constrained decoding (such as Outlines or LMFE) enforce a grammar or JSON schema during generation? What is the runtime overhead?

Architecture

  1. Compare encoder-only, decoder-only, encoder-decoder, and PrefixLM architectures.
  2. Why are most modern chat LLMs decoder-only?
  3. In what scenarios would you prefer an encoder-only model (BERT-style) over a decoder-only model?
  4. What is a mixture of experts (MoE)? Why does routing matter?
  5. In MoE, what is load balancing? Why can it fail?
  6. Which architecture choices tend to dominate inference memory during decoding?
  7. Summarize state space models (for example, Mamba). What bottleneck do they target?
  8. Compare Transformers, SSMs, and recurrent hybrid models (RWKV) on long-context tasks.
  9. What is a diffusion language model? When might it beat autoregressive generation?
  10. Explain the “modern decoder recipe”: RoPE + RMSNorm + SwiGLU + GQA. Why have LLaMA, Mistral, Qwen, and most open models converged on this pattern?
  11. What is “parallel attention + FFN” in PaLM? What is the tradeoff between compute savings and quality?
  12. What is multi-token prediction (MTP), as used in DeepSeek V3? How does predicting multiple future tokens improve the training signal and support speculative decoding?

Pretraining

Data Pipeline and Quality

  1. Walk through the pretraining data pipeline: data sources → filtering → deduplication → tokenization → mixing.
  2. Why is “data quality > data quantity” the most important principle in pretraining?
  3. What filters would you apply to a Common Crawl dataset before pretraining?
  4. Explain train-train deduplication and train-test contamination—why does each matter?
  5. How would you build a pipeline to detect and prevent benchmark contamination?
  6. You discover that the model is memorizing verbatim fragments of the training data—how would you diagnose and fix it?
  7. What data governance practices matter in pretraining (licenses, PII, audit trails)?
  8. How do you handle PII in training data at scale?
  9. What are common deduplication strategies (exact hash, MinHash, embedding similarity)?
  10. You need to build a classifier to filter toxic content from a pretraining corpus. What architecture would you use? How would you handle false positives that remove valuable data?
  11. What is the role of a “quality classifier” (for example, one trained on Wikipedia vs random web pages) in a pretraining data pipeline? How did Llama 3 handle this?

Data Mixing and Distribution

  1. Why does the training mixture matter? Give an example where adding more code data hurts dialogue quality.
  2. How would you design a data mixture for a general-purpose LLM?
  3. You added domain-specific data, domain performance improved, but general benchmarks fell—what happened?
  4. What is curriculum learning in pretraining?
  5. How do you estimate the marginal value of adding a new data source to the mixture? What proxy metrics can you track before running a full pretraining job?

Compute and Scaling

  1. What are the main cost drivers in pretraining (parameters, context, number of tokens, precision)?
  2. Explain the Chinchilla scaling law—what is the compute-optimal ratio between parameters and tokens?
  3. Roughly estimate: how many FLOPs does it take to train a 70B model on 2T tokens?
  4. If you double model parameters, how does FLOPs per token change approximately?
  5. Explain data parallelism, tensor parallelism, and pipeline parallelism. When do you use each?
  6. What is ZeRO / FSDP? How does it reduce memory?
  7. What is activation checkpointing? What is the tradeoff?
  8. Explain mixed-precision training (BF16/FP16). Why does BF16 dominate in LLMs?
  9. What is the roofline model? How does it help you reason about GPU utilization in inference?
  10. Estimate the minimum number of H100 GPUs and wall-clock time required to pretrain a 7B-parameter model on 1T tokens. State your assumptions.
  11. What is sequence parallelism? How does it work with tensor parallelism for long-context training?
  12. Explain the difference between Megatron-LM 3D parallelism and DeepSpeed ZeRO-3. When do you use each?

Training Recipe and Monitoring

  1. What is a typical learning-rate schedule for LLM pretraining?
  2. Loss spikes 3× in the middle of training and then slowly recovers—how would you debug it?
  3. What are the 5 most common failure modes in pretraining (instability, contamination, memorization, distribution shift, safety regression)?
  4. How do you set up a probe set to monitor quality during pretraining?
  5. Training loss is going down, but downstream evaluation scores are flat—what went wrong?
  6. When do you decide to stop pretraining? What signals do you watch?
  7. What is gradient norm monitoring? If gradient norm suddenly spikes, what training problem might it indicate? What should you do?

Evaluation and Downstream Impact

  1. What does perplexity measure? What are its limitations?
  2. Why does lower perplexity not always mean better instruction following?
  3. How do pretraining choices constrain mid-training, SFT, and inference?
  4. Your base model is weak at code—should you fix it in pretraining or later?
  5. You pretrained two models with the same architecture but different data mixtures. Model A has lower perplexity, but model B scores higher on downstream tasks. How do you explain that?

Mid-training / Continued Pretraining (CPT)

When to Use It and Why

  1. What is CPT? How is it different from pretraining and SFT?
  2. CPT vs SFT vs prompt engineering—when do you choose which?
  3. The model has high perplexity on legal documents but decent general performance—which training stage should fix that?
  4. CPT uses the same next-token objective as pretraining, but on a different distribution—what does that mean in practice?
  5. A startup wants to specialize an open-source 7B model for medical QA. They have 50,000 clinical notes. Should they do CPT, SFT, or both? Walk through the decision.

Data and Mixing

  1. What is general replay? Why is it critical for CPT?
  2. What replay ratio would you start with in CPT (for example, 80/20)? How would you tune it?
  3. What is document packing in CPT? Why does it matter for throughput?
  4. Should you use curriculum learning in CPT (more replay first, then anneal)?
  5. How does the data format differ between CPT and SFT? Why does packing without attention masking at document boundaries leak information?

The Stability Gap

  1. What is the “stability gap” in CPT? Why does it happen?
  2. General benchmarks collapse right after you start CPT—walk through your debugging steps.
  3. What role does learning rate play in CPT stability?
  4. How does regression gating work in CPT monitoring?
  5. You run 20B tokens of CPT, and MMLU drops by 5 points. After 50B tokens, it partly recovers. Explain this “U-shaped” curve, and how you would set stopping criteria.

Tokenizer Extension

  1. When should you extend the tokenizer during CPT?
  2. How do you initialize embeddings for newly added tokens?
  3. What is the “undertrained token” problem? How do you mitigate it?

Training Topology

  1. Compare the “three topologies”: packing (CPT), masking (SFT), and rollout (RL)—which tokens contribute gradients in each?
  2. How do CPT, SFT, DPO, and RL differ in data construction and where loss is applied?

CPT → RL Compatibility

  1. How does mid-training improve RL scaling (for example, OctoThinker)?
  2. Given a product requirement, how do you decide among prompting → SFT → DPO → RL → CPT → distillation?

Post-training

SFT (Supervised Fine-Tuning)

  1. What is SFT? When is it the first-choice solution?
  2. Describe the four forms of SFT data: single-turn dialogue, multi-turn dialogue, tool-use trajectories, safety demonstrations.
  3. What is the “chat template trap”? How can a single whitespace change break your model?
  4. What is completion-only loss (that is, user-token masking), and why can it improve SFT?
  5. Write the masked SFT loss formula. What do m_t = 0 and m_t = 1 mean?
  6. Your SFT model parrots the user’s prompt back verbatim—what went wrong?
  7. Too many refusal examples in SFT data cause the model to refuse everything—what is this called, and how do you fix it?
  8. How do you ensure training-inference consistency for the chat template?
  9. Your SFT model follows instructions correctly, but it is too verbose—what do you do?
  10. You did SFT on 10,000 examples, benchmark scores look good, but users say the model “sounds robotic.” What is likely wrong with your data? How do you fix it?
  11. How many SFT examples do you typically need? Discuss the “less is more” finding from the LIMA paper versus the need for diversity.

PEFT / LoRA

  1. Explain LoRA—what is low-rank factorization, and why is it cheaper than full fine-tuning?
  2. You use LoRA with rank r = 16 on a 4096×4096 weight matrix—how many trainable parameters is that compared to full fine-tuning?
  3. What is QLoRA? How does it reduce memory even further?
  4. Where should you attach LoRA adapters (attention layers vs MLP), and why?
  5. Merged adapters (single-tenant) vs online stacking (multi-tenant)—when do you use which?
  6. What is adapter multi-tenancy? How do you serve one base model + 100 LoRA adapters?
  7. A routing bug sends tenant A’s request to tenant B’s adapter—what is the impact?
  8. How do you run regression tests for each adapter in a multi-tenant system?
  9. Compare LoRA with other PEFT methods: prefix tuning, Houlsby adapters, IA3. When does LoRA win? When are alternatives better?
  10. You trained a LoRA adapter on LLaMA-3-8B. The base model ships a minor patch release—can you reuse your adapter? What are the risks?

Preference Alignment (DPO / RLHF / PPO)

  1. What is preference learning? Why is “which is better” easier to label than “write the perfect answer”?
  2. Write the KL-regularized reward maximization objective. What does β control?
  3. Explain the full RLHF pipeline: preference labeling → reward model → PPO.
  4. What is reward hacking? Give a concrete example.
  5. Why is full RLHF expensive as a system, not just as a training job?
  6. Explain DPO in one sentence—how does it avoid the full RL loop?
  7. Compare DPO and ORPO—where would you start with each?
  8. What is the main limitation of DPO/ORPO (hint: data coverage)?
  9. “More preferred” does not always mean “more correct”—why does that matter?
  10. Your DPO-tuned model has a higher preference win rate but lower factual accuracy—what happened?
  11. Explain iterative DPO / online DPO. Why is on-policy data generated during training better than fixed offline preference pairs?
  12. What is Constitutional AI (CAI)? How does it reduce dependence on human annotators?
  13. After RLHF, your model becomes extremely sycophantic (“Great question!”). What went wrong in the reward model? How would you fix it?

Tool Use and RAG Training

  1. Why is prompting alone not enough for reliable tool use in production?
  2. Describe the tool-use interaction chain: choose a tool → fill arguments → consume outputs → answer.
  3. What is the difference between underusing tools (hallucinating) and overusing tools (unnecessary calls)?
  4. What is constrained decoding for tool calls? How is schema validity different from semantic correctness?
  5. How is RAG a special case of tool use?
  6. Your agent hallucinates an answer instead of calling the retrieval tool—how do you fix it?
  7. How do you construct training data for multi-turn tool use? Discuss synthetic rollout generation, success-based filtering, and the cold-start problem.

Reasoning and Agent RL

  1. Compare outcome reward models (ORMs) and process reward models (PRMs).
  2. What are STaR/ReST-style self-training loops? Why do they help with trajectory-data scarcity?
  3. Explain GRPO—how does it avoid a learned critic and save memory?
  4. Why is GRPO important for training 70B+ reasoning models?
  5. What is reward hacking in reasoning RL? How does it differ from its manifestation in conversational alignment?
  6. Your reasoning model starts generating longer and longer outputs during RL training—what happened?
  7. Compare Dr. GRPO, GSPO, DAPO, and LUFFY—what knob does each change?
  8. Explain the “aha moment” in DeepSeek R1-Zero. What emergent behavior appeared in RL training without SFT, and why does it matter?
  9. What is the difference between chain-of-thought prompting, chain-of-thought fine-tuning, and training a reasoning model with RL? When do you use each?

Knowledge Distillation

  1. Black-box distillation vs white-box distillation—when do you use which?
  2. Write the standard distillation loss (KL divergence between teacher and student distributions).
  3. Your distilled student matches the teacher on benchmarks but fails on safety—what went wrong?
  4. When the teacher is a closed API, how do you manage teacher version drift?
  5. What is the practical sequence: black-box first, then white-box optimization?
  6. DeepSeek R1 distilled reasoning capability from R1 (671B MoE) into dense student models from 1.5B to 70B. What data did it use? What was the training recipe? How large was the student-teacher gap?

Post-training System Design

  1. Design a “medical scribe”: it should know rare drug names (CPT), but refuse to prescribe medication (alignment + evaluation).
  2. Given a fixed compute budget: DPO (training) or test-time scaling (inference)? How do you decide?
  3. Your SFT model emits incorrect tool-call formats—do you add more data, or switch to constrained decoding?
  4. Your agent fails on multi-step tool chains—what would you change in data, rewards, and search?
  5. You are the ML lead at a legal-tech startup. You have: an open-source 8B model, 100,000 contract documents, 5,000 labeled QA pairs, and a 2-person team. Design the full post-training pipeline from CPT to production.

Common Models and Benchmarks

Architecture Families

  1. Compare BERT, T5, GPT, LLaMA, Mistral, Qwen, GLM, and DeepSeek—what are the core architectural choices of each?
  2. What is the “modern decoder recipe” (RoPE + RMSNorm + SwiGLU + GQA)? Which models use it?
  3. PaLM uses parallel Transformer sublayers—what does that mean, and why?
  4. What is multi-head latent attention (MLA) in DeepSeek V2/V3?
  5. DeepSeek V3 uses multi-token prediction (MTP)—what is it, and why does it help?
  6. How did DeepSeek R1 train reasoning without SFT (R1-Zero)?
  7. What is sliding-window attention in Mistral? What does it sacrifice?
  8. Mixtral is an MoE: 8 experts, top-2 routing, 47B total params / ~13B active—why is it efficient?
  9. Trace the evolution: GPT-2 → GPT-3 → GPT-3.5 → GPT-4. What changed at each step (scale, RLHF, multimodality, MoE)?
  10. Compare LLaMA 1 vs LLaMA 2 vs LLaMA 3. What were the key training-recipe changes in each generation (not just scale)?
  11. How do Gemma/Gemini handle long context (up to 1M tokens)? How is that different from RoPE extrapolation methods?

Evaluation and Benchmarks

  1. List a minimal benchmark suite for a general-purpose LLM (knowledge, math, code, dialogue, safety).
  2. What is pass@k in code benchmarks? How is it different from pass@1?
  3. MMLU vs MMLU-Pro—what is the difference?
  4. What is Chatbot Arena (LMSYS)? Why are Elo-based rankings useful?
  5. Your model scores 90% on MMLU but performs poorly on real customer queries—where is the gap?
  6. What are common evaluation traps (prompt formatting, temperature, contamination, scoring method)?
  7. LLM-as-a-judge: what biases exist, and how do you mitigate them?
  8. When should you use human preference tests instead of automatic benchmarks?
  9. How do you evaluate long-context models (Needle in a Haystack, LongBench)?
  10. What is TruthfulQA? What failure mode does it target?
  11. How do you evaluate multimodal models (MMMU, MathVista)?
  12. How do you detect benchmark contamination (data leakage) in a trained model?
  13. You need to evaluate model safety: which benchmarks and red-teaming methods do you use?
  14. What is the difference between “static” benchmarks (MMLU, HumanEval) and “dynamic” benchmarks (Chatbot Arena, LiveCodeBench)? Why are dynamic benchmarks becoming more important?
  15. Design an internal evaluation suite for a customer-facing chatbot. Which dimensions do you test (accuracy, safety, latency, cost, style)? How do you weight them?

Inference and Compression

Inference Physics

  1. Explain the two phases of LLM inference: prefill vs decode. Why do they have different bottlenecks?
  2. Prefill is compute-bound; decode is memory-bandwidth-bound—explain why.
  3. What is arithmetic intensity? How does it determine whether a workload is compute-bound or memory-bound?
  4. What is TTFT? What dominates it? What is TPOT/ITL? What dominates it?
  5. TTFT is too high—walk through your debugging steps.
  6. TPOT got worse after a configuration change—what do you check?
  7. An H100 has ~3.35 TB/s memory bandwidth and ~990 TFLOPS BF16. A BF16 70B model is ~140GB. At batch size = 1, what is the theoretical maximum tokens/sec during decoding? Where is the bottleneck?

KV Cache

  1. Estimate KV cache size: given layers = 80, KV heads = 8, head dim = 128, FP16—how many bytes per token?
  2. At 8K context, how much KV memory does each sequence require under that calculation?
  3. How does GQA reduce the KV cache compared with MHA? Give the scaling factor.
  4. What is PagedAttention? How is it analogous to virtual memory?
  5. Why does naive contiguous KV allocation waste memory?
  6. What happens when you quantize the KV cache (FP16 → INT8)? Which tasks degrade first?
  7. Your model runs out of memory on long prompts—walk through the mitigation steps.
  8. Compare KV cache compression techniques: quantization, eviction (H2O, StreamingLLM-style sink tokens), sliding windows, cross-layer sharing. Which works best at extreme context lengths?

System Optimization

  1. What is continuous batching / in-flight batching? How is it different from static batching?
  2. Why is admission control important for continuous batching?
  3. What is chunked prefill? How does it solve the convoy effect?
  4. What is prefix caching (prompt caching)? When does it help most?
  5. Explain speculative decoding: draft model → target verification. What is the win condition?
  6. What is guided/constrained decoding? Why is it important for JSON / tool outputs?
  7. Compare vLLM, TGI, TensorRT-LLM, and SGLang—what are the key architectural differences? When should you choose each?

Kernel Optimization

  1. How does FlashAttention reduce HBM traffic?
  2. What is kernel fusion? Why does it help in both prefill and decode?
  3. FlashAttention vs PagedAttention—they solve different problems. Explain.
  4. What is FlashDecoding? How does it parallelize the decode phase over the KV sequence dimension?

Serving Architecture

  1. Why split prefill and decode into different clusters (P/D disaggregation)?
  2. What is multi-LoRA serving (“Bento” mode)?
  3. In a multi-LoRA system, how do you batch requests by adapter_id?
  4. What is “tenant leakage” in multi-adapter serving? How do you prevent it?
  5. How would you design autoscaling for an LLM inference service? Which metrics should trigger scale-out and scale-in (queue depth, GPU utilization, p99 latency)?

Compression

  1. Compare weight-only quantization, activation quantization, and KV quantization—when do you use each?
  2. What is the “outlier problem” in LLM quantization? Why does naive quantization fail?
  3. What are AWQ and SmoothQuant? What does each solve?
  4. Compare unstructured pruning, structured pruning (N:M sparsity), and architectural sparsity (MoE).
  5. What is knowledge distillation for inference? Compare response distillation and logit distillation.
  6. Why can pruning introduce nonlinear “quality cliffs”?
  7. Compare GPTQ, AWQ, and GGML/GGUF quantization. Which is post-training quantization? Which is calibration-based? When do you use each?
  8. What is quantization-aware training (QAT)? How is it different from post-training quantization (PTQ)? When is the extra cost worth it?

Test-time Scaling

  1. What is test-time scaling? How can it improve reliability without retraining?
  2. Compare best-of-N, self-consistency voting, critique-revise loops, and tree search (MCTS).
  3. When should you spend compute on test-time scaling vs retraining?
  4. Explain how process reward models (PRMs) support tree search at inference time. What is the tradeoff between breadth (more candidates) and depth (more reasoning steps)?

Metrics and Evaluation

  1. What are the four core inference metrics (TTFT, TPOT, throughput, $/million tokens)?
  2. How do you set quality regression gates after quantization?
  3. You quantized the model, and Needle-in-a-Haystack accuracy dropped—why?
  4. Users report “unstable latency”—sometimes fast, sometimes 5× slower on similar prompts. Diagnose batching, caching, and queue-depth issues.

System Design Exercises

  1. Why does increasing batch size improve throughput but hurt single-request latency?
  2. Design an inference system for a latency-sensitive chat product.
  3. Design an inference system for batch document processing (throughput > latency).
  4. Your service handles tens of thousands of requests per second, each with long RAG context—what architecture would you propose?
  5. You need to serve a 70B model but only have one A100 80GB. Walk through the options (quantization, offloading, a smaller model, distillation).
  6. Compare cloud inference (OpenAI API, Bedrock, Vertex AI) vs self-hosted (vLLM on GPU, Ollama on Mac)—what factors determine the right choice?

Applications / Agents

Agent Fundamentals

  1. What is an “agentic application”? How is it different from a single LLM call?
  2. Workflow vs agent—when should you use which? Give a decision rule.
  3. Describe the agent control loop: perceive → think → act → check → update state → repeat.
  4. What are common agent products (enterprise search, support triage, deep research, coding agents, ops automation)?
  5. When should you not use an agent?
  6. What is the “bitter lesson” of agents—why can simple designs plus strong models beat complex frameworks?

Architectural Components

  1. What is “policy” in an agent system (model + routing + decoding settings)?
  2. What are output contracts? Why are they critical for production agents?
  3. Explain the validate → retry → repair prompting pattern.
  4. Agent state (structured facts) vs context (the token payload of each step)—what should each contain?
  5. When should you use an explicit planner vs implicit planning in the policy prompt?
  6. What are common orchestration patterns: prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer?
  7. Explain the ReAct pattern (reasoning + acting). How is it different from pure chain-of-thought? What failure modes does it have?

Tool Interface Design

  1. What makes a good tool contract (schema, validation, timeouts, retries, idempotency, observability)?
  2. What is MCP (Model Context Protocol)? How does it relate to tool services?
  3. What is idempotency for write tools? Why is it critical?
  4. How do you risk-tier tools (read_low → write_medium → write_high)?
  5. Your agent issued duplicate refunds because a tool was not idempotent—what is the fix?
  6. How do you enforce tool permissions outside the model (allowlists, authorization)?
  7. How do you handle tool errors gracefully? Should error messages be returned to the model? What are the risks?

RAG Pipeline

  1. Walk through the RAG pipeline: ingest → chunking → embeddings → index → retrieval → reranking → generation.
  2. Why does chunk size matter? What happens if chunks are too small or too large?
  3. Compare dense retrieval (embeddings), sparse retrieval (BM25), and hybrid retrieval—when do you use which?
  4. What is cross-encoder reranking? Why is it better than dual-encoder similarity but more expensive?
  5. How do you evaluate retrieval quality (recall@k, MRR, nDCG)?
  6. Your RAG system retrieved the right documents, but the model ignored them—what went wrong?
  7. How do you defend against prompt injection through retrieved documents?
  8. What are query rewriting and multi-hop retrieval? When do you need them?
  9. How do you handle stale indexes in a production RAG system?
  10. Compare vector databases (Pinecone, Weaviate, Qdrant, pgvector)—what are the tradeoffs?
  11. You built a RAG system over a 10-million-document legal corpus. Retrieval latency is acceptable, but answer quality is poor. Diagnose the system: is the problem chunking, the embedding model, retrieval, reranking, or generation?
  12. What is agentic RAG? How is it different from single-shot retrieve-then-generate (for example, iterative retrieval, query decomposition, self-reflection on retrieval quality)?
  13. How do you handle tables, images, and structured data in a RAG pipeline?

Guardrails and Safety

  1. What layers of guardrails should a production agent have (relevance, safety, PII, content moderation, rules, tool safety)?
  2. When should an agent escalate to a human?
  3. How do you handle prompt-injection attacks in agent systems?
  4. Your agent approved an unauthorized action—which layers failed?
  5. What is the difference between a jailbreak (bypassing safety) and prompt injection (hijacking intent)? How do you defend against each?
  6. How do you build red-team evaluations for agent systems? Which attack categories do you cover (direct injection, indirect injection, social engineering, tool abuse)?

Memory and Context Engineering

  1. What is context engineering—and how is it different from prompt engineering?
  2. How do you manage context growth in tool-calling loops that run for 25+ turns?
  3. What is context rot—and why does retrieval precision degrade in very long contexts?
  4. Explain progressive disclosure in agents (metadata → full instructions → deep resources).
  5. What are the context layers: pinned context, working set, retrieved context, summaries?
  6. When should you use a sub-agent architecture to manage context?
  7. How do you implement “memory” for a conversational agent across multiple sessions? Compare explicit memory stores vs retrieval over past conversation logs.

Evaluation and Observability

  1. How do you evaluate an agent as a system (task success, tool accuracy, cost, safety)?
  2. What should an agent observability system log?
  3. Build a debugging funnel for agent failures: tools → retrieval → context → generation → guardrails.
  4. How do you build a replayable evaluation framework with a golden task set?
  5. What does one agent task cost? How do you estimate and control it (token budgets, max iterations, cost caps)?

Skills and Process

  1. What is an Agent Skill? How is it different from tools and prompts?
  2. Explain the progressive-disclosure levels of skills (metadata → full SKILL.md → linked resources).
  3. How do skills and MCP work together in an agent system?

Agent System Design Exercises

  1. Design a “news research agent”: search, verify, and summarize news on a topic, with citations.
  2. Design a support-triage agent with routing, tool use, and human handoff.
  3. Design a coding/debugging agent that loops over code search and test-based verification.
  4. How do you prevent “search → fetch everything” tool abuse in an agent?
  5. Your agent takes 30 seconds per query—where do you look first?
  6. Design an “AI data analyst” agent: it accepts natural-language questions, writes SQL, executes it, interprets results, and visualizes them. What tools does it need? What guardrails?
  7. You are building a multi-agent system in which a “planner” agent delegates to an “executor” agent. If the executor fails halfway through the task, how do you handle failure recovery?

Taxonomy / Cross-Cutting Concepts

Key Distinctions

  1. Capability vs behavior—where does each come from across the LLM lifecycle?
  2. Latency vs throughput—when do you optimize which?
  3. Compute-bound vs memory-bound—how do you tell which regime you are in?
  4. Dense models vs MoE—what are the pros and cons of each?
  5. Online RL (PPO) vs offline preference learning (DPO)—when do you choose which?

Common Misconceptions

  1. “Tokenization is just preprocessing”—why is that wrong?
  2. “Lower perplexity = a better model”—when does that fail?
  3. “The KV cache is just an O(n²)-attention issue”—what else is more important?
  4. “Byte-level BPE = character-level tokenization”—what is the difference?
  5. “Packing more data = a better model”—when can packing hurt instead?
  6. “RAG solves hallucinations”—why is that only partly true?
  7. “A larger context window = better performance”—when can longer context actually hurt?
  8. “Quantized models are always worse”—when can a quantized model beat a larger full-precision model?

End-to-End Design

  1. Trace a single user query through the full stack: tokenization → embeddings → transformations → decoding → serving → return.
  2. How do training choices (architecture, tokenizer, context length) constrain inference?
  3. You are building an LLM-powered product from scratch for a vertical domain—walk through the full stack of technical decisions.
  4. Compare these engineering tradeoffs: latency vs quality, compute vs data, dense vs MoE, tool use vs model reasoning.

Miscellaneous

  1. How does tokenizer choice in pretraining affect KV cache size at inference? Trace the connection: vocabulary size → embedding dimension → KV heads → per-token memory.
  2. You trained a model at 4K context, but RAG at inference needs 128K. Trace the full solution path: positional-encoding extension, KV cache management, chunked prefill, and retrieval-pipeline design.
  3. Compare the cost of improving coding ability via: (a) adding code data in pretraining, (b) code-focused CPT, (c) code SFT, (d) code RL with execution feedback. When is each appropriate?
  4. The model emits incorrect tool-call JSON. Diagnose whether the problem is SFT data quality, tokenizer issues, decoding strategy, or the tool schema definition. How do you tell?
  5. Your production LLM system costs $0.50 per query. The business requires $0.05. Walk through every optimization lever: model size, quantization, batching, caching, distillation, reducing output tokens, switching to MoE.
  6. You are evaluating three competing LLMs for deployment. Model A has the highest maximum quality. Model B is middling but fast at inference. Model C is the smallest and cheapest. How would you build an evaluation for an enterprise customer-support setting, and what would you recommend?

References

  1. Attention Is All You Need (arXiv:1706.03762). The core Transformer architecture, the attention formula, and the multi-head mechanism.
  2. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv:1810.04805). Bidirectional pretraining and the MLM objective.
  3. Language Models are Unsupervised Multitask Learners (OpenAI, 2019). GPT-2 and the scaling of autoregressive language models.
  4. Language Models are Few-Shot Learners (arXiv:2005.14165). GPT-3, few-shot learning, and in-context learning.
  5. Neural Machine Translation of Rare Words with Subword Units (arXiv:1508.07909). BPE subword tokenization and the OOV tradeoff.
  6. Efficient Estimation of Word Representations in Vector Space (arXiv:1301.3781). word2vec, a foundational method for word embeddings.
  7. GloVe: Global Vectors for Word Representation (EMNLP 2014). Word embeddings from global co-occurrence statistics.
  8. Scaling Laws for Neural Language Models (arXiv:2001.08361). Scaling relations among parameters, data, and compute.
  9. Training Compute-Optimal Large Language Models (arXiv:2203.15556). The Chinchilla scaling law and the compute-optimal ratio of parameters to tokens.
  10. The Pile: An 800GB Dataset of Diverse Text for Language Modeling (arXiv:2101.00027). Building a large-scale pretraining dataset.
  11. Deduplicating Training Data Makes Language Models Better (arXiv:2107.06499). Why training-data deduplication matters, and how to do it.
  12. LLaMA: Open and Efficient Foundation Language Models (arXiv:2302.13971). The training recipe for an efficient open base model.
  13. Llama 2: Open Foundation and Fine-Tuned Chat Models (arXiv:2307.09288). The full pipeline across pretraining, SFT, and RLHF.
  14. Training Language Models to Follow Instructions with Human Feedback (arXiv:2203.02155). InstructGPT and the full RLHF pipeline.
  15. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv:2305.18290). DPO, preference alignment without an RL loop.
  16. LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685). Low-rank fine-tuning for parameter-efficient adaptation.
  17. QLoRA: Efficient Finetuning of Quantized Language Models (arXiv:2305.14314). LoRA fine-tuning on a quantized base model.
  18. LIMA: Less Is More for Alignment (arXiv:2305.11206). The effectiveness of a small amount of high-quality SFT data.
  19. Constitutional AI: Harmlessness from AI Feedback (arXiv:2212.08073). An alignment method that replaces part of human labeling with AI feedback.
  20. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (arXiv:2402.03300). GRPO, critic-free reasoning RL training.
  21. STaR: Bootstrapping Reasoning With Reasoning (arXiv:2203.14465). A self-training loop for reasoning.
  22. RoFormer: Enhanced Transformer with Rotary Position Embedding (arXiv:2104.09864). The definition and intuition of RoPE.
  23. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (arXiv:2108.12409). The motivation behind ALiBi and length extrapolation.
  24. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (arXiv:2305.13245). GQA as a quality-efficiency tradeoff.
  25. Mamba: Linear-Time Sequence Modeling with Selective State Spaces (arXiv:2312.00752). State-space models for linear-complexity sequence modeling.
  26. Mixtral of Experts (arXiv:2401.04088). A sparse MoE architecture with 8 experts and top-2 routing.
  27. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (arXiv:2405.04434). MLA and efficient MoE.
  28. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv:2501.12948). Training reasoning via pure RL and distillation.
  29. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (arXiv:2205.14135). IO-aware tiled attention computation.
  30. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (arXiv:2307.08691). Improved parallelism and work partitioning.
  31. Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv:2309.06180). vLLM and virtual-memory-like KV cache management.
  32. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration (arXiv:2306.00978). Activation-aware weight quantization.
  33. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (arXiv:2211.10438). A quantization method that smooths activation outliers.
  34. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv:2210.17323). Calibration-based post-training quantization.
  35. Fast Inference from Transformers via Speculative Decoding (arXiv:2211.17192). Speculative decoding, using a draft model to accelerate generation.
  36. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401). The original definition and motivation of RAG.
  37. ReAct: Synergizing Reasoning and Acting in Language Models (arXiv:2210.03629). An agent pattern that alternates reasoning and acting.
  38. Toolformer: Language Models Can Teach Themselves to Use Tools (arXiv:2302.04761). Models teaching themselves to call tools.
  39. Building Effective Agents (Anthropic, 2024). Design principles for production agents.
  40. LLM Powered Autonomous Agents (Lilian Weng, 2023). A survey of agent systems.
  41. MLStack.cafe — LLM Interview Questions (59 questions). Transformer architecture, attention, transfer learning, alignment.
  42. MLStack.cafe — ChatGPT Interview Questions (42 questions). RLHF, tokenization, context handling, evaluation.
  43. MLStack.cafe — NLP Interview Questions (38 questions). Positional encoding, encoder-decoder models, CNN vs Transformer.
  44. Awesome Generative AI Guide — 60 Common Generative AI Interview Questions. Generative models, LLMs, embeddings, multimodality, training, and evaluation.
  45. Hugging Face NLP Course. Chapters 1–4 fundamentals.
  46. Stanford CS324 — Large Language Models. Training, evaluation, social impact.
  47. Full Stack Deep Learning — LLM Bootcamp 2023. End-to-end LLM application development.
  48. Chip Huyen — Designing ML Systems (O’Reilly, 2022). Training, serving, evaluation, data distribution.
  49. Jay Alammar — The Illustrated Transformer. A visual guide to Transformer architecture.
  50. Sebastian Raschka — Build a Large Language Model (From Scratch) (2024). Architecture, pretraining, SFT, alignment.
  51. UC Berkeley CS294/194-196 — Large Language Model Agents. Agent systems, tool use, planning.