Transformers

81 problems

• Add & Norm (post-norm residual)Medium
• Fuse token + positional embeddingsEasy
• Attention step 3 — weighted sum of valuesEasy
• Teacher forcing — next-token targetsMedium
• Concatenate the headsEasy
• Cross-attention — decoder Q, encoder K/VMedium
• Cross-attention score shape (target × source)Easy
• One decoder layer (post-norm, 3 sublayers)Medium
• How many sublayers per layer?Easy
• Head dimension d_k = d_model / hEasy
• The input layer — token lookup + positionalMedium
• Per-layer complexity — attention vs recurrenceMedium
• Self-attention is O(n²)Easy
• Map a model to its transformer familyEasy
• Why total compute is invariant to hMedium
• Does this attention need a causal mask?Easy
• Token-to-token path lengthEasy
• Why position matters — permutation-invariant poolingEasy
• The position-wise feed-forward networkEasy
• Pretraining objective per familyEasy
• Project X into Q, K, VEasy
• Attention step 2 — scale by √dₖ, then softmaxMedium
• Attention step 1 — the scores QKᵀEasy
• Full self-attention from XMedium
• Self or cross attention?Easy
• Sequential steps — serial vs parallelEasy
• Which transformer family?Easy
• Token embedding lookupEasy
• Sinusoidal positional encodingEasy
• Scaled dot-product self-attentionMedium
• Causal mask: build + applyEasy
• Multi-head split + combineMedium
• Multi-Head Attention (full layer)Medium
• LayerNorm forwardEasy
• Transformer block forward (pre-LN, residual)Hard
• Cross-Attention (Encoder-Decoder)Medium
• Dynamic-Tanh (DyT)Medium
• Efficient sparse window attentionMedium
• FlashAttention tiled forwardHard
• GELU forward (tanh approximation)Easy
• Attention block param budget = 4·C²Easy
• √dₖ and softmax saturation (row entropy)Medium
• The autoregressive generation loopMedium
• Parameter breakdown of a GPT-2 blockMedium
• The block_size hard capEasy
• GPTConfig.head_size with the divisibility checkEasy
• Slide the context windowEasy
• Fixed vs growing memory (the RNN bottleneck)Easy
• How big is GPT-2's embedding table?Medium
• Embeddings.forward — token + position (batched)Medium
• Split the fused QKV projectionEasy
• Gradient paths through L residual blocksEasy
• Total LayerNorm parameters in GPT-2Easy
• LayerNorm statistics — per token, over featuresEasy
• Slice the mask buffer to [:T, :T]Easy
• The 4× MLP hidden dimensionEasy
• How big is the nucleus?Easy
• One-hot tokens are all equidistantEasy
• Why one-hot inputs blow up memoryEasy
• Learned positional lookup P[:T]Easy
• Pre-LN vs post-LN blockMedium
• Why mask before softmax, not afterMedium
• Where loss starts: log(V)Easy
• Depth-aware init for residual projectionsEasy
• The residual Jacobian I + ∂f/∂xEasy
• The residual stream grows without LayerNormMedium
• Attention vs MLP — who does whatEasy
• Weight tying — the LM head reuses EMedium
• Assemble the full GPT-2 parameter countMedium
• How many positions can each query see?Easy
• Grouped-query attentionMedium
• KV cache for autoregressive inferenceMedium
• KV cache compression (MLA)Hard
• Noisy top-k gatingMedium
• RMSNormEasy
• Rotary Position Embedding (RoPE)Hard
• Scaled dot-product attentionMedium
• Sequence padding & maskingEasy
• Sparse mixture-of-experts layerHard
• SwiGLU activationMedium
• ViT Patch EmbeddingMedium