LLMs

98 problems

• Build the chat messages listEasy
• Format retrieved docs as contextEasy
• One model, three roles — route by promptEasy
• MockLLM keyword routingMedium
• MockSearchEngine keyword retrievalEasy
• The RAG run: search → ground → answerMedium
• The ReAct loop with a max_steps guardMedium
• Validate a chat messageEasy
• Teacher forcing — next-token targetsMedium
• Beam search decodingMedium
• Best-of-N SelectionEasy
• Binary Quantization + Hamming RetrievalMedium
• BM25 ScoringMedium
• Token embedding lookupEasy
• Sinusoidal positional encodingEasy
• Scaled dot-product self-attentionMedium
• Causal mask: build + applyEasy
• Multi-head split + combineMedium
• Multi-Head Attention (full layer)Medium
• LayerNorm forwardEasy
• Transformer block forward (pre-LN, residual)Hard
• Citation Attribution AssemblyMedium
• Constrained Decoding Logit MaskMedium
• Continuous Batching SchedulerHard
• Cross-Encoder Rerank Top-kEasy
• Scaling laws — estimate the power-law exponentMedium
• DPO lossMedium
• Efficient sparse window attentionMedium
• Faithfulness / Groundedness ScoreMedium
• Generalised Advantage Estimation (GAE)Medium
• Attention block param budget = 4·C²Easy
• √dₖ and softmax saturation (row entropy)Medium
• The autoregressive generation loopMedium
• Parameter breakdown of a GPT-2 blockMedium
• The block_size hard capEasy
• GPTConfig.head_size with the divisibility checkEasy
• Slide the context windowEasy
• Fixed vs growing memory (the RNN bottleneck)Easy
• How big is GPT-2's embedding table?Medium
• Embeddings.forward — token + position (batched)Medium
• Split the fused QKV projectionEasy
• Gradient paths through L residual blocksEasy
• Total LayerNorm parameters in GPT-2Easy
• LayerNorm statistics — per token, over featuresEasy
• Slice the mask buffer to [:T, :T]Easy
• The 4× MLP hidden dimensionEasy
• How big is the nucleus?Easy
• One-hot tokens are all equidistantEasy
• Why one-hot inputs blow up memoryEasy
• Learned positional lookup P[:T]Easy
• Pre-LN vs post-LN blockMedium
• Why mask before softmax, not afterMedium
• Where loss starts: log(V)Easy
• Depth-aware init for residual projectionsEasy
• The residual Jacobian I + ∂f/∂xEasy
• The residual stream grows without LayerNormMedium
• Attention vs MLP — who does whatEasy
• Weight tying — the LM head reuses EMedium
• Assemble the full GPT-2 parameter countMedium
• How many positions can each query see?Easy
• Greedy decoding loopEasy
• GRPO objectiveHard
• Post-Training int8 QuantizationMedium
• IPO / SimPO LossMedium
• KTO (Kahneman-Tversky) LossHard
• KV cache for autoregressive inferenceMedium
• KV cache compression (MLA)Hard
• KV-Cache Eviction (Sliding Window)Medium
• LoRA forward passEasy
• Maximal Marginal Relevance (MMR)Medium
• Multi-Query Fan-Out + DedupEasy
• Noisy top-k gatingMedium
• ORPO Loss (Odds-Ratio Preference Optimisation)Medium
• Paged-Attention KV Block AllocationMedium
• Parent-Document RetrievalMedium
• pass@k Unbiased EstimatorMedium
• Perplexity from log-probsEasy
• PPO clipped objectiveMedium
• Prefix-Cache Hit AccountingEasy
• Product Quantization (PQ) Encode + ADCHard
• Recall@k (Retrieval)Easy
• Reciprocal Rank Fusion (RRF)Medium
• Recursive Chunking with OverlapMedium
• Refusal / Confidence GateEasy
• Repetition penalty (HuggingFace-style)Medium
• Reward-Model Margin LossMedium
• RMSNormEasy
• Self-Consistency Majority VoteEasy
• Sparse mixture-of-experts layerHard
• Speculative decoding verificationMedium
• Stop-Sequence Detection (Streaming)Easy
• SwiGLU activationMedium
• Temperature scalingEasy
• Token-Budget Context ManagerMedium
• Top-k samplingEasy
• Top-p (nucleus) samplingMedium
• Unigram probability from a corpusEasy
• Viterbi algorithmMedium