Activation Functions

31 problems

• Attention step 2 — scale by √dₖ, then softmaxMedium
• ReLU forward + backwardEasy
• Sigmoid forward + backwardEasy
• The classifier head — FC + softmaxMedium
• Feature map — bias and ReLUEasy
• Binary output head — probability and decisionEasy
• Dense layer — sigmoid forward passEasy
• Why nonlinearity — collapsing two linear layersMedium
• Neuron — one unit, three activationsEasy
• Output layer — predict the classEasy
• ReLU — find the dead neuronsEasy
• Dense layer with ReLU (the production hidden default)Easy
• Sigmoid derivative from the activationEasy
• Softmax with temperatureEasy
• Vanishing gradients through sigmoid layersEasy
• Dynamic-Tanh (DyT)Medium
• GELU backward (tanh approximation)Medium
• GELU forward (tanh approximation)Easy
• √dₖ and softmax saturation (row entropy)Medium
• Why mask before softmax, not afterMedium
• He weight initializationEasy
• Leaky ReLU activationEasy
• Log-softmaxEasy
• Logistic regression — probability from a scoreEasy
• Logistic regression — the sigmoid reflection identityEasy
• Noisy top-k gatingMedium
• SELU activationEasy
• Softmax backwardMedium
• Softmax (multinomial) regressionMedium
• SwiGLU activationMedium
• Swish / SiLU activationEasy