Layer Normalization & Residuals

Layer Normalization & Residual Connections

Residual Connections (Skip Connections)

output = x + SubLayer(x)

Benefits:

Layer Normalization

Normalize across features (not batch): LayerNorm(x) = γ × (x - μ) / σ + β

Pre-LN vs Post-LN:

Transformer Block

Pre-LN Transformer Block: 1. x = x + Attention(LayerNorm(x)) 2. x = x + FFN(LayerNorm(x))

Feed-Forward Network

FFN(x) = GELU(x × W₁ + b₁) × W₂ + b₂