Gradient paths through L residual blocks

Background

Expanding the stacked residual Jacobian ∏_{ℓ=1}^{L} (I + ∂f_ℓ) gives a sum over paths: at each block the gradient either goes through the sublayer or skips it. With L blocks that's:

$$2^L \text{ paths}$$

The single "skip every block" path is the pure identity I — the highway that never vanishes.

Problem statement

Implement gradient_path_count(L) returning the number of gradient paths through L residual blocks.

Input

• L — int, number of residual blocks (L ≥ 0).

Output

Returns an int: 2**L.

Examples

Example 1

Input:  L = 1
Output: 2

(Through the sublayer, or skip it.)

Example 2 — GPT-2 small depth

Input:  L = 12
Output: 4096

Constraints

• Return 2**L.
• L = 0 → 1 (just the identity path).

Notes

• The short paths (few layers traversed) dominate early training and all carry healthy gradient — which is why deep residual nets train from step one.