Token-to-token path length

Background

A key reason attention beat recurrence is path length — how many hops information takes to travel from token $i$ to token $j$:

• RNN/LSTM: the signal flows along the chain of hidden states, so the distance is $|i - j|$.
• Attention: any token can read any other in a single layer, so the distance is $1$.

Long paths are exactly what make long-range dependencies hard for RNNs.

Problem statement

Implement path_length(i, j, arch) returning the information path length between positions i and j.

Input

• i, j — int token positions.
• arch — "rnn" or "attention".

Output

Returns an int: abs(i - j) for "rnn", 1 for "attention".

Examples

Example 1 — far apart in an RNN

Input:  i = 0, j = 5, arch = "rnn"
Output: 5

Example 2 — direct in attention

Input:  i = 0, j = 5, arch = "attention"
Output: 1

Constraints

• "rnn" → abs(i - j); "attention" → 1.
• Return a plain int.

Notes

• Shorter paths mean shorter gradient routes too — one reason attention trains more stably over long contexts.