Cross-attention — decoder Q, encoder K/V

Background

Cross-attention builds the query from the decoder and the keys/values from the encoder, then runs the usual scaled dot-product attention:

$$Q = H^{\text{dec}}W^Q,\;\; K = H^{\text{enc}}W^K,\;\; V = H^{\text{enc}}W^V, \qquad \text{out} = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

The softmax is over the source axis, so each target position pulls a blend of encoder values.

Problem statement

Implement cross_attention(H_dec, H_enc, Wq, Wk, Wv) returning the attention output.

Input

• H_dec — (target_len, d_model): decoder states (queries' source).
• H_enc — (source_len, d_model): encoder outputs (keys'/values' source).
• Wq, Wk — (d_model, d_k); Wv — (d_model, d_v).

Output

Returns an np.ndarray of shape (target_len, d_v).

Examples

Example 1 — uniform attention averages encoder values

Input:  H_dec = [[1, 0]], H_enc = [[1, 0], [0, 1]],
        Wq = Wk = [[0, 0], [0, 0]],   Wv = [[1, 0], [0, 1]]
Output: [[0.5, 0.5]]

Explanation: zero Q/K → equal scores over the 2 source positions → output is the mean of the encoder values $([1,0]+[0,1])/2 = [0.5, 0.5]$.

Constraints

• Q from H_dec; K, V from H_enc.
• scores = Q @ K.T / sqrt(d_k); softmax over the source axis (axis=1); multiply by V.
• Return shape (target_len, d_v).

Notes

• This is the only place a translation decoder reads the source — each generated token is grounded in a soft alignment over the input sentence.