Attention & transformer math

What the problems look like

A few statements from this skill. Answers, hints and worked solutions live in the practice session.

1. Warm-up

   In one head of self-attention, a query $\mathbf{q} = (2, 1, 3)$ attends to a key $\mathbf{k} = (4, 0, 1)$. Compute the unnormalized attention score $\mathbf{q}\cdot\mathbf{k}$.

2. Warm-up

   In one head of self-attention, a query $\mathbf{q} = (4, 2, 2)$ attends to a key $\mathbf{k} = (2, 3, 4)$. Compute the unnormalized attention score $\mathbf{q}\cdot\mathbf{k}$.

3. Warm-up

   In one head of self-attention, a query $\mathbf{q} = (4, 1, 2)$ attends to a key $\mathbf{k} = (2, 3, 1)$. Compute the unnormalized attention score $\mathbf{q}\cdot\mathbf{k}$.

Problems shown under their original terms: Generated (template, solver-computed).