Formula & theorem reference

Take a sequence w_1, \ldots, w_n and a language model's conditional probabilities P(w_i \mid w_{<i}). Perplexity is then \operatorname{PPL} = \exp\left(-\frac{1}{n}\sum_{i=1}^n \log P(w_i \mid w_{<i})\right). This is the exponentiated average negative log-probability the model assigns to each token. It is an exponentiated average of per-token surprise. A perplexity of k means something specific. On average, the model was as uncertain at each step as if it had been choosing among k options at random. Lower perplexity is better. It means the model assigned higher probability to what actually came next.

Take query and key vectors q, k \in \mathbb{R}^{d_k} with independent, zero-mean, unit-variance entries. Their dot product q \cdot k then has variance d_k. Dividing the score by \sqrt{d_k}, \operatorname{score}(q, k) = \frac{q \cdot k}{\sqrt{d_k}}, restores unit variance regardless of d_k. This keeps the softmax over attention scores away from the saturated, near-zero-gradient region that a growing d_k would otherwise push it into.

For vectors u, v \in \mathbb{R}^n, cosine similarity is \operatorname{cossim}(u, v) = \frac{u \cdot v}{\lVert u \rVert \lVert v \rVert}, This is the cosine of the angle between u and v. It equals 1 for the same direction, 0 for perpendicular vectors, and -1 for opposite directions. Both norms in the denominator cancel out vector length. So this measures direction, not size. For a text embedding, direction matches meaning. Size often just reflects passage length instead.

For logits z_1, \ldots, z_K and temperature T > 0, \hat{y}_k = \operatorname{softmax}(z/T)_k = \frac{e^{z_k/T}}{\sum_{j=1}^K e^{z_j/T}}. Divide every logit by T before the usual softmax. This rescales how sharply the distribution favors the largest logit. As T \to 0, the highest-logit token's probability approaches 1. Sampling then converges to greedy decoding: always picking the top choice. As T \to \infty, the distribution flattens toward uniform over all K options.

Take a frozen weight matrix of shape d_{\text{in}} \times d_{\text{out}} and a LoRA rank r. Full fine-tuning updates d_{\text{in}} \cdot d_{\text{out}} parameters. LoRA instead trains two low-rank matrices: A (shape d_{\text{in}} \times r) and B (shape r \times d_{\text{out}}). Their product approximates the update. This trains only r \cdot (d_{\text{in}} + d_{\text{out}}) parameters. The ratio \frac{r(d_{\text{in}} + d_{\text{out}})}{d_{\text{in}} d_{\text{out}}} shrinks as the matrix grows, and as r shrinks. For a square matrix, where d_{\text{in}} = d_{\text{out}} = d, this simplifies to 2r/d.