Glossary

A notebook line that downloads and installs one Python package; the leading ! tells Colab to run it as a command to the computer, not as Python code.

A nonlinear function applied to a neuron's weighted sum, letting a network represent more than a straight line.

An optimizer that combines momentum with a per-parameter step size, rescaled by each parameter's own recent history of how large its gradient has been.

A predefined reference box, at a fixed size and position, that a detector adjusts to match a real object nearby.

A network's full shape, fixed before training starts -- how many layers it has (depth) and how many neurons sit in each one (width).

A number indicating how strongly one position attends to another, produced by an attention mechanism's scoring step.

The area under an ROC curve -- the chance a random actual positive scores higher than a random actual negative.

A network trained to reconstruct its own input, forcing it to learn a compressed representation in between.

A framework feature that computes exact gradients automatically, without a programmer deriving them by hand.

An algorithm that computes every parameter's gradient by applying the chain rule backward through a computational graph.

Backpropagation applied to a recurrent network unrolled across every time step of a sequence.

The second sweep through a computational graph, running from the loss back to every leaf, that computes each node's gradient.

A group of training examples averaged together before one parameter update -- the whole dataset (a full batch), a small random subset (a minibatch), or a single example (the stochastic case).

The leading axis of a stacked matrix counting how many independent examples ride together -- a bookkeeping choice, not a mathematical property of the data.

A technique that normalizes a layer's inputs using statistics computed across the current minibatch.

A search strategy that keeps several of the most likely partial sequences at each generation step, instead of just one.

The constant added at the end of a weighted sum; it is the output when every feature is zero, and it shifts every output by the same amount.

More capacity lowers a model's systematic error (bias) but raises how much its fit swings across training runs (variance); neither direction improves both at once.

The loss for a single yes/no probability, equal to Chapter 6's surprise evaluated at whichever probability the true label selects.

A library convention -- not a rule of linear algebra -- that lets a smaller shape stand in for a copy of itself, repeated enough times to match a larger shape, so mismatched shapes can still combine.

Whether a classifier's stated confidence matches its actual, empirical correctness rate.

How many adjustable numbers a model has to work with -- more capacity can fit finer patterns, but also finer noise.

Chapter 6's cross-entropy, with the true one-hot label as p and a softmax output as q -- the loss for choosing one of several classes.

The rule for differentiating a function built by feeding one function's output into another: multiply the outer function's slope by the inner function's slope.

A saved snapshot of a model's parameters at some point during training.

One class vastly outnumbering another in the data, which can make accuracy misleadingly high.

A model whose output is a category, or a probability over categories, rather than a plain number -- "spam or not" instead of "how much."

A diagram of the operations in a calculation, showing how each value depends on the ones before it.

A prepared, optional notebook of runnable code that reproduces a section's exact numbers; opened from the 🧪 card in Google Colab or downloaded.

The probability of one outcome, restricted to the cases where another outcome is already known to be true.

How close a model's probability output sits to 0 or 1, rather than 0.5 -- a description of the number, not a guarantee the model is right.

A table comparing a classifier's predictions against the true labels, broken down by class.

An operation that slides a small kernel across an input, computing a weighted sum at each position.

The two numbers (x, y) naming a point: the step across, then the step up.

How surprised you'd be by the true distribution, if you believed a guessed distribution instead.

A loss measuring how far a predicted probability distribution is from the true class label.

Splitting data into k folds, training k times, and averaging the k validation scores instead of trusting one split.

Expanding a training set with label-preserving variations of existing examples, instead of collecting new data.

Information from outside the training set (often from the validation or test set) improperly influencing training.

The PyTorch object that splits a dataset into batches and, when looped over, yields one batch at a time -- the tool actually performing the batch count Chapter 11, Section 2's own formula, m=\lceil N/B\rceil, predicts by hand.

The whole collection of examples used to train or evaluate a model -- not one row of it, the whole table.

The line (or flat surface) where a classifier's total z is exactly zero, separating the inputs it answers 1 for from the ones it answers 0 for.

How many layers a network has, counting every hidden layer plus the output layer, not the input layer.

The slope of a curve at one exact point, written f'(x); the number the curve's secant slopes settle on as the two points forming the secant move together.

A parameterized family of related network structures, searched systematically rather than hand-tuned one architecture at a time.

Storage looked up by a chosen key instead of a position, written {key: value, ...}.

A list of probabilities, one per possible outcome, that always adds up to exactly 1.

When the data a model meets after deployment no longer matches the data it was trained and evaluated on.

Multiply two vectors' matching entries and add every product -- one number out, written w \cdot x or \sum w_i x_i.

A regularization technique that randomly disables a fraction of neurons during each training step.

Which kind of number a tensor's entries are stored as -- PyTorch defaults new floating-point tensors to 32-bit precision (torch.float32), a library convention rather than a mathematical requirement, differing from NumPy's own 64-bit default.

Halting training once validation performance stops improving, to avoid overfitting.

A smooth ReLU-like activation function that matches ReLU exactly for positive inputs but eases its negative side smoothly toward a floor instead of a hard cutoff.

A two-part architecture where one network compresses an input into a representation and a second network generates an output from it.

The average surprise of a distribution, weighted by how likely each outcome is.

One complete pass through the entire training dataset.

The long-run average of a random quantity, found by multiplying each value by its own probability and adding.

A gradient that grows uncontrollably large as it's propagated backward through many layers, destabilizing training.

The harmonic mean of precision and recall, one number that's high only when both are reasonably high.

One fact about an example, recorded as one number in an agreed unit; a model's input is a fixed-length list of features.

Replacing each raw feature value x with z=(x-\mu)/\sigma, computed once from the training set, so every feature ends with mean 0 and variance 1.

Continuing to train a pretrained model's parameters (or some of them) on a new, usually smaller, dataset.

Repeats an indented block of code once per item in a list, naming the current item each pass.

A recipe written with numbers, letters, and operation signs; fill the letters with numbers to get one answer.

Computing a network's output by passing an input through each layer in order.

A rule that takes an input and returns exactly one output; written f(x), read "f of x".

A named block of code, built with def, that takes inputs (parameters) and hands back an answer with return -- Chapter 0's function machine, written in code.

Feeding one function's output straight in as the next function's input, written g(f(x)); a network's forward pass is composition applied once per layer.

A smooth ReLU-like activation function used in Transformer feed-forward layers, scaling the input by a smooth curve from statistics instead of cutting it off sharply.

How well a model performs on new examples it never trained on, not just the ones it already memorized.

A free, browser-based place, run by Google, that runs Python notebooks on Google's own computers -- nothing installs on your own machine.

A chip built to do many small arithmetic operations, like the multiply-and-adds in a forward pass, at the same time instead of one after another.

Every partial derivative of a function, stacked into one list (written \nabla f); one entry per input variable, or one entry per weight for a network's loss.

Rescaling a gradient when its size exceeds a threshold, to prevent an unstable, oversized parameter update.

The update rule w \leftarrow w - \eta L'(w) (or its gradient version for several parameters): repeatedly move every parameter a small distance against its own slope, so the loss keeps falling.

Every point (x, f(x)) of a function drawn at once on a pair of axes.

The number of new feature channels each layer of a DenseNet block adds to the running concatenation.

A simplified gated recurrent architecture with fewer gates than an LSTM.

A weight-variance formula, \sigma_W^2=2/n (fan-in n), that doubles Xavier's variance to compensate for ReLU zeroing about half its input's variance.

A layer between a network's input and output whose values aren't directly observed.

A recurrent network's running summary of everything it has processed in a sequence so far.

A loss that behaves like squared error near zero but grows only linearly past a chosen threshold delta, capping one outlier's influence.

A setting chosen before training (like the learning rate or number of layers), not learned from data.

A fork in the code: runs one indented block when a condition is true, a different one when it is false, never both.

Brings in a package of already-written code under a chosen nickname, such as import numpy as np.

Using an already-trained, fixed model to make a prediction on new data -- as opposed to training, which is still adjusting the weights.

The facts fed into a model, written x (or a list x_1, x_2, \ldots).

Labeling every pixel with both its class and which individual object instance it belongs to.

The height at which a line crosses the y-axis, its value at x = 0; the b in y = mx + b.

A measure of overlap between two boxes -- the area they share divided by the total area they cover together.

One parameter update, using one batch of data.

A small grid of learned weights that a convolutional layer slides across its input.

The extra, avoidable surprise a guess costs above the unavoidable minimum the true distribution's own uncertainty forces.

A transformation whose output provably still deserves the original label -- the one requirement every valid data augmentation must meet.

A group of neurons that all take the same input and compute their outputs in parallel.

A technique that normalizes a layer's inputs using statistics computed across that layer's own features, per example.

ReLU with a small nonzero slope on the negative side instead of a hard floor at zero, so a unit stuck on the negative side still receives a nonzero reading.

The fixed positive number \eta in the update rule that sets how big each step is; too small and training crawls, too large and it can diverge.

A rule that changes the learning rate \eta over the course of training, instead of holding it fixed -- usually shrinking it (decay), and sometimes ramping it up from a small value first (warmup).

A model whose output is a weighted sum of its features plus a bias; features are only ever multiplied by fixed weights and added.

The classical name for a linear model -- a numeric output predicted as a weighted sum of the features plus a bias.

Describes two classes of data that a single straight line (or hyperplane) can perfectly divide.

An ordered row of values in one name, written with square brackets and indexed from 0; the same object as a vector, once you allow for that indexing convention.

The answer to "what power do I need?" -- \log_2 8 = 3 because 2^3 = 8.

A raw, not-yet-a-probability score a network outputs for one class, before softmax turns it into a percentage.

A single number, built from every example's own error, where smaller always means a better rule.

A recurrent architecture with gates that let it selectively remember or forget information over long sequences.

A rectangular table of numbers, m rows by n columns; entry W_{ij} sits in row i, column j.

Multiplying two matrices, A then B; entry (i,j) of AB is row i of A dotted with column j of B. Defined only when A's column count equals B's row count.

The sum of a list divided by how many items it has, \bar{x} = \frac{1}{n}\sum x_i; the balance point of the list.

A loss that averages the squared difference between each prediction and its true value.

The fixed rule that turns an input into an output; every different setting of its weights and bias is a different model.

An update rule that keeps a decaying running average v of past gradients and steps against v instead of the current gradient alone, so a consistent direction builds speed.

Running several attention computations in parallel on different learned projections, then combining the results.

A single unit that computes a weighted sum of its inputs, adds a bias, and applies an activation function.

PyTorch's base building block for a network layer or model -- an object that holds its own parameters and defines how to compute an output from an input, so calling it runs exactly the math its parameters describe.

A post-processing step that discards overlapping duplicate detections, keeping only the highest-confidence box in each cluster.

A file made of small boxes called cells: some hold code that runs, some hold plain text and math that only display.

A list with a single 1 marking the correct class and 0 everywhere else.

Parentheses first, then powers, then multiplication and division, then addition and subtraction, left to right.

The number a model produces from its input; for a predicting model, the prediction \hat{y}.

A model fitting the noise and quirks of its training data so closely that it performs worse on new data.

Adding extra border values (usually zero) around an input before convolving, to control the output size.

A weight or bias the training process learns, as opposed to a hyperparameter chosen beforehand.

The total number of weights and biases in a layer or network, m(n+1) for a layer of m neurons taking n inputs each.

The derivative of a function of several variables with respect to just one of them, every other variable held fixed at a constant value; written \partial f/\partial w_1.

The simplest neural network -- a single neuron with a step-function output, used for binary classification.

Turns two matched lists of numbers into a picture; plt.plot(xs, ys) draws a line through the points they pair into, position by position.

An operation that reduces a feature map's size by summarizing (e.g. taking the max or average of) small regions.

Information added to a token's representation so a model that otherwise ignores order can tell positions apart.

Of everything a model predicted as positive, the fraction that actually was.

A guess produced by a fixed rule from facts already known, written \hat{y}; the truth it is compared against later is y.

A number between 0 and 1 saying how likely an outcome is; 0 means never, 1 means always.

The three vectors an attention mechanism derives from its input to decide what to look up and what to retrieve.

Of everything that actually was positive, the fraction a model correctly predicted.

The region of the original input that ultimately influences one value in a later layer.

A network that reuses the same weights at every step of a sequence, carrying a running hidden state forward from one step to the next.

Any technique that adds a preference to the loss beyond "fit the training data," to fight high variance.

An activation function that outputs its input unchanged if positive, and zero otherwise.

A re-expression of the input into new numbers chosen because they make the next layer's job easier, built by a hidden layer rather than designed by hand.

A flag set on a tensor telling PyTorch to track it on the autograd tape, so .backward() can later fill in its .grad -- the concrete switch behind Chapter 10's own tape metaphor.

Truth minus prediction for one example, y - \hat{y}; positive when the guess was too low, negative when too high.

A shortcut that adds a layer's input directly to its output, helping gradients flow through very deep networks.

Layer normalization with the mean-subtraction step dropped: rescale by the root mean square, \text{RMS}(z)=\sqrt{\frac1d\sum z_k^2}, instead of by the standard deviation.

A plot of true positive rate against false positive rate as a classifier's decision threshold sweeps through every value.

The computer, somewhere on Google's network, that actually runs a notebook's code; Runtime > Change runtime type picks whether it has a CPU only or a CPU plus a GPU.

A region where an activation function's output barely changes as its input changes, making its gradient near zero.

A single plain number, as opposed to a vector's ordered list -- 0.5 is a scalar, (0.5, 0.2) is not.

A straight line through two points on a curve; its slope is the average rate of change between those two points, not the curve's own changing slope.

The starting value that fully determines a pseudo-random number generator's whole sequence of 'random' draws -- the same seed on the same code always reproduces the same shuffle order and initial parameters.

Attention where the queries, keys, and values all come from the same sequence.

Labeling every pixel of an image with its class, without distinguishing separate objects of the same class.

Data where order carries meaning and length isn't fixed in advance -- a sentence, an audio clip, a price over time -- unlike a fixed-length feature vector.

How many rows, columns, (and further axes) a vector or matrix has, written as a tuple like (4, 2).

The capital Greek S, \Sigma, meaning "add these up"; \sum_{i=1}^{n} x_i adds items 1 through n.

The S-shaped activation function \sigma(z) = 1/(1+e^{-z}) that squashes any number into the range between 0 and 1.

A smooth ReLU-like activation function, SiLU(z) = z times sigma(z), scaling the input by its own sigmoid reading instead of cutting it off sharply.

How much a straight line climbs for each single step to the right; the m in y = mx + b, found from two points by rise over run.

A function turning a vector of raw scores into probabilities that sum to one.

A classifier that outputs a probability for each of several classes, using the softmax function.

The square of one example's residual, (y - \hat{y})^2; never negative, and larger misses count more than proportionally more.

The on-off activation rule -- output 1 when the neuron's total z is zero or more, output 0 when it is below zero.

Gradient descent whose gradient is estimated from a small random subset of the training data (a minibatch) instead of the whole dataset -- noisier per step, but far cheaper to compute.

The number of positions a kernel moves between each application during convolution.

Square each item, then add: \sum x_i^2 -- not the same as squaring the sum.

A number, -\log_2 p in bits, attached to a probability p; rare outcomes carry more surprise than common ones.

Giving every neuron in a layer a different starting weight, so gradient descent can make them specialize instead of updating them identically forever.

A list of a function's inputs beside their outputs.

An activation function, tanh(z) = (e^z - e^{-z})/(e^z + e^{-z}), squashing any number into the range between -1 and 1; the sigmoid stretched and shifted, since tanh(z) = 2*sigma(2z) - 1.

PyTorch's own name for the same ordered container Appendix B's NumPy array and Chapter 3's own vector already are -- with two abilities added on top: tracking its own gradient, and moving onto a GPU.

Data held out entirely until final evaluation, never used to make any training decision.

Splitting text into the smaller units (words, characters, or subwords) a model actually processes.

Adjusting a model's weights and bias to make its error score smaller -- by eye in this chapter, by a slope-following rule later.

The repeated cycle of computing a batch's loss, its gradients, and updating parameters.

Reusing a model trained on one task as the starting point for a related task.

An architecture built entirely from attention and feed-forward layers, with no recurrence or convolution.

To render math written with symbols and dollar signs as the textbook-style equation this course's own pages show, instead of the plain keyboard characters that were typed.

A model too simple to capture the real pattern in the training data, performing poorly even there.

Data held out from training, used to check a model's performance during development.

A gradient that shrinks toward zero as it's propagated backward through many layers, stalling learning in early layers.

A letter in a formula that stands for a number to be filled in later; the blank on the recipe card.

A labeled box holding one number, made with =; unlike math's variable-dl, a Python variable can be refilled by assigning it again.

How spread out a list of numbers is around its own mean, \sigma^2=\frac1n\sum(x_i-\mu)^2; its square root, \sigma, is the standard deviation.

An ordered list of numbers, written like (2, 3); each number is one entry, found by its position.

The fixed, finite list of distinct tokens a tokenizer can produce; every piece of text a model sees is broken into entries drawn from this one list.

The fixed multiplier on one feature in a weighted sum; it says how much the output changes per one-unit change in that feature.

A regularization technique that shrinks parameters toward zero during training, discouraging overly large weights.

Each feature multiplied by its own weight, the products added, and a bias added last -- z = w_1 x_1 + w_2 x_2 + b.

How many neurons sit in a given layer.

A learned vector representing a word, positioned so that similar words end up nearby in vector space.

A weight-variance formula, \sigma_W^2=2/(n+m) (fan-in n, fan-out m), that keeps a layer's pre-activation variance close to its input's, matched to sigmoid and tanh.

The two-input rule that outputs 1 when its inputs differ and 0 when they agree -- the smallest problem no single straight line can separate.