Problem-solving recipes

A step-by-step procedure for every problem type this course teaches — what to do first, what to do next, and how to tell which case you're in.

What are you being asked to do?

What are you actually trying to do? Given a network's weights and an input, and need its output -> a forward pass (one neuron, or a whole network). Have a model and a loss and want to make the loss smaller -> gradient descent. Need the exact gradient of the loss with respect to every parameter, so gradient descent has something to work with -> backpropagation. Have all of the above and want to actually run a full training run -> the training loop, plus early stopping so it stops at the right time. Have a trained classifier and want an honest report of how good it is -> evaluate a classifier.

Computing a network's output

Forward pass of one neuron

Use when: you are given a neuron's inputs, its weights, its bias, and the name of its activation function, and need its output

Goal: the neuron's output $a$ , with the sum $z$ written down on the way

A single neuron, start to finish

Forward pass through a network

Use when: you are given a network's full architecture -- every layer's weights, biases, and activation -- along with its input, and need the network's final output

Goal: the output of every layer, ending with the network's prediction, $\mathbf{a}^{(L)}$

The forward pass, one layer at a time

The one-neuron recipe run once per layer, in order.

Making the loss smaller

Fit a model with gradient descent

Use when: you have a model with adjustable weights and a bias, a loss function, and a dataset, and you want to find weights that make the loss small

Goal: a full sequence of updates that drives the loss toward its lowest reachable value

Fitting the line with gradient descent

Backpropagation

Use when: you have a computational graph for a network (or any calculation) and need the exact gradient of its loss with respect to every one of its parameters

Goal: every trainable parameter's gradient, computed exactly once, ready to hand to gradient descent's update rule

Backpropagation through the 2-2-1 network, by hand

Supplies the gradient gradient descent's update rule needs.

Running a real training procedure

Train a network with the training loop

Use when: you have a network, a chosen loss function, and a training dataset already split into batches, and need to actually run gradient-based training rather than describe one forward pass in the abstract

Goal: a set of trained parameters, produced by repeated forward and backward passes over the whole dataset

Putting it together: the training loop

Stop a run at the right time, and keep the right weights

Use when: you have a training loop already checking validation loss on some schedule (Section 4), and want to stop automatically once it stops improving, without losing the best version

Goal: the best-performing set of parameters this run reached, saved and ready to use, with training halted once continuing stopped helping

Early stopping and checkpoints

An add-on to the training loop, not a replacement for it.

Reporting how good a trained model is

Evaluate a classifier

Use when: you have a trained classifier and a labeled test set, and need to report how good it actually is

Goal: a confusion matrix, precision, recall, and F1, and an honest read on whether those numbers can be trusted as reported

Precision, recall, and F1

Later in the course (chapters not yet redesigned)

Debug a network that isn't learning

Use when: training loss barely moves, explodes, or a network's predictions look no better than guessing, and you don't yet know which of the usual causes is responsible

Goal: the specific, confirmed cause of the training failure, found by the cheapest and most isolating checks first

A fixed diagnostic order

Apply transfer learning to a new task

Use when: you have a target task with limited labeled data and a network already pretrained on a large, related source task (most often ImageNet)

Goal: a trained model for the target task, adapted from the pretrained network at the point on the freeze/fine-tune spectrum its data and task similarity actually support

Transfer learning: freeze, replace, and fine-tune

Applying a pretrained CNN to a new task.

Build and train a small Transformer

Use when: you have Sections 1-5's pieces (embeddings, positional encoding, self-attention, the FFN, residual and norm, masked self-attention, cross-attention) and need to assemble them into one trainable encoder-decoder model

Goal: a complete, trained encoder-decoder Transformer that maps an input sequence to an output sequence

Training a small Transformer

Take a project from data to a deployed model

Use when: you're starting a new deep-learning project -- or evaluating an existing one -- and need to decide what representation, architecture, loss, and training strategy actually fit, not execute one technique you've already committed to

Goal: a trained, honestly evaluated, ready-to-ship model, with every major decision justified by what came before it, or a clear diagnosis of which earlier step needs to be redone

The end-to-end decision chain

The course's closing capstone -- data to a deployed model, one decision at a time.