Recipe

Backpropagation

Use when: you have a computational graph for a network (or any calculation) and need the exact gradient of its loss with respect to every one of its parameters

Goal: every trainable parameter's gradient, computed exactly once, ready to hand to gradient descent's update rule

Taught in Backpropagation through the 2-2-1 network, by hand

1. 1

   Draw the calculation as a graph of small, local operations (Section 2). Use one node per elementary step, ending at the loss.

   Example

   For this network: two hidden-neuron subgraphs (each a weighted sum then a sigmoid) feed one output subgraph (a weighted sum then a sigmoid). That output subgraph feeds the loss node.

   Why: the chain rule only works node by node. Each node needs its own well-defined local derivative, so the calculation must first be broken into pieces small enough for that.

2. 2

   Compute every node's value in dependency order, and write each one down -- don't discard anything.

   Example

   a_1\approx0.5744, a_2\approx0.3318, \hat y\approx0.6131, L\approx0.0748.

   Why: the backward pass's local derivatives (like the sigmoid's a(1-a)) are computed FROM these forward values, not re-derived from scratch.

3. 3

   At each node, use only its own forward-pass values. Find how its output changes with respect to each of its own inputs.

   Example

   The output sigmoid's local derivative: \hat y(1-\hat y)\approx0.6131(0.3869)\approx0.2372.

   ⚡ Shortcut

   A sigmoid node's local derivative is always a(1-a). Use the activation it already computed on the forward pass. Never re-evaluate \sigma from scratch at this step.

   Why: these are the individual factors the chain rule is about to multiply together. One wrong local derivative corrupts every gradient computed from it afterward.

4. 4

   Set \bar L=1. Then move backward one layer at a time. Each node's gradient is the sum of every downstream node's gradient times the local derivative connecting them (Section 3's \sum rule).

   Example

   \bar z_3\approx-0.0918, then \bar a_1=\bar z_3\cdot v_1\approx-0.0734 and \bar a_2=\bar z_3\cdot v_2\approx0.0551.

   • a node feeds only one place — the sum has a single term. Exactly the two-function chain rule from Chapter 8.
   • a node feeds more than one place, reused in two products or shared by two output paths — sum every path's contribution before moving further back.

   Why: a node's gradient can't be found until every node it feeds into already has its own. This is exactly why the pass must run output to input, never the other order.

5. 5

   At each hidden neuron, multiply the gradient arriving from above by that neuron's own activation slope. That gives \delta. Then multiply \delta by each of the neuron's own inputs to get its weight gradients, and by 1 for its bias.

   Example

   \delta_1=\bar a_1\cdot\sigma'(z_1)\approx-0.0734(0.2445)\approx-0.0179; then \bar w_{11}=\delta_1\cdot x_1\approx-0.0179 and \bar b_1=\delta_1\approx-0.0179.

   Why: this is the identical four-step neuron procedure Section 4 already ran in full, executed once per hidden neuron.

6. 6

   By the end of the backward pass, every trainable number in the network has exactly one gradient. Each was computed exactly once.

   Example

   All nine: \bar w_{11}\approx-0.0179, \bar w_{12}\approx0.0179, \bar b_1\approx-0.0179, \bar w_{21}\approx0.0122, \bar w_{22}\approx-0.0122, \bar b_2\approx0.0122, \bar v_1\approx-0.0527, \bar v_2\approx-0.0305, \bar b_3\approx-0.0918.

   Why: gradient descent (Chapter 9) needs one slope per parameter. No more, no fewer — before it can take a single step.

7. 7

   Apply Chapter 9's w\leftarrow w-\eta\cdot\bar w to every one of the nine parameters. Each uses its own gradient from the step above.

   Why: backpropagation only finds the slope. Gradient descent, not backpropagation itself, actually changes any weight. Confusing the two is one of this recipe's most common mistakes (see below).