Recipe

Apply transfer learning to a new task

Use when: you have a target task with limited labeled data and a network already pretrained on a large, related source task (most often ImageNet)

Goal: a trained model for the target task, adapted from the pretrained network at the point on the freeze/fine-tune spectrum its data and task similarity actually support

Taught in Transfer learning: freeze, replace, and fine-tune

1. 1

   Ask two questions before touching any code. How much does the target task look like the source task? How much labeled target data do you actually have?

   Example

   A team has 2,000 labeled photos of flower species. That task looks close to ImageNet's own natural-photo classes. Small dataset, similar task: this points toward freezing the whole backbone and training only a new head.

   • small target dataset and a target task that looks similar — freeze most or all of the backbone (see Steps 3-4).
   • a large target dataset, or a task genuinely different from the source — fine-tune more of the backbone (see Steps 3-4).

   Why: this decision drives every later step. Getting it wrong -- fine-tuning everything on a tiny dataset, or freezing everything on a large, different one -- is the single most common transfer-learning mistake.

2. 2

   Load the network with its pretrained weights intact. Then swap only the final classification layer for a freshly initialized one, sized to the target task's class count.

   Example

   model.fc = nn.Linear(model.fc.in_features, num_target_classes) replaces a 1,000-class output layer with, say, a 5-class one. Every earlier layer's pretrained weights stay untouched by this line.

   Why: the backbone's earlier layers already encode general-purpose image features (Chapter 4). Only the final class mapping is tied to the source task's specific 1,000 classes. That's the only part that must be replaced outright.

3. 3

   Set requires_grad = False on every parameter in the part of the backbone that Step 1 decided not to update yet. The new head's parameters stay trainable by default.

   Example

   for param in model.parameters(): param.requires_grad = False freezes everything. Run this right before the head is replaced. The new model.fc layer, created afterward, is still trainable by default -- it didn't exist yet when the loop ran.

   • freezing the whole backbone (Step 1's first case) — only the new head's parameters need requires_grad = True.
   • partial fine-tuning (Step 1's second case, moderate version) — unfreeze the later stages closest to the head, keep the earliest layers frozen.

   Why: an unfrozen backbone, trained on very little target data, risks overwriting good pretrained features with noise. That's exactly the overfitting Chapter 12 covers. It undoes the transfer this whole technique exists to exploit.

4. 4

   Fit the trainable parameters -- the new head, plus any unfrozen backbone layers -- on the target data. Use a noticeably smaller learning rate (Chapter 11) than an equivalent from-scratch training run would.

   Why: pretrained weights already sit in a good region of the loss landscape. A large learning rate risks kicking them out of it, before the new head has even stabilized. That throws away the very knowledge being transferred.

5. 5

   Check validation results (Chapters 6-7). Suppose they have plateaued below what the task needs, and there's enough labeled target data to risk it. Then unfreeze more layers, and continue training at an even smaller learning rate.

   • results are already good enough, or target data is genuinely small — stop here.
   • results plateau and there's enough labeled data to support it — unfreeze earlier layers too. This moves further along the same freeze-to-fine-tune spectrum Step 1 assessed.

   Why: this is Step 1's assessment, revisited with real evidence: validation results. It stands in for a guess made before any training happened.