Skip to content
Free public beta until December 4, 2026: every course is open, no card needed. After that, the first chapter of each course stays free and full access is a one-time $39 ($19 for up to 100 founding members chosen for helpful feedback). Your progress is kept. How the beta works

Recipe

Transfer learning and fine-tuning a pretrained backbone

Use when: you have a backbone pretrained on a large source task and a new target task with limited labeled data

Goal: a working classifier for the new task, trained in far less time and data than training from scratch

Taught in Transfer learning and fine-tuning

Quick recipe

  1. Judge the new dataset's size and similarity to the source task
  2. Replace the head with one sized to the new task
  3. Freeze the backbone's weights
  4. Train the head alone, at a normal learning rate
  5. If Step 1 called for fine-tuning, unfreeze the chosen layers at a low learning rate
  6. Evaluate once on the held-out test set
  1. 1

    Decide how much of the backbone is safe to retrain, before touching any code.

    • small dataset (a few hundred to a few thousand images), similar domain — freeze the whole backbone; train only a new head.
    • larger dataset, similar domain — freeze early layers, fine-tune later layers plus the new head.
    • larger dataset, very different domain (e.g. medical images vs. natural photos) — fine-tune most or all of the backbone, at a low learning rate.

    Why: a backbone's early layers learn generic features (edges, textures) that transfer almost everywhere. Late layers learn features increasingly specific to the source task. How much to retrain depends on how much labeled target data exists to retrain safely, and how different the target task looks from the source.

  2. 2

    Remove the pretrained backbone's original head and attach a new one sized to the new number of classes.

    Example

    Swapping a 784-input, 10-class head for a 5-class one: the old head's 7850 parameters are discarded; the new head has 784(5)+5=3925.

    Why: the pretrained head was trained to predict the source task's own classes, which almost never match the target task's classes.

  3. 3

    Exclude every backbone parameter from gradient updates, so only the new head trains at first.

    Why: a randomly-initialized new head starts out noisy. Training it alongside an already-good backbone risks large, noisy early gradients from the head damaging the backbone's useful pretrained features, before the head has learned anything sensible.

  4. 4

    With the backbone frozen this is an ordinary training run (Sections 1–3), except only the head's parameters ever change.

    ⚡ Shortcut

    If the target task is nearly identical to the source task, a frozen backbone with a trained head is often already enough. Skip straight to Step 6 without ever unfreezing anything.

    Why: the backbone isn't moving at all during this step, so nothing about earlier sections' training discipline changes except which parameters are included.

  5. 5

    Continue training with some backbone layers now included, at a noticeably lower learning rate than the head used alone.

    • training/validation loss starts diverging early — the dataset is likely too small for how many layers got unfrozen; freeze more of the backbone and retry.
    • validation performance improves and the train/val gap stays healthy — keep going, or unfreeze one more block if there's room to improve further.

    Why: the backbone's weights are already close to a good solution. A normal-sized learning rate applied to them, rather than to a freshly-initialized head, risks large updates -- updates that could overwrite useful pretrained features instead of gently adjusting them.

  6. 6

    Compare against the frozen-only baseline from Step 4 to judge whether fine-tuning was worth its cost.

    Why: Section 3's own test-set rule -- check once, after every training decision is already final, for an honest final number.

Common mistakes

  • Using the same learning rate for a freshly-initialized head and an already-trained backbone once both are training together.
  • Skipping Step 4 and unfreezing the whole backbone immediately, letting the random head's early noisy gradients damage pretrained features.
  • Assuming a strong pretrained backbone never needs any fine-tuning, without ever checking it against held-out target-task data.
  • Fine-tuning a small dataset's entire backbone at once instead of just the later layers, overfitting the backbone itself.