Recipe

Turn per-pixel class scores into a segmentation map

Use when: a network has already produced a raw score for every class, at every pixel, and you need the actual labeled map

Goal: a class map the same height and width as the input image, one class label per pixel

Taught in Semantic vs. instance segmentation

1. 1

   A trained network's output layer produces a raw score for each of the C possible classes. It does this independently at every pixel position. (Section 2 covers how those scores actually get computed; for now, treat them as already in hand.)

   Why: a classifier's raw output is always a score per class, never a label directly. Segmentation just runs this same decision once per pixel instead of once per whole image.

2. 2

   At every pixel, independently, pick whichever class scored highest.

   Example

   Three pixels in a row, three classes (car, road, sky):

   Pixel	car	road	sky	argmax
   1	2.1	0.4	-1.0	car
   2	1.8	1.9	-0.5	road
   3	0.6	0.6	-2.0	tie

   • one class scores strictly highest — keep that class, no special handling needed (pixels 1 and 2 above).
   • two or more classes tie exactly — break the tie with a fixed, consistent rule (e.g. the lower class index wins), so the output stays the same every time you run it (pixel 3 above).

   Why: the highest-scoring class is the network's single best guess for that one pixel. Picking it is the actual step that turns raw scores into a label.

3. 3

   Stack every pixel's chosen class back into a grid the same height and width as the input image.

   ⚡ Shortcut

   Sometimes the only question that matters is foreground vs. background, not which specific class. In that case, collapse the class map into two groups: the target class versus everything else. Do this before any downstream step. It throws away information the full map still has, so only reach for it once you've confirmed the class name itself doesn't matter for the task.

   Why: the input was a full image, so a genuinely useful output has to label every one of its pixels at once. It can't just report a handful of scattered decisions.

4. 4

   The class map tells you what class every pixel belongs to. It carries nothing about which specific object a pixel came from. So two touching or overlapping same-class objects merge into one region with no seam. Separating individual objects of the same class is Section 2's own subject. Scoring how well a predicted map matches ground truth is Section 3's.

   Why: that gap is exactly the semantic/instance boundary this section just drew. A semantic map was never built to answer "how many of these are there," so it's not a bug when it can't.