Recipe
Run 2D/3D pose estimation on an image
Use when: you have an image (or short frame sequence) with one or more people and need each person's own skeleton, not just a bounding box
Goal: a full skeleton per person -- every keypoint located (or explicitly flagged as unreliable) and connected in the correct anatomical order
Taught in 2D/3D pose keypoints and skeletons
Quick recipe
-
1
Find where each person is in the image before looking for any individual joint.
Example
A photo with three people detected at three boxes (top-down) versus the same photo with no boxes at all, only a single whole-image pass planned (bottom-up).
- top-down — run Chapter 13's own person detector first. The keypoint model then runs once per detected box.
- bottom-up — skip the separate person-detection pass. The keypoint model below runs once over the whole image instead, and step 3's grouping answers "which person" for you.
Why: every later step needs to know whether it's already working inside one person's own crop, or still looking at the whole scene.
-
2
For every keypoint the model tracks, compute its heatmap and read off the peak as that keypoint's location. The peak's own score becomes its confidence.
Example
The right-elbow heatmap above peaks at one cell with score 0.91. That cell's own image coordinate becomes the reported elbow location.
Why: a heatmap's peak is the model's single best guess. Its height is the honest confidence that guess deserves, separate from any other keypoint's own confidence.
-
3
Connect the located keypoints in the fixed order a real skeleton connects (wrist to elbow, elbow to shoulder, and so on) to form one complete skeleton per person.
- top-down — every keypoint already belongs to the one person whose box it came from. Assembly is just connecting the dots in order.
- bottom-up — first GROUP the whole image's loose keypoints by which person they belong to (nearby, matching joints), then connect each person's own group in order.
Why: a pile of correctly-located but unconnected points isn't a skeleton yet. Assembly is what turns individual detections into one structured object per person.
-
4
Compare each keypoint's confidence against a threshold. For anything below it, don't just plot the raw heatmap peak as if it were reliable.
Example
A right wrist heatmap peaks at only 0.22 confidence, because the wrist is hidden behind the torso. That's too low to trust directly, but the left forearm's own known length (\approx 0.26) narrows down a plausible depth for it.
- a nearby, high-confidence joint plus a known body constraint (bone length, symmetry with the other side of the body) exists — use it to guess a plausible position, the way the widget above checks a hidden wrist against the opposite forearm's own length.
- no such constraint is available — report the joint as missing rather than plotting an untrustworthy guess.
Why: a low-confidence peak is still a number. Plotting it unchallenged looks exactly like a confident detection to anything downstream that only reads the coordinate -- the confidence score exists specifically so this case can be caught.
Common mistakes
- Treating a person-detector's box confidence as if it also certified every keypoint inside that box. They're separate scores, and a confident box can still contain a low-confidence, hard-to-see joint.
- Assembling a bottom-up model's output by proximity alone with no grouping step, silently swapping two nearby people's joints.
- Plotting every heatmap peak at face value, regardless of its own confidence, instead of flagging the low ones.
- Assuming 3D lifting from a 2D skeleton needs no separate step. A 2D skeleton has no depth information at all until a lifting model (or, in the occluded case, a body constraint) adds one.