The definitive guide to keypoint annotation for ML teams. Covers COCO keypoint format, CVAT keypoint annotation workflow, free tools comparison, use cases in pose estimation and robotics — plus when to outsource to a professional keypoint annotation service.
A bounding box tells your model that a person exists in a frame. That’s it.
It cannot tell your model how that person is standing. It cannot tell a surgical robot where to grip. It cannot tell a sports analytics system whether the athlete’s knee angle increases injury risk. It cannot tell a face authentication system which facial micro-expressions belong to a live human versus a spoofing attack.
That’s the problem keypoint annotation solves — and it’s why demand for precise keypoint annotation services has accelerated sharply across robotics, healthcare, biometrics, augmented reality, and human pose estimation.
If you’re an ML engineer, CV researcher, or AI Founder building any model that needs to understand structure, pose, or movement — this guide covers everything: what keypoint annotation is, the COCO format, the best tools (free and paid), real-world use cases, and what to look for when outsourcing to a keypoint annotation service.
Keypoint annotation is the process of marking specific, precisely defined coordinate points on objects in images or video frames. Each marked point — called a landmark or keypoint — represents a meaningful structural location: a joint on a human body, a corner of an eye, a mechanical pivot on a robotic part, or a reference point on a vehicle.
Unlike bounding boxes (which capture presence) or segmentation masks (which capture shape), keypoints capture geometry and spatial relationships — the internal architecture of the object.
A trained model learns not just where an object is, but how it’s configured. This distinction is what separates models that can detect something from models that can understand it.
Common keypoint annotation schemas:
The COCO (Common Objects in Context) keypoint format is the industry standard for human pose estimation and the most widely supported keypoint schema across ML frameworks. If you’re building a pose estimation model with YOLOv8-pose, MMPose, or any other modern architecture — you’ll encounter COCO keypoint format.
The COCO keypoint schema (17 points):
0: nose
1: left_eye
2: right_eye
3: left_ear
4: right_ear
5: left_shoulder
6: right_shoulder
7: left_elbow
8: right_elbow
9: left_wrist
10: right_wrist
11: left_hip
12: right_hip
13: left_knee
14: right_knee
15: left_ankle
16: right_ankle
How keypoints are stored in COCO JSON format:
Each annotation object includes a keypoints field — a flat list of [x, y, visibility] triplets for each point:
{
"id": 1,
"image_id": 101,
"category_id": 1,
"keypoints": [
320, 240, 2,
310, 225, 2,
330, 225, 2,
295, 220, 1,
345, 220, 0,
...
],
"num_keypoints": 14,
"bbox": [280, 180, 140, 320],
"area": 44800,
"iscrowd": 0
}
Visibility flags — the detail most annotators get wrong:
| Value | Meaning |
|---|---|
0 |
Not labeled (point not annotated) |
1 |
Labeled but not visible (occluded or out-of-frame — coordinates are estimated) |
2 |
Labeled and visible (annotator can clearly see the point) |
This visibility flag is not optional noise. Models trained with correct visibility labeling learn to handle occlusion gracefully. Models trained without it — or with incorrect flags — produce jittery, unreliable keypoint predictions in production, especially in crowded scenes.
YOLOv8-pose keypoint format:
For YOLOv8 pose training specifically, keypoints are stored in YOLO TXT format with normalized coordinates:
<class_id> <x_center> <y_center> <w> <h> <kp1_x> <kp1_y> <kp1_vis> <kp2_x> <kp2_y> <kp2_vis> ...
All coordinates normalized between 0 and 1. Each person gets one line. All 17 COCO keypoints listed in order.
CVAT keypoint annotation is the gold standard for professional teams running structured annotation workflows. CVAT supports the full COCO 17-point skeleton schema natively and lets you define custom keypoint schemas for non-human use cases.
How to annotate keypoints in CVAT:
CVAT’s interpolation feature is critical for video: annotate the keyframe skeleton, set interpolation mode, and CVAT propagates the skeleton across frames. Human reviewers then correct tracking errors. This workflow is 5–8x faster than frame-by-frame manual annotation on video datasets.
CVAT keypoint annotation: best for — team workflows, video datasets, self-hosted data privacy requirements, and COCO-format output.
Roboflow supports keypoint annotation with SAM-2-powered assistance and direct export to YOLO Pose format. The auto-labeling tools can accelerate initial keypoint placement on common human pose tasks, though human QA remains mandatory for precision applications.
Best for — solo ML engineers, startup MVPs, and teams that need fast annotation with direct YOLO Pose export.
Label Studio supports fully custom keypoint schemas via its XML template system. If you’re annotating something other than human pose — industrial parts, animal skeletons, custom facial landmark schemas — Label Studio gives you the most flexibility in defining your own keypoint taxonomy.
Best for — research teams, non-standard object types, multi-modal projects where keypoint annotation is one of several data types.
Supervisely has one of the strongest keypoint annotation interfaces in the market, with built-in smart tooling and team management. Strong choice when you’re running keypoint annotation at scale with multiple annotators and need robust QA workflows.
Best for — larger teams, mixed annotation types alongside keypoints, enterprise data governance requirements.
The most established use case. Keypoint annotation trains models to detect the human body skeleton across a range of applications:
The most precision-demanding keypoint annotation task. A 68-point facial landmark schema — covering the jaw line, eyebrows, eyes, nose, and mouth — powers:
This is the emerging frontier for keypoint annotation beyond human pose:
Keypoint annotation has higher precision requirements than any other annotation type. The margin for error is measured in pixels — and pixel-level errors compound across training.
The five failure modes that wreck keypoint-trained models:
Incorrect visibility flagging. An annotator marks an occluded knee as “visible” because they can infer its position. The model learns that it should produce confident predictions for invisible joints — the exact opposite of what you want.
Inconsistent anatomical convention. Two annotators have different intuitions about where exactly the “shoulder keypoint” sits — the top of the deltoid, the glenohumeral joint center, or the acromion. Without a strict labeling guide with reference images, your dataset will have two different “shoulders” and your model will learn a noisy average of both.
Temporal jitter in video. A keypoint that jumps 5–10 pixels between consecutive frames where the subject hasn’t moved. Models trained on jittery video annotations produce jittery predictions — a critical failure in surgical and rehabilitation applications where temporal smoothness is a hard requirement.
Missing edge case coverage. No annotations at extreme joint angles (arms fully extended overhead, deep squat positions, lateral bends). The model will fail exactly at the positions that matter most for athletic performance or injury detection.
Schema drift. Annotators progressively shift their interpretation of a keypoint placement convention over a long project without anyone catching it. The first 500 images use one convention; the last 500 use a slightly different one. Your training data contains a silent, invisible inconsistency.
These failures are invisible in your annotation tool. They only surface when model performance degrades in production — often weeks after the dataset was “complete.”
Be honest with yourself about what in-house keypoint annotation actually costs.
Your ML engineers spend their time debugging annotation inconsistencies, writing labeling guides, and reviewing annotator output instead of building model architecture and training pipelines. That’s your highest-cost resource doing your lowest-leverage work.
Outsource when:
Keep in-house when:
Improving annotation quality from 85% to 97% accuracy on a keypoint dataset has a measurable impact on model performance that exceeds most architecture upgrades at equivalent cost. This is not a theoretical claim — it’s the operational reality of data-centric AI.
The coordinate precision of your keypoints sets the upper bound of your model’s localization accuracy. You cannot train above your label quality.
At AI and ML Network, keypoint annotation is one of our core specializations. We work daily with COCO 17-point human pose schemas, custom facial landmark schemas (68-point and 98-point), hand skeleton annotation, and industrial part keypointing. Every project includes:
We work at affordable rates compared to other keypoint annotation services, and we maintain accuracy standards that give your model a clean foundation to train from.
Need a free 50-image sample batch? Send us your keypoint schema and we’ll annotate a sample batch — COCO format, visibility flags included. Judge the quality before committing to anything.
Alt text for cover image: Diagram showing COCO 17-point keypoint annotation schema on a human body with visibility flags, alongside CVAT annotation interface for keypoint labeling in 2026