What Is Data Labeling? The Human Work That Teaches AI

Have you ever been asked by a website to "select all images with traffic lights" before you could log in? You probably clicked a few squares, sighed, and moved on. But in that moment you were doing something surprisingly important: telling a computer what's in a picture.

That's data labeling, and it's one of the least glamorous but most essential parts of artificial intelligence. Behind almost every AI system that recognizes faces, filters spam or understands speech, there are huge numbers of examples that people carefully labeled first.

This article explains data labeling in plain language: what it is, why AI needs it, who does it, and why getting it right matters so much.

You may have labeled data already

Those challenges are called CAPTCHAs, and their main job is to tell humans apart from bots. But Google, which runs the popular reCAPTCHA service, has said that people's answers over the years have also helped digitize old books and newspapers and improve its image recognition, for example in Google Maps. Millions of people became part-time data labelers without realizing it.

What is a label?

A label is simply the right answer attached to an example. The example might be a photo, an email, a sentence or a sound clip. The label says what it is, or what it means:

Why does AI need these? Most machine learning learns the way a student practices with an answer key. Show a computer thousands of photos labeled "cat" or "not cat", and it gradually learns what cats look like. Without labels, it has examples but no answers, and nothing to learn the right answer from.

Data labeling (also called data annotation) is the work of adding those labels.

Many kinds of labeling

Labeling isn't just writing one word next to a picture. Depending on what the AI needs to learn, labelers might:

• Tagging: one label for a whole item. "This photo shows a dog." "This review is negative."
• Boxes: drawing a rectangle around every object, like each car and pedestrian in a street photo. Self-driving car companies need millions of these.
• Outlines: tracing the exact shape of things, pixel by pixel. Used for medical scans and for blurring video call backgrounds.
• Text marking: highlighting parts of a sentence, like names of people, places and dates, so an AI learns to find them.
• Transcription: typing out exactly what's said in a recording, so voice assistants learn to understand speech.
• Ranking: comparing two answers from an AI chatbot and picking the better one. This kind of human feedback is a key step in making chatbots helpful and polite.

Who does the labeling?

The answer is: lots of people, in lots of ways.

• Crowd workers: online platforms such as Amazon Mechanical Turk, launched in 2005, split labeling into tiny tasks that anyone can do for a small payment. The famous ImageNet photo collection, with more than 14 million labeled images, was labeled this way with the help of about 49,000 workers from 167 countries.
• Labeling companies: businesses that employ teams of trained labelers, often in many different countries.
• Experts: some labels need specialist knowledge. Doctors label medical scans, lawyers label contracts, and linguists label grammar.
• Everyday users: when you click "report spam", tag a friend in a photo or give a thumbs-down to a chatbot answer, you may be producing labels too.
• AI itself, with humans checking: increasingly, an AI suggests labels and people correct them, which is much faster than starting from scratch.

How a labeling project works

Good labeling is much more organized than people clicking randomly. A typical project runs in a loop:

1. Write guidelines. Clear rules with examples: does a dog in a poster count as "dog"? Is a sarcastic review positive or negative? These decisions must be made up front.
2. Label. People (often helped by tools) label the examples.
3. Check quality. Reviewers spot-check work, and the same item is often given to several people to compare answers.
4. Train the model. The labeled data is used to teach the AI.
5. Find mistakes. Where the model struggles, there are often confusing guidelines or wrong labels. Fix them and go around again.

Why it's harder than it sounds

Labeling seems easy until you try it. Is that animal in a blurry photo a wolf or a husky? Is "Great, another Monday" a positive or negative sentence? Where exactly does a tumor end on a scan? Reasonable people disagree, and tired people make mistakes.

One common fix is to ask several people and go with the majority. Here's how much that helps, in a simple model of the math:

With three people who are each right 80% of the time, the majority is right about 90% of the time. With five, about 94%. The catch: this assumes people make mistakes independently. For genuinely tricky items, many people get fooled the same way, so in practice teams also use experts, clear guidelines and careful review.

Why label quality matters so much

There's a well-known saying in computing: garbage in, garbage out. If the answer key is wrong, the student learns the wrong lessons.

To show this, we ran a small real experiment. We trained a simple model on 400 made-up examples, then did it again and again, each time deliberately flipping more of the labels to the wrong answer:

With perfect labels, the model gets 93% of new examples right. When 20% of the training labels are wrong, it drops to 88%. At 40%, it's down to 71%. Every wrong label teaches the model a little bit of nonsense.

And wrong labels are more common than you might think. A 2021 study by researchers at MIT and elsewhere checked ten widely used AI test datasets and estimated that, on average, at least 3.3% of their labels were wrong, including at least 6% in a popular part of ImageNet. Those are datasets that researchers use to decide which AI systems are best.

The human side of labeling

Data labeling is real work done by real people, and it raises real questions:

• Pay and conditions: much labeling is done for low wages, often in lower-income countries. In 2023, a TIME magazine investigation reported that workers in Kenya labeling harmful content to make a chatbot safer were paid less than $2 an hour.
• Disturbing content: to teach AI to filter violence, abuse and hate speech, someone first has to look at it and label it. That can take a serious toll on mental health.
• Recognition: labelers are rarely credited, even though AI systems couldn't exist without them.
• Privacy: labeling sometimes involves personal data, like voice recordings or photos, which needs to be handled carefully and with consent.

Many researchers and companies are working on better standards: fair pay, mental health support, clear consent, and tools that reduce how much disturbing content any one person has to see.

Where labeling is heading

Labeling isn't going away, but it's changing:

• AI-assisted labeling: models suggest labels and people review them, making the work faster.
• Fewer labels, smarter choices: techniques that pick the most useful examples to label, so fewer are needed.
• Synthetic data: computer-generated examples that come pre-labeled, like simulated street scenes for training self-driving cars.
• Learning without labels: today's chatbots first learn from huge amounts of unlabeled text by predicting the next word. But human feedback, like ranking answers, is still used to make them helpful and safe.

A small glossary

• Data labeling (or annotation): adding the right answers to examples so AI can learn from them.
• Label: the answer attached to an example, like "cat" or "spam".
• Labeler (or annotator): a person who adds labels.
• Guidelines: the rules that tell labelers how to label consistently.
• Bounding box: a rectangle drawn around an object in an image.
• Transcription: writing down what's said in audio.
• Majority vote: giving the same item to several labelers and keeping the most common answer.
• Label noise: wrong or inconsistent labels in a dataset.
• Crowdsourcing: splitting work into small tasks done by many people online.
• Human feedback: people rating or ranking AI answers to help improve it.
• Synthetic data: artificially generated examples, often created already labeled.

Wrapping up

Behind the impressive things AI can do is a lot of patient human work. Data labeling turns raw photos, text and sound into examples with answers, and those answers are what machine learning learns from. The quality of the labels sets a ceiling on the quality of the AI: clear guidelines, careful checking and fair treatment of the people doing the work all show up in the final result.

So the next time a website asks you to find all the traffic lights, you'll know you're doing a tiny bit of the work that teaches computers to see.