How to Label AI Data: A Practical Guide to Better Training Examples

Suppose your support inbox is a parade of messages: “I was charged twice,” “My password reset link expired,” and “Where is my package?” You want an AI system to route each note to the right team. Before the model can learn that job, someone needs to show it examples with the right answers attached.

That work is data labeling. It can be as simple as choosing a category or as detailed as tracing every pixel around a pedestrian. The hard part is rarely clicking the label. The hard part is deciding what each label means, handling the weird cases consistently, and checking that the finished data is useful for the real task.

This guide walks through a small, realistic project from blank page to reviewed dataset. No annotation platform or machine-learning degree required.

Step 1: Start with the decision the AI needs to make

“Label these messages” is not a complete task. What should the AI do with the label? For our example, it should send each customer message to a support queue. That goal suggests a small set of labels:

Notice that OTHER and NEEDS_REVIEW mean different things. “Other” is a clear, valid request that does not fit the supported queues. “Needs review” means we do not have enough evidence to decide. A good labeling scheme gives people a way to say “the data doesn’t tell me” instead of forcing a confident guess.

Also notice what the task does not ask: whether the customer sounds happy. “Your service is great, but my card was charged again!” is still BILLING. Labels should answer the project’s question, not every question an annotator could ask.

Step 2: Write a tiny rulebook before the big batch

A label name is a heading, not an instruction. Write down:

1. The unit: one complete customer message, not each sentence.
2. The question: which team should handle the main request?
3. Label definitions: what evidence counts for each category.
4. Priority rules: if a message mentions two topics, label the request that needs action first; if the main request is unclear, use NEEDS_REVIEW.
5. Exclusions: don’t infer a billing issue from the word “card” if the message is actually about account access.
6. Examples and counterexamples: pair an example that qualifies with one that looks similar but does not.
7. Escalation: specify which examples a reviewer or subject-matter expert must decide.

Here is our rule for the borderline message “My card won’t let me into the account”: assign ACCOUNT_ACCESS if the customer cannot sign in and there is no evidence of a failed payment. The word card alone is not enough for BILLING.

Keep the guide beside the data and give it a version, such as support-routing-v1. If the rule changes, record a new version. That small habit lets you explain later why two batches were labeled differently.

Step 3: Run a pilot and fix the confusing bits

Before labeling thousands of messages, take a small sample that includes common cases and tricky ones. A useful first pilot might be a few dozen messages, reviewed by at least two people if the project has more than one annotator. The number is a starting point, not a magic quality threshold.

Ask annotators to label independently, without seeing each other’s answers. Then compare the results. If both people call “I can’t get into my account” ACCOUNT_ACCESS, great. If one calls it BILLING because the customer mentions a card, the guide needs a better counterexample. Fix the rule, then try a fresh batch.

Disagreement is a signal about the task. It can mean an instruction is unclear, the example lacks context, the label set is missing a category, or the real-world case is genuinely ambiguous. A majority vote can help with clear factual categories, but it cannot manufacture certainty when everyone is missing the same information. Preserve NEEDS_REVIEW or ask a qualified reviewer when appropriate.

Step 4: Label the data, with or without AI suggestions

Once the pilot rules make sense, label the main batch. You can use a spreadsheet for a small project or an annotation tool for images, audio, and larger teams. A simple text dataset might look like this:

1item_id,text,label,guideline_version,review_status
2001,"I was charged twice for order 1842",BILLING,support-routing-v1,approved
3002,"My password reset link expired",ACCOUNT_ACCESS,support-routing-v1,approved
4003,"Where is my package?",DELIVERY,support-routing-v1,approved
5004,"How do I change newsletter emails?",OTHER,support-routing-v1,approved
6005,"Can you help?",NEEDS_REVIEW,support-routing-v1,escalated

For modern projects, AI can suggest a label so people spend their time checking rather than starting from a blank screen. Keep the suggestion visibly separate from the human decision, and measure how often people correct it. Review all uncertain suggestions and sample confident ones too: models can be confidently wrong, especially on rare groups or new kinds of messages.

An efficient next step is active learning: after an initial labeled batch, let a model flag examples it is unsure about, then prioritize those for people to review. Also sample rare categories and ordinary cases at random. If you review only the model’s most uncertain items, you can miss systematic errors where it is confidently wrong. In a 2025 Google Ads case study, researchers reported reducing the training examples for a specific ad-safety task from 100,000 to fewer than 500 with high-fidelity expert labels. That is a striking company-reported result for one task, not a promise that every project can shrink its dataset by the same amount.

Step 5: Measure quality without worshipping one score

Use more than one quality check:

• Agreement: when the same item is labeled independently by multiple people, how often do they choose the same answer? Check the results by label, not only as one overall percentage.
• A reviewed reference set: mix in examples whose labels were carefully decided by qualified reviewers. See whether annotators and any AI pre-labels match it.
• Random audits: inspect a random slice of completed work, including examples that looked easy.
• Targeted review: inspect uncertain items, rare categories, and groups where errors would have greater consequences.
• Error analysis: group mistakes by cause. Are BILLING and ACCOUNT_ACCESS often confused? Does OTHER swallow too many examples?

Agreement is a useful alarm bell, not proof that a label is correct. Two people can agree because the guide is clear, or because both made the same assumption. A reviewer should look at the disagreements and ask why they happened before changing the rules.

Keep an evaluation set separate from the examples used to train the model. If a message, duplicate, or near-duplicate leaks into both sets, your test score may look better than the system’s performance on genuinely new data.

Step 6: Leave a trail for the next person

When you share or reuse a dataset, document how it came to be. Record the source and collection period, what population or language it covers, how labels were defined, whether people or models produced the first pass, what quality checks were used, known gaps, and intended uses. Google’s Data Cards Playbook and the “Datasheets for Datasets” research both make the same practical point: dataset context is part of understanding the data, not decorative paperwork.

Protect people in both the data and the work. Use data you have permission to label; remove or protect personal details when possible; limit who can see sensitive examples; and give human labelers clear training, fair conditions, and a way to flag disturbing or ambiguous items. If the task is high stakes, use qualified reviewers and a documented escalation path.

Your 10-minute first project

Pick one small prediction, such as sorting support notes into BILLING, ACCOUNT_ACCESS, and DELIVERY. Write one paragraph defining each label, add an OTHER and NEEDS_REVIEW option, and collect 20 representative examples you are allowed to use. Label them yourself, then ask another person to label the same examples independently. Compare the disagreements and improve the guide before scaling up.

That’s the core loop: define, pilot, label, review, learn, repeat. Great AI data is not just a pile of answers. It is a careful record of what question you asked, what evidence counted, where people disagreed, and how you checked the result.

Sources and further reading

• Scaling AI Data Quality: Best Practices for Onboarding and Evaluating Annotators — practical guidance on instructions, training, calibration, and quality gates.
• How to Measure Inter-Annotator Agreement and Build Human Consensus — agreement, consensus, and targeted review.
• Achieving 10,000x Training Data Reduction with High-Fidelity Labels — a Google Ads case study of expert labels and active data selection; the reported results are task-specific.
• The Data Cards Playbook — documenting a dataset’s origin, annotation methods, decisions, and uses.
• Datasheets for Datasets — research proposing structured documentation for dataset creation and use.