Build a trustworthy text dataset
Write a labeling guide, handle ambiguous messages, and split by conversation to prevent leakage.
- Define labels with examples and an escalation rule for ambiguous text
- Explain why near-duplicate messages must stay in the same data split
- Create a group-aware training and test split
A support team wants to route incoming messages to billing, account access, or other. A model can only learn from examples that people labeled consistently. Before training anything, write a short label guide: what each category means, what to do when a message mentions two problems, and when to ask a human to decide. Keep private customer details out of examples used for demos.
Label the work, not just the words
A useful rule is billing for charges, invoices and refunds; access for sign-in, password and account lockouts; other for everything else. If one message has two independent requests, send it for human review instead of silently choosing one. Two annotators should try the same small batch and discuss disagreements before labeling thousands of rows.
Try it
Up to 15 XPRoute the support inbox
Apply the label guide. Watch for a message that needs a human because it contains two different requests.
“I was charged twice for my plan.”
“The password reset link has expired.”
“Can your app export a PDF?”
“Refund the extra charge and unlock my account.”
“My invoice lists the wrong tax amount.”
Split conversations before training
A customer often sends a first message and a near-identical follow-up in the same thread. If one goes to training and the other to test, the test score rewards memorization. Keep every thread in one split. Use training data to fit the model, validation data to tune it, and a held-out test set once for the final check. For a real inbox, also compare results across time, languages and customer groups.
Try it
Up to 15 XPRepair the leaky split
These are messages from four support threads. Move cards until no thread appears in both sets while keeping at least two entire threads for testing. The live leak counter shows why a random row split is misleading.
Keep every thread in one set. Leave at least one training example and 2 whole threads in the test set.
Training set
Thread A17
I was billed twice this month.
Thread B04
My reset link expired.
Thread B04
Still locked out after resetting.
Thread C12
Any update on that invoice?
Test set
Thread A17
Following up on the duplicate charge.
Thread C12
The invoice has the wrong tax.
Thread D09
Can I export a PDF?
Thread D09
I need a PDF copy of the report.
1rows = [('A17', 'charge'), ('A17', 'follow-up'), ('B04', 'reset')]
2test_threads = {'B04'}
3for thread, message in rows:
4 split = 'test' if thread in test_threads else 'train'
5 print(split, thread, message)train A17 charge train A17 follow-up test B04 reset
Key takeaways
Write a label guide and review ambiguous cases with people.
Group related messages before splitting, or your test can leak training examples.
Tune on validation data; reserve the test set for the final estimate on new data.
Lesson quiz
5 questions · pass with 4 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: apply NLP with Python
Try each text-processing idea in Python, run it against sample inputs, and use the results to see where the method works or falls short.
Split support messages by thread
The first line is the number of whole threads to reserve for test. Each later line is THREAD|message. Sort the distinct thread IDs alphabetically and put the last N IDs in test. Print all training rows first, then all test rows, preserving the original order within each set, as train: THREAD|message or test: THREAD|message. This deterministic toy rule makes the grouping visible; real teams also check label balance and time.
- Follow-ups stay together
- One held-out thread
- No held-out threads
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…