Generate text, then verify it
Compare summarization and translation, inspect source evidence, and catch unsupported numbers.
- Distinguish summarization, translation and open-ended generation
- Review a generated output for coverage, factual support and useful wording
- Build a narrow numeric-fact check and explain its limits
NLP is not only about labeling text. A support team summarizes long tickets; a multilingual service translates help articles; a writing assistant drafts replies. These are text-to-text tasks. An encoder–decoder model can read an input and generate an output, while a decoder-style language model can perform the same task when given a suitable prompt. Both generate tokens one at a time, and neither guarantees that every claim is supported.
What does a good output preserve?
| Task | Must preserve | Example failure |
|---|---|---|
| Ticket summary | decision, amount, deadline, unresolved questions | says “refund issued” when only requested |
| Translation | meaning, names, numbers, negation and audience | turns “do not ship” into “ship” |
| Draft reply | approved policy and current case facts | invents a delivery promise |
A short answer can be fluent but omit a required fact. A long one can include unsupported details. Evaluate coverage, support from the source, and whether a person can act on the result. For consequential content, show the source and require human review.
Try it
Up to 15 XPReview three generated summaries
Source: “The warehouse shipped 12 crates on Monday. Three crates remain. The customer requested a status update; no refund has been approved.” Sort each candidate by its main problem.
“12 crates shipped Monday; 3 remain. Customer requested an update. No refund is approved.”
“The warehouse shipped 12 crates on Monday. The customer wants an update.”
“12 crates shipped Monday. A refund for the other 3 was approved.”
Automate narrow checks; review the rest
A simple checker can compare numbers in a source and a summary. For a task that requires every quantity, a missing number deserves review and a new number is suspicious. This check does not understand whether “3” means crates, days or dollars, and it cannot detect a false claim with no numbers. Use it as a warning signal, not a quality score.
1import re
2source = 'We shipped 12 crates and kept 3.'
3summary = 'We shipped 12 crates and kept 4.'
4numbers = lambda text: set(re.findall(r'\b\d+(?:\.\d+)?\b', text))
5print('missing:', sorted(numbers(source) - numbers(summary)))
6print('new:', sorted(numbers(summary) - numbers(source)))missing: ['3'] new: ['4']
Try it
Up to 15 XPChoose the right quality check
Match each problem to a useful check. No single automatic metric establishes that generated text is correct.
“A summary says 40 orders shipped; the source says 14.”
“The summary leaves out the customer’s deadline.”
“A translation uses a rude form of address for a customer.”
“A summary claims a refund was approved, but the source says only requested.”
Key takeaways
Summarization, translation and drafting are different text-to-text tasks with different success criteria.
Check important claims against source evidence, especially numbers, names, status and negation.
Automatic checks catch some errors; people still review meaning, omissions and appropriate language.
Lesson quiz
5 questions · pass with 4 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: apply NLP with Python
Try each text-processing idea in Python, run it against sample inputs, and use the results to see where the method works or falls short.
Flag missing and unsupported numbers
Read a source line and a proposed summary line. For this exercise, every numeric fact in the source is required. Find whole or decimal numbers, keeping first-seen order and ignoring repeats. Print missing: ... for source numbers absent from the summary and unsupported: ... for summary numbers absent from the source, comma-separated or none. This is a narrow audit, not a complete factuality test.
- Changed quantity
- All numbers preserved
- Decimal and repeated value
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…