Choose a tokenization strategy
Split sentences consistently and make punctuation handling explicit.
- Extract word tokens with a regular expression
- Compare what different tokenizers keep and discard
- Split text into sentences and spot the classic traps
Tokenization divides text into units. A whitespace split is quick, but leaves punctuation attached ("hello,"). A regular expression - a pattern for matching text - can pick out exactly the spans you want. Real tokenizers also handle contractions, other writing systems and subwords; there is no single best boundary for every task.
1import re
2text = "NLP is useful, isn't it?"
3tokens = re.findall(r"[a-z]+(?:'[a-z]+)?", text.lower())
4print(tokens)['nlp', 'is', 'useful', "isn't", 'it']
Reading the pattern [a-z]+(?:'[a-z]+)? left to right:
[a-z]+- one or more lowercase letters (a word),(?:'[a-z]+)?- optionally, an apostrophe followed by more letters, soisn'tstays one token.
Three tokenizers, three answers
Try it
Up to 15 XPWhich tokenizer made this?
Each list of tokens came from one of three tokenizers. Match them. Pay attention to what happens to punctuation and capital letters.
“['Hello,', 'world!']”
“['Hello', ',', 'world', '!']”
“['hello', 'world']”
“['E-mail', 'me']”
“['E', '-', 'mail', 'me']”
“['e', 'mail', 'me']”
Sentences are tokens too
Many tasks first split text into sentences. Splitting on . sounds easy until you meet “Dr. Smith arrived at 3.30 p.m.” - three of those periods do not end a sentence.
Other languages bring other puzzles: Chinese, Japanese and Thai are usually written without spaces between words, so even finding word boundaries needs a learned model or a dictionary.
Key takeaways
Regular expressions let you say exactly what a token is.
Different tokenizers make different choices about punctuation, case and hyphens - pick one for your task and apply it consistently.
Sentence splitting is harder than “split on periods”, and some languages need help even to find words.
Lesson quiz
7 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: apply NLP with Python
Try each text-processing idea in Python, run it against sample inputs, and use the results to see where the method works or falls short.
Extract word tokens
Read a line, lowercase it, extract words with re.findall(r"[a-z]+", text), and print them separated by one space. Ignore punctuation.
- Punctuation
- Repeated spaces
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Split text into sentences
Read one line and split it into sentences at ., ! or ? (one or more in a row). Print each non-empty sentence on its own line, with surrounding spaces removed.
- Three sentences
- Excited punctuation
- No ending mark
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…