Tech World Software SchoolTWSS
0x00Lesson 1 of 18

Turn text into tokens

Normalize text and split it into pieces a language system can process.

15 min 7-question quiz 2 code exercises
By the end of this lesson you can
  • Explain what natural language processing is and where you already use it
  • Normalize text and split it into tokens
  • Tell tokens (every occurrence) from types (distinct words)

Natural Language Processing (NLP) is how computers work with human language. You use it dozens of times a day: the spam filter that hides junk mail, the search box that finds “running shoes” when you typed “shoes for runners”, autocorrect, translation, and chat assistants.

All of them start the same way. To a computer, text is just a sequence of characters - "Cats, DOGS" is C, a, t, s, , and so on. Before anything clever can happen, the text is cleaned up (normalized) and cut into units called tokens.

normalize.py
1text = "  Cats, DOGS, and cats!  "
2normalized = " ".join(text.lower().split())
3print(normalized)
4print(normalized.split())
Output
cats, dogs, and cats!
['cats,', 'dogs,', 'and', 'cats!']

lower() makes Cats and cats the same, split() cuts on any run of whitespace, and " ".join(...) glues the pieces back with single spaces. Notice the punctuation still stuck to cats, and cats! - the next lesson fixes that.

What is a token?

A token is one unit of text that a system processes. Depending on the tool, tokens can be whole words, punctuation marks, single characters, or pieces of words (“token” + “izer”). A collection of text you study or learn from is called a corpus.

Try it

Up to 15 XP

Tokenizer playground

Type anything, then flip the switches one at a time. Watch two numbers: tokens (every word occurrence) and distinct tokens (the vocabulary).

  • Turn on Lowercase: why does the distinct count drop while the token count stays the same?
  • Turn on Remove punctuation: which count changes now?
13 tokens8 distinct (the vocabulary)
  1. The
  2. cats
  3. chased
  4. the
  5. mice
  6. ,
  7. and
  8. the
  9. mice
  10. chased
  11. the
  12. cats
  13. !

Tokens and types

“the cat saw the dog” has 5 tokens but only 4 types: the appears twice. Types are the distinct tokens - the vocabulary. Normalizing (like lowercasing) merges types, which shrinks the vocabulary a model has to learn.

Key takeaways

  • NLP turns raw characters into units a program can count and compare - starting with tokens.

  • Normalization (lowercasing, tidying whitespace) makes text consistent, but can throw away useful information.

  • Tokens count every occurrence; types count distinct tokens - the vocabulary.

Lesson quiz

7 questions · pass with 5 correct · up to 50 XP

Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.

Practice: apply NLP with Python

Try each text-processing idea in Python, run it against sample inputs, and use the results to see where the method works or falls short.

Exercise 1

Normalize a sentence

+25 XP

Read one line, lowercase it, trim the ends, and collapse repeated whitespace to one space. Print the result.

  • Mixed case and spaces
  • Already tidy
main.py
Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Exercise 2

Count tokens and types

+25 XP

Read one line. Lowercase it and split it on whitespace. Print tokens: N and then types: M, where N is the number of tokens and M the number of distinct tokens.

  • Repeated words
  • All different
main.py
Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Questions about this lesson

Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.

Loading posts…