Loading
0x40Lesson 5 of 9

Let the model think: reasoning and verification

Use step-by-step reasoning where it helps, separate thinking from the answer, and check the steps.

20 min 5-question quiz 2 code exercises
By the end of this lesson you can
  • Explain when chain-of-thought prompting helps
  • Separate reasoning from the final answer for reliable parsing
  • Verify reasoning steps instead of trusting them

Chain-of-thought prompting (Wei et al., 2022) asks a model to work through intermediate steps before answering. On multi-step problems - math, logic, planning, comparing options - this often improves accuracy, because each generated step becomes context for the next.

Practical patterns:

  • Ask for reasoning in <thinking> tags and the result in <answer> tags, so code can extract just the answer.
  • Many current models have built-in extended thinking; with them, a high-level instruction (“think carefully about edge cases”) often beats a hand-written step list.
  • Self-consistency (Wang et al., 2022): sample several reasoning paths and take the majority answer.

Reasoning text is not a guarantee: steps can contain mistakes, and the explanation may not reflect how the answer was really produced. Check what matters - especially arithmetic, which tools do better.

Try it

Up to 15 XP

Does it need step-by-step reasoning?

Decide whether asking for reasoning is likely to help.

0 of 5 sortedScore 0/0
  • “Which of three phone plans is cheapest for 14 GB a month with roaming in two countries?”

  • “Translate “Good morning” into French”

  • “Does this contract clause conflict with clause 7?”

  • “Extract the email address from this signature”

  • “Plan a database migration with zero downtime”

extract_answer.py
import re
reply = "<thinking>14 GB costs 25 on plan B...</thinking><answer>Plan B</answer>"
print(re.search(r"<answer>(.*?)</answer>", reply, re.DOTALL).group(1))
Output
Plan B

Key takeaways

  • Ask for step-by-step reasoning on multi-step problems; skip it for simple ones.

  • Keep reasoning and answer in separate tags.

  • Verify important steps - reasoning text can be wrong or unfaithful.

Lesson quiz

5 questions · pass with 4 correct · up to 50 XP

Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.

Practice: write Python

Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.

Exercise 1

Extract the answer from a reasoning reply

+25 XP

Replies are separated by lines containing only ===. For each reply, print the text inside <answer>…</answer> (stripped; it may span lines - join its lines with spaces), or no answer tag if it’s missing.

  • Three replies
main.py
Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Exercise 2

Check the arithmetic in a reasoning trace

+25 XP

Each input line is a reasoning step. For every line containing A OP B = C (integers, OP one of + - * /), print line N: ok or line N: wrong, A OP B is X (for /, X may be a decimal - print it with :g). Skip lines without an equation.

  • A phone-plan comparison
main.py
Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Questions about this lesson

Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.

Loading posts…