0xD0Lesson 14 of 16

Evaluate LLMs and use them responsibly

Build evals for your task, use metrics like exact match and pass@k, and know the limits of benchmarks and LLM judges.

22 min 6-question quiz 2 code exercises

By the end of this lesson you can

Design a task-specific evaluation set
Compute exact match and pass@k
Recognize benchmark contamination and LLM-judge biases

Public benchmarks tell you how a model does on someone else’s test. For your product you need evals: a set of realistic inputs with a way to grade outputs, run every time you change the model, prompt or tools.

Grading options, from strict to flexible:

Exact match (after normalizing case, punctuation and articles) for short factual answers.
Execution for code: run the tests. pass@k (Chen et al., 2021) is the chance that at least one of k samples passes.
Rubrics graded by people or by an LLM judge for open-ended text.

LLM judges scale well but have known biases - favoring longer answers, the first option shown, or their own style - so check them against human labels before trusting them.

Try it

Up to 15 XP

How would you grade it?

Pick the most appropriate grading method for each task.

0 of 5 sortedScore 0/0

““What year was the Eiffel Tower completed?””
““Write a function that parses ISO dates””
““Draft a polite reply to an angry customer””
““Convert this SQL query to pandas””
““Classify this ticket as billing, bug or feature request””

pass_at_k.py

from math import comb
n, correct, k = 10, 3, 1
print(1 - comb(n - correct, k) / comb(n, k))

Output

0.30000000000000004

Key takeaways

Build task-specific evals and rerun them on every change.
Exact match for short answers, execution for code, rubrics (human or LLM judge) for open text.
Watch for benchmark contamination and LLM-judge biases.

Lesson quiz

6 questions · pass with 5 correct · up to 50 XP

Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.

Practice: write Python

Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.

Exercise 1

Normalized exact match

+25 XP

Each line is prediction|reference. Normalize both: lowercase, remove punctuation, remove the articles a/an/the, collapse spaces. Print match or miss per line, then exact match: M/N (P%) (whole percent).

Four answers

main.py

Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Exercise 2

Compute pass@k

+25 XP

Each line is n c k: n samples were generated, c passed the tests. Print the unbiased $\text{pass@}k = 1 - \binom{n-c}{k} \big/ \binom{n}{k}$ with 3 decimals: pass@k = X.

Three cases

main.py

Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Questions about this lesson

Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.

Loading posts…