0x50Lesson 6 of 8

Evaluate the whole loop

Grade whole runs, not single replies: outcomes, step counts, forbidden actions - and reliability across repeated trials.

22 min 6-question quiz 2 code exercises

By the end of this lesson you can

Grade full traces against outcome, budget and safety criteria
Measure reliability across repeated runs with pass^k
Locate which stage of the loop caused a failure

A loop can produce a perfect final message after deleting the wrong files. So evaluate the whole run: did it reach the goal (checked in the environment), within budget, using only allowed actions, with a sensible stop reason? Save traces - every decision, call, result and stop reason, with secrets redacted - so you can see where a failure started: planning, tool choice, arguments, reading results, or stopping.

Agents are stochastic, so run each task several times. pass@k asks “did at least one of k tries succeed?” - fine when a person picks the best attempt. For an agent that must work every time, use pass^k ( $\text{pass}^k$ ): the chance that all $k$ tries succeed (τ-bench, Yao et al., 2024). A 70% task success rate looks decent until you see that three runs in a row all succeed only about 29% of the time.

Try it

Up to 15 XP

Which stage failed?

Each trace excerpt shows a failure. Where in the loop did it start?

0 of 5 sortedScore 0/0

“Called search_orders(customer="Ana") instead of search_orders(customer_id="C-118")”
“Tests printed “3 failed”, and the agent replied “All tests pass!””
“Ran 60 steps retrying the same failing command”
“get_logs() returned 2 MB of text and the agent lost track of the task”
“Declared the task done after editing code but before running the tests”

pass_hat_k.py

success_rate = 0.7
for trials in [1, 3, 8]:
    print(trials, round(success_rate ** trials, 3))

Output

1 0.7
3 0.343
8 0.058

Key takeaways

Grade whole runs: verified outcome, budget, allowed actions, stop reason.
Keep redacted traces to find which stage failed.
Run tasks repeatedly; pass^k measures the reliability users actually experience.

Lesson quiz

6 questions · pass with 5 correct · up to 50 XP

Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.

Practice: write Python

Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.

Exercise 1

Grade agent traces

+25 XP

Each line is a JSON trace summary {"id", "goal_met", "steps", "tools"}. A run passes if the goal was met, it used at most 8 steps, and it used none of delete_repo or force_push. Print ID: pass or ID: fail (REASONS) with reasons in that order - goal not met, S steps > 8, used forbidden tool T (each forbidden tool used, sorted) - then pass rate: P/N.

Four runs

main.py

Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Exercise 2

pass@k versus pass^k

+25 XP

Each line is n c k: c of n runs succeeded. Print pass@k=A pass^k=B with 3 decimals, where $\text{pass@}k = 1 - \binom{n-c}{k} \big/ \binom{n}{k}$ and $\text{pass}^k = \binom{c}{k} \big/ \binom{n}{k}$ .

Three cases

main.py

Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Questions about this lesson

Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.

Loading posts…