Debugging from the trace

Your agent ran for eight steps and replied "Done! The report is ready." There is no report. What happened?

The final answer can't tell you. It's the model's summary of what it thinks happened, and the model is exactly the part you can't trust right now. To debug an agent you need the trace: a record of every step, what the model asked for, and what your code sent back.

Log every step

One JSON line per step is enough. Write it as you go, so a crash still leaves you a trace up to the crash.

import json, os

def log_step(run_id, step, reply, results):
    os.makedirs("traces", exist_ok=True)
    with open(f"traces/{run_id}.jsonl", "a") as f:
        f.write(json.dumps({
            "step": step,
            "reply": reply_to_dict(reply),          # the model's full reply, as your SDK serialises it
            "input_tokens": reply.usage.input_tokens,
            "results": [r[:300] for r in results],  # previews are enough to read
        }) + "\n")

reply_to_dict stands for whatever your API client gives you to turn a reply into plain data; most SDKs have one. Call log_step at the end of each loop iteration, after the tools have run, so the line holds both the request and what came back.

Reading a trace

Here is the trace behind that fake "Done!":

You don't read a trace top to bottom like a story. You scan it for a few shapes that come up again and again:

Notice that most fixes land in your tools, not in the prompt. The model behaves sensibly given what it's shown. The trace shows you what it was shown.

Replay instead of rerun

The obvious way to debug is to run the agent again and watch. That's slow, costs money every time, and the model may not make the same mistake twice, so the bug you're chasing disappears and comes back next week.

Instead, replay the trace: feed the recorded replies back in place of the real model.

class ReplayModel:
    """Stands in for call_model, returning recorded replies in order."""

    def __init__(self, trace_path):
        with open(trace_path) as f:
            self.replies = [json.loads(line)["reply"] for line in f]
        self.step = 0

    def __call__(self, messages, tools=None):
        reply = reply_from_dict(self.replies[self.step])
        self.step += 1
        return reply

answer = run_agent(task, tools, ReplayModel("traces/run-42.jsonl"))

This is why run_agent has taken call_model as a parameter since lesson 1: the loop doesn't care where replies come from. The model's decisions are frozen, your code runs for real, and the failure happens at the same step every time. Put a breakpoint there and look around.

Two things to watch:

• Replay runs your tools again. Fine for read-only tools. For tools that change the world (lesson 2's "doing" tools) pass in fakes that only record what they were asked to do.
• Replay reproduces a bug; it doesn't test the fix. Once your fix changes a tool's result, the recorded replies no longer match what the model would say. To test the fix, replay up to the step before things went wrong, then hand over to the real model:

class ReplayThenLive(ReplayModel):
    def __init__(self, trace_path, live_model, until_step):
        super().__init__(trace_path)
        self.live, self.until = live_model, until_step

    def __call__(self, messages, tools=None):
        if self.step < self.until:
            return super().__call__(messages, tools)
        return self.live(messages, tools=tools)

You skip the first steps for free and only pay for the part you're testing.

Keep the traces that taught you something

A trace that reproduces a bug is a ready-made test. Replay it with fake tools and assert on what your code did, not on the model's words:

def test_refund_needs_approval():
    refunded = []
    tools = {"get_order": lambda id: {"price": 250}, "refund_order": lambda id: refunded.append(id)}
    run_agent("Refund order 4411", tools, ReplayModel("traces/refund-no-approval.jsonl"))
    assert refunded == []    # the approval check from lesson 3 must have blocked it

Over time you collect a folder of small, fast, free tests built from real failures.

Recap

• The final answer is a claim. The trace is the evidence.
• Log one JSON line per step, as you go.
• Scan traces for repeats, token jumps and errors followed by "done".
• Replay recorded replies to reproduce a bug for free; replay-then-live to test the fix.

Next: you have every piece. Let's build a complete agent with all of them.