Spaces

Evals for Agent Pipelines: Scoring Every Handoff in Phoenix

November 4, 2026 · Rayed Bajwa

You can't unit-test whether one agent handed the next everything it needed. You can score it on every real run, and chart it.

A multi-agent pipeline fails in ways a single agent doesn't. Each stage can do its job well and the run still goes wrong, because something got lost between stages.

In Spaces, every stage (specify, plan, tasks, implement, review, verify, deliver) is its own agent. When a stage finishes, its output is summarized and handed to the next one as context. That handoff is the pipeline's memory, and for a while we had no idea how good it was.

Then we started scoring it.


Three things worth scoring

We trace every agent turn to Arize Phoenix. Each run is a Phoenix session. As each stage hands off, the worker scores it and writes the scores onto the session as annotations:

1. handoff-quality: six checks on what the next stage received, scored 0–1:

Check What it catches
complete the stage's output was cut off or empty
compacted a long output went on raw instead of summarized
keeps-status "Verification Status: PARTIAL" didn't survive the summary
keeps-references commit hashes, PR numbers or spec files were dropped
keeps-question the stage stopped on a question, and the handoff doesn't say so
not-over-redacted secret-masking hid more than the stage's output had

2. report-format: whether the verification report and code review are in the form the pipeline parses. The pipeline branches on lines like Code Review Status: APPROVED; a report that drifts from the format is a silent bug.

3. loop-progress: whether each verify moved the count of acceptance criteria met, compared with the previous verify. same means an implement pass changed nothing that mattered.

All three are plain code, not LLM judges. Most of what goes wrong in a pipeline is checkable with a regex.

What a week of real runs showed

We backfilled a week: 303 handoffs from 96 runs.

None of that was visible in logs unless you went looking. All of it was obvious on a chart.

The fixes the scores pointed to

Why annotations, not a dashboard

Phoenix's project Metrics page already charts session annotation scores over time, next to traces, errors, latency, tokens and cost. So we didn't build a dashboard. We wrote scores onto runs and let Phoenix chart them.

That has a nice property: when a score drops, you click through to the exact session, see which stage failed which check, and read the trace.

How to start

  1. Trace every agent turn, grouped by run.
  2. Write down what each handoff must keep. For us: status, references, open questions.
  3. Check it in code on every handoff. Score 0–1, explain failures in plain words.
  4. Write the scores where your traces are, so a bad score is one click from its cause.
  5. Backfill a week before you trust your intuition about where the pipeline is weak. Ours was wrong.

Spaces is an AI software factory: specs go in, reviewed and tested pull requests come out. It's on GitHub, source-available and free for personal use.