Testing AI agents is fundamentally different from testing traditional software.
Traditional systems follow a simple contract:
input → expected output → assertion
Agents don't.
They reason dynamically, choose tools, compose responses, and operate on data that changes constantly. Hardcoded expected answers quickly become stale and brittle.
To address this, we use a three-layer evaluation strategy that combines deterministic validation, semantic evaluation, and model migration testing.
🚫 No Hardcoded Expected Results
One of the biggest challenges in agent testing is avoiding stale expectations.
Market data changes.
Accounts change.
Reference data evolves.
A hardcoded answer is often outdated the moment it's written.
Instead of storing static expected responses, each test case points to one or more live reference APIs.
Before evaluation runs, the framework fetches current ground-truth data directly from upstream services and evaluates the agent response against that live data.
This shifts the goal from validating snapshots to validating reasoning and communication against current reality.
⚡ Layer 1: Deterministic Validation
The first layer performs fast, reproducible checks against the agent response.
Examples include:
- Required values present
- Expected identifiers returned
- Missing fields detected
- Error or fallback messages detected
- Structured formats validated
A key capability is validating values against live reference data.
Expected values are extracted from the reference payload and normalized into multiple formats before checking whether they appear in the response.
For example:
659161818might be recognized as:
659,161,818
$659.2M
659.2 millionThis layer catches:
- Empty responses
- Missing information
- Incorrect values
- Malformed output
- Tool execution failures
Fast, deterministic, and easy to debug.
🤖 Layer 2: LLM-as-Judge
Deterministic checks verify facts.
They cannot determine whether an answer is complete, coherent, or grounded.
For that, a second LLM acts as a judge.
The judge evaluates:
- Completeness — Did the response answer everything that was asked?
- Coherence — Is the response clear and logically structured?
- Groundedness — Are all claims supported by reference data?
The judge receives:
- The original question
- The agent response
- The live reference data
It reasons step-by-step before assigning scores.
Importantly, the judge does not validate numeric accuracy. That responsibility remains with the deterministic layer.
Both layers use the same reference data but in different ways:
- Layer 1 verifies that required values appear in the response
- Layer 2 verifies that claims made in the response are supported by the reference
This separation prevents overlap and conflicting evaluations.
🔄 Layer 3: Model Migration Testing
Sometimes the question isn't:
Is this answer correct?
It's:
Can I safely switch from one model to another?
For migration testing, each test case runs twice:
- Baseline model
- Candidate model
A judge then compares the two responses and classifies the candidate
This mode answers a fundamentally different question from the first two layers.
The first two layers compare responses against objective ground truth.
The migration layer compares a candidate model against the current production baseline.
🏗️ Evaluation Pipeline
Ground Truth Evaluation
invoke_agent
↓
fetch_live_reference
↓
deterministic_validation
↓
(optional) llm_judgeModel Migration Evaluation
baseline_model
↓
candidate_model
↓
pairwise_judgeThis path focuses solely on migration safety and does not run the ground-truth evaluation layers.
🎯 Why Multiple Layers?
No single evaluation method is sufficient.
Layer 1 provides:
- Speed
- Exactness
- Reproducibility
Layer 2 provides:
- Semantic validation
- Contextual reasoning
- Groundedness checks
Layer 3 provides:
- Safe model migration
- Regression detection
- Comparative evaluation
Together they provide a practical framework for testing AI agents without relying on brittle hardcoded outputs.
As agents become more autonomous and business-critical, having a robust evaluation strategy becomes just as important as the agent itself.Testing AI agents is fundamentally different from testing traditional software.
