Agent Reliability Toolkit

Context

AI agents are moving from experiments into work that changes files, calls services, updates records, and makes recommendations people act on. That changes the standard. A convincing demo is useful, but it does not answer what happens when context is incomplete, tools fail, instructions conflict, or an agent is asked to operate near a boundary.

The Agent Reliability Toolkit is my public work on that gap. It treats reliability as a system of scenarios, evaluation, evidence, and release decisions rather than a claim attached to a model.

The operating problem

Agent teams often evaluate the thing they just built with the same assumptions used to build it. Happy paths get polished. Known edge cases get patched. The result can look increasingly capable while entire classes of failure remain untested.

The harder questions are behavioral:

Does the agent stop when authority is missing?
Does it preserve the right evidence?
Can it recover from a partial tool failure without duplicating an action?
Does it know when human review is required?
Can another evaluator reproduce what happened?

Those questions require more than a prompt and a pass rate.

My responsibility

I created and maintain the toolkit, its public documentation, and the companion Agent Reliability Certification Specification, or ARCS. My work spans the evaluation model, developer experience, scenario execution, scoring, reporting, and the language used to communicate reliability.

The project is deliberately open. Reliability improves when teams can inspect the rules, challenge the evaluator, and reproduce the evidence.

What I built

The toolkit provides production-oriented Python, pytest, and FastAPI building blocks for running adversarial scenarios and turning results into structured reports.

The shape is simple on purpose:

Define the behavior and boundary being tested.
Run the agent in a controlled scenario.
Capture outputs and relevant evidence.
Evaluate against explicit criteria.
Produce a report another person can inspect.

ARCS adds a shared vocabulary for reliability evaluation and maturity. It is not a guarantee that an agent can never fail. It is a way to state what was evaluated, under which rules, and with what evidence.

What I automated

Repeatability matters more than a clever one-off test. The toolkit structures the path from scenario execution through scoring and reporting so the same evaluation can be run again after a model, tool, policy, or implementation changes.

Automation handles the mechanical work:

Scenario setup and execution
Evidence collection
Structured scoring
Report generation
Repeatable audit runs after model, tool, policy, or implementation changes

The release decision remains understandable to a person. Automation gathers and organizes evidence. It does not hide the judgment.

How I hardened it

The central rule is separation:

Field note
Builder is not tester.

The evaluator should not simply restate the builder's assumptions. Scenarios need adversarial pressure, explicit failure conditions, and evidence that survives beyond a terminal window.

The toolkit also treats its own output as something that must be inspectable. A score without the scenario, criteria, and evidence is weak. A report that cannot be reproduced is weaker than it looks.

Outcome

The public toolkit, documentation, and ARCS specification give teams a concrete way to discuss agent reliability. Instead of asking whether an agent is "production ready" in the abstract, a team can ask which behaviors were tested, which boundaries were exercised, what evidence exists, and what remains unknown.

That shift is the real outcome: reliability becomes work that can be designed, reviewed, repeated, and improved.

Working principles

Test behavior, not confidence.
Make failure conditions explicit.
Keep consequential judgment visible.
Preserve enough evidence for independent review.
Re-run evaluations when the surrounding system changes.
Treat recovery as part of reliability.

Finding agent failures before production.