Agent Assurance
AI agent · Computer-use / browser agent by TestMu AI
Test what AI agents actually did, not what they said.
Agent Assurance is an evidence-based testing platform for AI agents that take actions, verifying actual changes to files and APIs rather than self-reported logs. It is ideal for enterprise teams needing to prove agent safety and reliability through automated, adversarial test suites derived directly from code.
Agent specs
The full spec card (context, pricing per token, modalities and more) on AntSpace. Sign up free to see the full spec card.
- Price
- From 0 USD
- Languages
- English
Category: Software Testing
- Channels
- API, Terminal (CLI)
- Integrations
- GitHub Actions, Jenkins, Argo CD, MCP servers
- Platforms
- macOS, Linux, Windows x64, WSL
- Deployment
- Cloud, Private Cloud, On-Premise
- CLI Tool
- Rook CLI
- Supported OS
- macOS, Linux, Windows x64, WSL
- Grades actual changes to tools, files, and APIs
- Derives test suites automatically from code or docs
- Adversarial testing for prompt injection by default
- Integrates with GitHub Actions, Jenkins, and Argo CD
Overview
Agent Assurance is TestMu AI's evidence-based testing platform designed for businesses deploying autonomous AI agents that take actions. It solves the problem of unreliable agent self-reporting by grading what an agent actually changed in tools, files, and APIs rather than what it claims to have done. Delivered via the Rook CLI, the platform automatically derives functional and adversarial test suites from repositories, PRDs, or policy documents. It distinguishes itself by reporting unconfirmed actions as "Unable to Verify" instead of a pass, ensuring release decisions are based on hard evidence. The system integrates into CI/CD pipelines like GitHub Actions and Jenkins to provide continuous assurance for coding, workflow, and back-office agents.
Discovery and Planning
- Automated Discovery: The agent reads a codebase, requirements document, or both together to understand the agent's intended behavior.
- Tool Identification: It connects to MCP servers to identify existing tools and their functions.
- Gap Detection: The system identifies undeclared fields and marks them as unknown rather than guessing.
Test Generation
- Scenario Generation Ent-to-End: Agent Assurance derives functional and adversarial scenarios directly from code or specifications without manual authoring.
- Adversarial Testing Suite: The adversarial family covers prompt injection, instruction override, and tool misuse by default.
- Criterion-Based Judgment: Each scenario carries its own gradable criteria, which serve as the unit of judgment.
Execution and Judging
- Live Execution Monitoring: The agent is invoked for real, and the system monitors files, artifacts, and every tool call made.
- Non-Destructive Judging: Judges verify read-only actions and ensure the testing process changes nothing in the target environment.
- Surface Validation: It checks every tool call against the agent's declared tool surface to detect unauthorized actions.
Reporting and Analysis
- Evidence-Backed VerdictsBase: Verdicts are reported as Pass, Fail, or Unable to Verify, with the latter excluded from the pass rate.
- Evidence Quoting: Each verdict quotes the specific evidence it rests on for transparency.
- Failure Clustering: Repeat failures are automatically clustered onto the specific criterion they share.
What it does
- Discover agent capabilities from code and docs
- Generate functional and adversarial test scenarios
- Monitor tool calls, file changes, and artifacts
- Judge agent actions against evidence-backed criteria
- Report verdicts as Pass, Fail, or Unable to Verify
Published results
- 70% faster test execution
- Tripled test volume
- Reduced system health monitoring failures
Use cases
- Engineering teams use Agent Assurance to verify that coding agents correctly modify files and pass builds before merging PRs.
- Product managers point the tool at policy docs to automatically generate and run compliance checks against back-office agents.
- DevOps engineers integrate Rook CLI into GitHub Actions to gate releases based on evidence-backed pass rates.
Specifications
- Channels
- API, Terminal (CLI)
- Integrations
- GitHub Actions, Jenkins, Argo CD, MCP servers
- Platforms
- macOS, Linux, Windows x64, WSL
- Human control
- Autonomous discovery and generation; human approval required for target environment and spend; manual review of unverifiable results
- Deployment
- Cloud, Private Cloud, On-Premise
- Setup
- One command installation via brew or npm; quickstart loop runs in minutes
- CLI Tool
- Rook CLI
- Supported OS
- macOS, Linux, Windows x64, WSL
- Runtime Requirement
- Node 20+ (for npm)
- Verdict Types
- Pass, Fail, Unable to Verify
- Input Formats
- Codebase, PRD, Policy Docs (PDF, Text)
- CI Integrations
- GitHub Actions, Jenkins, Argo CD
- Test Types
- Functional, Regression, Smoke, Automation, Continuous, Adversarial
- Adversarial Scenarios
- Prompt injection, instruction override, tool misuse
How it works
- Discover Agent Context: Point the tool at a repository, PRD, or folder of policy docs to let it read the codebase and requirements.
- Generate Scenarios: The system derives functional and adversarial scenarios, including prompt injection and tool misuse, from the discovered code.
- Run and Judge: The agent is invoked for real while the system watches files, artifacts, and every tool call made.
- Report Verdicts: Review verdicts as Pass, Fail, or Unable to Verify, with each result quoting the evidence it rests on.
Pricing
- 14-Day Free Trial
- Free
FAQ
How does Agent Assurance differ from standard AI evaluation tools?
Agent Assurance grades what an AI agent actually changed in tools, files, and APIs, whereas most eval tools only score what the agent said or recorded in its own logs.
What happens if the system cannot confirm an agent's action?
It reports them as 'Unable to Verify' and excludes them from the pass rate, ensuring that a lack of evidence never results in a quiet pass or a guessed fail.
Do I need to write manual test cases for my agents?
No, the system reads your repository, PRD, or policy documents and derives functional and adversarial scenarios from them automatically.
Added by AntSpace from public sources (testmuai.com).