Trustbut brings baseline-test validation to AI agent development
A new GitHub tool helps developers confirm that agent test failures stem from code changes, not inherent flakiness.
Last verified:
Baseline Testing for Nondeterministic Agents
Test flakiness is endemic in AI agent development. Unlike traditional software, agent behavior can vary across runs due to sampling, randomness in planning, or stochasticity in the environment—even when the code is unchanged. This makes it difficult to determine whether a test failure stems from a code change or from pre-existing instability. According to the Trustbut GitHub repository, the tool addresses this by running tests twice: once against the proposed change and once against the original codebase, isolating whether failures are attributable to the change itself.
The core insight is simple but operationally powerful: a test that fails both with and without a change indicates baseline flakiness, not a regression. A test that fails only with the change points to a real defect. This distinction is essential for CI/CD pipelines where false negatives (blocking valid commits due to flaky tests) and false positives (missing real regressions due to noise) both carry significant costs.
How Trustbut Works
Trustbut operates as a validation wrapper around existing test suites. According to the repository, the tool accepts a test command and a change specification, then executes the tests twice—once with the change applied and once with it reverted—comparing the results to determine causation. The exact workflow and supported test runners are documented in the project’s README.
The output distinguishes three failure modes: failures present only in the changed code (true regressions), failures present in both runs (baseline flakiness), and tests that pass consistently (stable coverage). This classification helps developers decide whether to investigate a failure as a bug or to mark the test as a known-flaky baseline.
Why This Matters
For teams integrating AI agents into continuous integration pipelines, test flakiness undermines confidence in automation. Trustbut changes the debugging workflow by making baseline instability explicit—developers can distinguish between “this change broke the test” and “this test is unreliable,” reducing false alerts and enabling more aggressive testing strategies. Teams already using agent-testing frameworks may find Trustbut valuable as a pre-commit or CI-stage guard to reduce noise in test reports.
The tool’s immediate value lies in operational clarity: fewer wasted debugging cycles on non-regressions, fewer blocked commits due to flaky tests. Longer-term, explicit baseline flakiness measurement could drive investment in test stabilization or inform decisions about which tests warrant rewriting versus which are acceptable as probabilistic checks.
Frequently Asked Questions
What problem does Trustbut solve?
AI agent tests often fail intermittently due to non-deterministic behavior unrelated to code changes. Trustbut runs tests with and without a change to determine whether failures are caused by the change itself or by pre-existing flakiness.
Is Trustbut specific to a particular agent framework?
According to the GitHub repository, Trustbut is designed as a general-purpose validation tool, but the README does not specify which agent frameworks or test runners it currently supports.
How does Trustbut differ from standard regression testing?
Standard regression testing assumes tests are stable. Trustbut explicitly accounts for test instability by running the same test suite against an unchanged codebase, making baseline flakiness visible before evaluating the impact of a change.