Your plugin passed every test. It was never running.
This week an agent tested a pull request of ours and reported a clean pass. Every row green. The pull request changed the startup hook of our Claude Code plugin, the piece that injects a user's library into the session so the agent knows what it can read. The test's last step was to start Claude Code headless inside our test container, prove the plugin loaded, and prove the hook fired. It passed.
On the second pass, a different agent wrote one line in its report: "the evidence I reported proves nothing."
It was right. The container's wrapper loaded our plugin from the repository's source directory, and that directory, by design, holds no hooks. Hooks ship inside our CLI binary and land on disk only when a user runs setup. So every headless test had loaded a plugin with skills and agents and no hook. The plugin was listed as loaded. There were no plugin errors. The agent called our CLI as expected. Every assertion passed, and none of them could have failed, because none of them looked for the hook.
That is the problem I want to talk about, because I think most people shipping agents, plugins, skills and MCP servers have it and do not know.
Where does your thing actually run?
If you build a web app, you know where it runs and you can test it there. If you build a plugin for Claude Code, a skill for Codex, an MCP server for Cursor, or an agent that gets spawned by some other agent, your product runs inside a host you do not control, on a machine you have never seen, under a configuration your user set up. The honest answer to "how do you test it" is usually: I install the branch on my laptop, in my own agent, signed in as myself, and I run a prompt. If it works, it ships.
That is not nothing. It is also not a test of what your user gets. Your laptop has a year of configuration on it. Your agent is signed in as you. The install script on your site has been run by one person, and that person wrote it. And the moment you point your own tools at a preview environment to test something, every other session on that laptop breaks, because the configuration is global and shared. We learned that one the expensive way.
We spent the last few weeks building the thing that answers the question properly, and we got it wrong in enough instructive ways that I think the story is worth more than the design.
The first version tested the door it was shown
Our first agent-driven QA tested the web app. It signed in, clicked through the dashboard, uploaded a document, checked the count. Clean runs, confident reports.
CandleKeep is a web app the way a bank is a lobby. Most of its use is through a CLI that other agents call, a plugin that injects context into a coding session, and an MCP server that a different class of agents talk to. None of those surfaces were touched, because the agent tested what it could see through a browser, and the browser was the only door we had shown it. The fix was not "test more". It was a mechanical rule: derive the surfaces to test from the file paths a pull request touches, and give the CLI and plugin journeys the same weight as the browser one.
The preview that talked to production
We test against a deployed preview of every pull request: its own database, object store, worker, identity provider, running the branch's migrations from clean. That part was right from the start. What we missed is that a preview environment on our platform is a fork of production and inherits its variables. A test persona clicking "Upgrade" posted "new paying customer" to the real team channel. A preview could send real email to real addresses. Test traffic went into real analytics. Three leaks, one pattern, and the fix was one startup hook that runs in test mode and silences every outward channel, as late in each path as possible so a broken email template still fails.
A container that holds the coding agent
The piece almost nobody builds is the runner: one container per pull request, deployed next to the preview, that holds what a user's machine would hold and nothing of ours. It compiles our CLI from the branch. It installs Claude Code and Codex from their public packages. It signs the CLI in as a test persona against the preview's own API, so the identity inside the container is the same one a browser session can open. It loads our plugin from the branch through a wrapper, and at the end of boot it runs Claude Code once, headless, and prints a banner into the deploy log saying what loaded and what fired.
Then you SSH in and run one command:
claude -p "what books do I have" --output-format stream-json --verbose \
--permission-mode acceptEdits --allowedTools "Bash(ck *)"
and you get one JSON event per line, which is the only form an agent can assert on.
The runner lied to us twice before it told the truth. The first time, the wrapper lived in a directory that the boot script put first on the path, and the self-check ran inside that script, where the path applied. A command arriving over SSH runs in a non-interactive shell that inherits the service's own path, where the raw binary comes first. The agent got the raw binary and no plugin. The self-check got the wrapper and a clean banner. Both were telling the truth about what they saw. The fix was to replace the binary at its own location and to run the self-check with the environment cleared and the path set to what SSH gives, so it sees what an agent sees.
The second time is the story at the top. The fix was to stage the plugin source together with the hook files from the branch into one directory, point the loader there, and print two more banner lines: whether the hook files are present, and how many times the hook's proof string appeared in the transcript. Exactly one. Zero is the old failure. Two means a cached copy fired as well.
A check that cannot fail is not a check
The rule we took from all of this is short. Every assertion names the literal output that would be absent if the thing under test were broken. Before you trust a check, break the feature on purpose and confirm the check fails. If it is still green, you have learned something about the check.
We now run a third agent whose only job is to refute the first one. It reads the evidence comment, reproduces every count itself, and lists every assertion that would also pass if the feature were broken. On the first two runs with a trustworthy hook check, it found three: a PASS awarded to a retest after an in-run failure, a checklist premise that was stale in the deployed build, and an absence claim ("no overflow file was written") with no presence claim next to it to give it meaning. All three are now rules.
Every machine a user has
The last piece was the install. Our install page tells a stranger to paste one line into a terminal. On a Mac with an Apple chip, an Intel Linux box and a Windows laptop with PowerShell, that is three scripts, three archive formats, three places for a binary to live, and three sets of coding-agent directories to detect. We wrote a sandbox script for each shell family that does what a new user does, in their order: temporary home directory, the real served install script, the production binary, sign in as a persona, run setup, then assert a fixed list including "the real home directory is byte-identical to before". A CI matrix runs it on three hosted operating systems on demand and posts one comment on the pull request.
Its first real run passed on two operating systems and failed silently on the third, because the harness had thrown away the install log in the one mode the matrix uses. Which is its own lesson: the log of the thing under test is evidence, and evidence leaves with the sandbox unless the harness keeps it.
What we have now
Every pull request to CandleKeep gets a preview, a checklist an agent wrote from the diff, and an agent that becomes a real user, seeds the state it needs, drives the browser, the CLI, the plugin inside Claude Code, the MCP server and the background worker, posts one evidence comment, and files one ticket per failure. A second agent tries to refute it. The runner proves at boot that the plugin's hook fired, with a count. The install is tested on three operating systems as a stranger. And the whole method lives in one document that the agents read before every run and that we republish when the code moves.
None of that is specific to us. The routes, the personas, the runner, the scripts and the rules are the same shape for any product, and the Part II problem, testing something that runs inside someone else's agent, is one I think a lot of people are about to have.
The book
I wrote it down as a book, because a blog post is the wrong length for "here is how to build it". It is called Agentic QA, and it is on the CandleKeep marketplace, which means your coding agent can read it while it builds your version. Part I is the general method: environment, identity, state, surfaces, side effects, journeys. Part II is the rare part: the runner, the hook proof, fresh installs on every OS, and evaluating whether a plugin behaves rather than whether it loads. Part III is how it sticks. Every chapter says how we built it, with the routes and scripts and their interfaces, and ends with rules in a shape an agent can execute.
If you read one chapter, read chapter 9. It is the story at the top of this post, with the three shell lines that make it impossible to repeat.