Why a green check is not proof: the gates between an AI agent and production
On October 6, 2026, I merged a navigation redesign into main. The pull request showed eight checks, eight green. The next pull request touched a different directory, the browser tests ran for the first time against the new navigation, and twelve of them went red. One was a product failure: on desktop, there was no longer any way to add a car.
The e2e job had not run on the redesign. Its path filter only started it for four directories, and the redesign touched none of them. A skipped job reports as success. From the outside, a check that did nothing and a check that passed are identical. That is the sentence I keep coming back to, because it is the core problem of letting agents write code quickly: the volume of change goes up tenfold, and your ability to read it does not.
The gates, as they stand now
Every pull request on Acervo passes five required checks before main will accept it: lint with seven house rules, strict types, 4,300 unit and integration tests plus 5,900 database assertions, a check that the auth email templates are intact, and a secrets scan on every commit of the branch. A sixth, 590 browser tests, runs on every change that is not documentation. Since October 6 the filter is inverted: instead of a list of what triggers the browser tests, a list of what may skip them.
Then the release gate. Before any promotion to production, a security agent runs a full penetration test against local and staging, never production, and writes a report with a verdict. A FAIL blocks the promotion like a red check. On October 15 it said FAIL on a rule outside the feature being promoted; the feature shipped behind a flag and the rule was fixed first. Promotion itself is one human command, logged with who and when.
Rules I had to write down
- A skipped check does not count as green. Before merging anything with a screen, open the e2e run and read it: which steps executed, how many tests passed.
- Nothing is declared, everything is produced. No agent writes "the deployment works". A script produces JSON with the deployment id and status, and the agent interprets it. "CI green" and "the preview" are not phrases; a run id and a URL are.
- A test that documents a bug protects the bug. If writing a test reveals something broken, it becomes a ticket and the test is marked pending, never green. A green test on a known failure makes the fix look like a regression.
- Break a policy on purpose. On any change to database security, the reviewer deliberately breaks one isolation rule and confirms the suite goes red. If it stays green, the tests do not test what they claim.
- Read the annotations before the code. Seven red checks in four seconds are not seven failures. One morning they were a billing limit on the CI account.
Why this matters more with agents
None of these rules are new to engineering. What changed is that the agents are tireless and literal. They will produce a passing suite that proves nothing if that is what the rules permit, and they will do it 445 times. The only defence is to be precise about what counts as proof, and to make the human gates cheap enough that they are actually used. A gate nobody has time to open is not a gate.
Nothing here required a better model. It required a rule about what counts as proof.
Part of a series: AI agent teams for software development and the two human gates. The failure story, beat by beat, with the run ids, is in the Trust chapter of the Acervo case. If your team is introducing agents and wants the guardrails with them: how I set that up.
Copy the checklist: Release gate checklist, with a prompt to use it and its changelog.
Working on something like this?
This is the method behind how I build: AI native development with human gates. Discuss a product idea → Or email me.