Where AI coding agents fail on real products: the eight failures I hit and the gate for each
In seven weeks of building a production product with eight AI coding agents, I kept a table: one line per incident, the date, and the rule it left. It has over sixty lines. Reading them back, every one fits one of eight patterns, and none of the eight is "the agent wrote wrong code". They are all a form of trusting something that looked like proof. Here are the eight, each with the gate that now catches it.
1. The green check that did not run
A navigation redesign merged with eight green checks. The browser tests had not run: their path filter only triggered on four directories, and the change touched none. A skipped job reports as success. The next pull request ran them, and twelve went red, one of them a product failure. This pattern came back five times in different shapes: a test suite exiting zero with a hundred skipped files because the local stack was down, a build that compiled and then returned 500 on every request, a deployment marked ready that served a 404, a loop asserting over zero elements.
Gate: a skipped check does not count as green. Before merging anything with a screen, open the run and read which steps executed and how many tests passed. Filters list what may skip, never what runs. A test that measures elements asserts how many it measured.
2. The fact declared instead of produced
An agent wrote "the deployment works". It had not opened it. Another wrote "CI is green" about a commit that was not the one being merged. Both sentences are grammatical and both were false, and nothing in the sentence distinguishes them from true ones.
Gate: no agent declares an external fact. A script produces it as JSON with an identifier (pull request number, commit, run id, deployment id, URL) and the agent interprets the JSON. "The preview", "the build" and "CI green" are not phrases on my build. A script in FAIL is not turned into PASS by anyone's text.
3. The ticket that contradicted itself
Line 195 of a ticket said a phase was delivered with 42 frames. Line 240 said three of those frames did not exist. I read the second, wrote a false starting condition into a decision, and sent an agent to build something that already existed. The agent, to its credit, checked by two methods before building and found it.
Gate: when a ticket says something does not exist, look before building. And a ticket with a measurable criterion and an explicit scope, validated by schema, has fewer places to contradict itself. The template exists because of this day.
4. The test that protected the bug
A comment on a test said a redirect "returns on localhost, existing behaviour, not introduced by this ticket". It was the bug: the session was lost on every magic link. The green test now asserted the broken behaviour, so fixing it would turn the test red and look like a regression. It sat there for three weeks.
Gate: no test asserts a behaviour as pre existing. A suspected bug becomes a ticket the day it is seen, and the test is marked pending with the reference, never green. The same rule covers code: a known failure without a ticket does not exist, and a comment saying "escalated" is not an escalation.
5. The guard that was retired because another one covered it
A recovery path was removed with the argument that it was strictly weaker than a database trigger. True in an instant, false over time: the trigger ran once at account creation, the recovery ran on every visit. Removing it locked one user out permanently.
Gate: before retiring a redundancy, measure when each copy runs, not only what it checks. Two guards that check the same thing at different moments are not the same guard. The pattern repeated at the design level: two decisions each removed an entry point "because it is in the other place", and together they removed both.
6. The file transplant that deleted finished work
Integrating three branches by copying files from each ("checkout other-branch, this file") silently removed work already in main, twice: the branches had an older base, so their copy of the file was older and the copy won. Separately, a conflict resolution script opened a file for writing before reading it, which truncates it, and two files entered main empty. Six checks passed; an empty script exits zero.
Gate: a file transplant is only valid for files main has not touched since the branch base; otherwise a real three way merge. After resolving conflicts, diff against the base before pushing: a "keep both sides" resolution cannot remove lines. Generated files are regenerated, never copied.
7. The number that was too round to be true
Four sweeps in one day returned a false number: a contrast of 1.39 that was 12.84 measured against the real background, 61 screens without an exit that were 0, 192 instances out of size that were 236, and a grep with an escaped pipe that returned zero of zero of zero. The agent caught all four by checking the instrument before reporting.
Gate: a result that confirms what you already suspected gets checked more, not less. A sweep that returns zero is repeated by another method before it is reported. And every count is stated with its scope: "zero out of size" without "on which page" is the same class of sentence as "CI green".
8. The check that used the same path as the work
A sweep replaced values by searching for a layer name, which is the structure. The verification searched for the rule (a masked value followed by more than four characters) and found twelve the first pass had missed. Had the verification used the same search, it would have been green by construction.
Gate: verify by the criterion, never by the route the work took. A check that repeats the work does not check the work. This is also why the verifier agent never reads the ticket or the report: it only sees claims and a script.
What the eight have in common
None is a model failure. Each is a place where something looked like proof: a green badge, a confident sentence, a comment, a round number, a passing check. Agents multiply the volume of those things by ten, and your capacity to inspect them does not grow. The answer is not to read faster. It is to make proof mechanical: scripts that produce facts with identifiers, tests that count what they measured, filters that list what may skip, and a rules file where each of these has its date.
This is the last section of how to build an MVP with AI agents. The gates in detail are in why a green check is not proof; the rules file where each incident is recorded is in the project rules file.
Questions people ask
Are AI coding agents reliable enough for production?
Yes, with gates; no, without them. None of the eight failures below was a wrong line of code. All were an agent, or a human, trusting something that looked like proof and was not. Reliability comes from the gates, not from the model.
Which failure is the most common?
The green check that did not run. It appears in five different shapes in my lessons table: a skipped job reporting success, a suite exiting zero with a hundred skipped files, a build that compiled and did not start, a loop over zero elements that passed, a deployment marked ready that served a 404.
Do agents make the same mistakes as humans?
Mostly, faster and with more confidence. The difference is volume: ten times the changes, the same number of eyes. Which is why the fix is never reading faster; it is making proof mechanical.
Working on something like this?
This is how I build products with AI agent teams, from scope to production. Discuss a product idea → Or email me.