Skip to content
Just IoT
Menu
← Research
21 November 2026 · Assured AI Engineering

Red before green: proving AI-written fixes before they ship

An AI agent can write a fix and a passing test in a minute. The test only means something if it fails without the fix. How we make that a rule, and the traps that fake it.

AI coding agents are good at producing a fix and a test that passes. That is the problem: a test that passes with the fix may also pass without it, in which case it proves nothing. With AI writing most of the code, “the tests are green” stops being reassuring on its own.

So the first evidence gate in our method is simple to state:

A test written for a bug fix must be shown to fail on the code before the fix.

How we prove it

  1. Commit the fix and its test.
  2. Put back only the source files the fix changed, as they were before the fix, leaving the new test in place.
  3. Run the test. It must fail, and fail for the reason the bug describes.
  4. Restore the fix, confirm the working tree is clean, run again: green.
  5. Record both results with the change.

The traps that fake it

Stashing after committing. Stashing uncommitted work to “remove the fix” does nothing once the fix is committed. The test runs against fixed code, passes, and the record shows a convincing “red” step that never happened. We revert the specific files from the parent commit instead.

Testing a model instead of the code. For ordering, lifecycle and concurrency bugs, a hand-written model of the component tests the author’s understanding, not the code. At least one test must import the real component and mock only at its boundary.

Trusting the step log. On a device, a test runner records what it sent, not what the app did. One of our UI tests reported 30 of 30 steps passed while the picker it was driving never opened. A device test is not a pass until its screenshots have been read.

Not looking at the whole screen. A form test failed for a session because debug buttons from an earlier investigation were sitting on top of the submit button. Nobody had looked at the full screenshot. Now the first thing a failing device test gets is a screenshot, read end to end.

Why it matters more with AI

A person who writes a test usually remembers the bug they were chasing. An agent produces plausible tests at speed, and a plausible test that cannot fail is the most dangerous kind: it turns a guess into apparent evidence. Requiring red before green costs a few minutes per fix. It also catches the repairs that quietly reintroduce the defect they set out to fix, which our own records show is not rare.

Every rule like this one is written down with the incident that caused it. That record is what lets the method improve without anyone having to remember why.