The Flaky Test Is Trying to Tell You Something
Every team has one. The test that fails on Tuesday, passes on rerun, and fails again next week on a different branch. Eventually someone adds a retry, or tags it @flaky and moves it to a quarantine suite nobody looks at. The pipeline turns green, and everyone agrees the problem is solved.
It is not solved. It is muted. A flaky test is a test that sometimes sees a different world than the one you think you built. That difference is information, and rerunning until it disappears is the same as pulling the battery out of a smoke alarm because it keeps beeping.
Flakiness Has Causes, Not Moods
"Random" is not a root cause. When I go through flaky tests with a team, almost every one lands in a small number of buckets:
- Timing. The test assumes something has finished: a request, an animation, a database write. Usually it has. Sometimes it has not. Fixed sleeps hide this until the machine is slower than the one the test was written on.
- Shared state. Two tests touch the same user, the same file, the same row. Run alone, each passes. Run in parallel or in a different order, they collide.
- Environment. A third-party sandbox that throttles, a clock that crosses midnight, a time zone, a locale, a container with less memory than a laptop.
- The product itself. A real race condition, a missing lock, an eventually consistent read treated as immediate. Here the test is not flaky. The system is, and the test is the only witness.
The last bucket matters most, and it is exactly the one a retry hides best.
Why Retries Are the Wrong Default
A retry asks a simple question: can this test pass at all? The question it should ask is why it failed this time. When you rerun until green, you keep the lucky execution and throw away the unlucky one. That unlucky one is the run your users will eventually get in production: under load, on a bad network, on a Friday evening.
Retries also teach the team something corrosive: that a red build might not mean anything. Once people believe that, they stop reading failures. The suite keeps running, but nobody trusts it anymore, and an untrusted suite is just a slow way to spend CI minutes.
Reading the Signal
Treating flakiness as a signal means doing a little engineering instead of a little hoping:
- Record every failure with its context. Commit, runner, duration, execution order, timestamps. A flaky test with no history is a rumor. With history, it becomes a pattern.
- Reproduce on purpose. Run the test in a loop, in parallel, with a throttled network or a slower CPU. If you can make it fail on demand, you are halfway to the fix.
- Replace waiting with knowing. Wait for a condition, not for a number of seconds. If the product gives you no way to know when it is ready, that is a testability gap worth raising with the developers.
- Isolate data. Each test creates what it needs and cleans up after itself. Shared fixtures are shared failure points.
- Quarantine with a deadline. If you must pull a test out of the main pipeline, give it an owner and a date. A quarantine with no exit is a graveyard.
This Is Where Building Knowledge Pays Off
You cannot diagnose a race condition you do not understand. Knowing how the frontend schedules rendering, how the API handles concurrent writes, and how the queue retries is what turns "it sometimes fails" into "it fails when the cache is cold and two requests arrive within fifty milliseconds." That sentence is the difference between a tester who reruns and an engineer who finds bugs. It is the argument this whole series keeps making: to test something well, you need to know enough to build it.
So the next time a test fails and then passes, resist the reflex to click rerun. Ask what it saw. More often than you would like, it saw something true.