Flaky tests (hands-on): measure them, reproduce them, fix the cause, and the one that was a real bug

A flaky test passes and fails on the same code. Nothing changed, you ran it again, and this time it was green.

One flaky test is an annoyance. Twenty of them change how a team behaves. People stop reading red builds, because "it's probably flaky". They press re-run until it's green. And then a real failure arrives and gets re-run away like all the others.

This series has already met two flaky tests by accident:

• In Part 12, a Cypress test passed 2 runs in 8 on a page with a real race bug.
• In Part 14, a test failed every night from 21:46 to 07:59 because it read the real clock.

This post takes flakiness on deliberately. We wrote a small FoodNow module with five tests and four classic causes of flakiness, measured how often each test fails over 100 runs, reproduced the failures, tried the usual shortcut (retries), and fixed each cause. Every number below is from a real run.

The code and its tests

foodnow/basket.py:

"""FoodNow basket and pricing."""
import random
import threading
import time
from datetime import datetime

PRICES = {"Paneer wrap": 180, "Masala chai": 40, "Veg biryani": 260}

_basket: list[str] = []  # one basket for the whole process


def add(dish: str) -> None:
    _basket.append(dish)


def items() -> list[str]:
    return list(_basket)


def total() -> int:
    """Food total in rupees. Orders of 10 or more dishes get 10% off."""
    food = sum(PRICES[d] for d in _basket)
    if len(_basket) > 10:
        food = food * 90 // 100
    return food


def is_kitchen_open(now: datetime | None = None) -> bool:
    now = now or datetime.now()
    return 11 <= now.hour < 23


class EtaService:
    """Works out the delivery time in the background, like a call to a maps service."""

    def __init__(self):
        self.minutes: int | None = None

    def start(self) -> None:
        def work():
            time.sleep(random.uniform(0.05, 0.15))  # the maps service takes 50-150 ms
            self.minutes = 25
        threading.Thread(target=work, daemon=True).start()

And tests/test_basket.py. Each test looks reasonable on its own:

import random
import time

from foodnow import basket


def test_new_basket_is_empty():
    assert basket.items() == []
    assert basket.total() == 0


def test_add_a_wrap():
    basket.add("Paneer wrap")
    assert basket.total() == 180


def test_kitchen_is_open_now():
    assert basket.is_kitchen_open()


def test_delivery_estimate():
    eta = basket.EtaService()
    eta.start()
    time.sleep(0.1)  # give the maps service time to answer
    assert eta.minutes == 25


def test_big_orders_get_ten_percent_off():
    # Any order of 10 to 20 wraps gets 10% off.
    n = random.randint(10, 20)
    basket._basket.clear()
    for _ in range(n):
        basket.add("Paneer wrap")
    assert basket.total() == 180 * n * 90 // 100

Install pytest and three plugins that help with flaky tests:

pip install pytest==9.2.0 pytest-randomly==5.1.0 pytest-repeat==0.9.4 pytest-rerunfailures==16.7

• pytest-randomly runs the tests in a different random order every time, and prints the seed so you can repeat an order.
• pytest-repeat runs a test many times (--count=50).
• pytest-rerunfailures re-runs failed tests (--reruns 2). We use it in Step 4 to show why it isn't the answer.

Step 1: Run it once

In the order the tests are written, with random ordering turned off:

$ pytest -p no:randomly -q
FAILED tests/test_basket.py::test_delivery_estimate - assert None == 25
1 failed, 4 passed in 0.19s

One failure, which looks like one problem. A single run tells you almost nothing about flakiness. You have to run the suite many times, and in different orders, and count.

Step 2: Measure it

This script runs the whole suite 100 times, each time in a new random order, and counts how often each test fails. Save it as flake_rate.py:

"""Run the test suite many times, each in a new random order, and report how often each test fails."""
import collections
import subprocess
import sys
import tempfile
import xml.etree.ElementTree as ET
from pathlib import Path

runs = int(sys.argv[1]) if len(sys.argv) > 1 else 100
extra = sys.argv[2:]
fails, seen = collections.Counter(), collections.Counter()
with tempfile.TemporaryDirectory() as tmp:
    for i in range(runs):
        report = Path(tmp) / f"run{i}.xml"
        subprocess.run([sys.executable, "-m", "pytest", "-q", "-p", "randomly", f"--junitxml={report}", *extra],
                       capture_output=True)
        for tc in ET.parse(report).iter("testcase"):
            name = tc.get("name")
            seen[name] += 1
            if tc.find("failure") is not None or tc.find("error") is not None:
                fails[name] += 1

print(f"{runs} runs, random order each time\n")
print(f"{'test':<40}{'failed':>8}  rate")
for name in sorted(seen):
    print(f"{name:<40}{fails[name]:>8}  {fails[name] / seen[name]:5.0%}")

$ python flake_rate.py 100
100 runs, random order each time

test                                      failed  rate
test_add_a_wrap                               55    55%
test_big_orders_get_ten_percent_off           10    10%
test_delivery_estimate                        42    42%
test_kitchen_is_open_now                       0     0%
test_new_basket_is_empty                      59    59%

It took 73 seconds. Four of the five tests fail some of the time, for three different reasons:

• test_new_basket_is_empty (59%) and test_add_a_wrap (55%) share one basket for the whole test run. Whichever test runs before them leaves dishes in it.
• test_delivery_estimate (42%) sleeps for 0.1 seconds and hopes the answer has arrived. The answer takes 50 to 150 milliseconds.
• test_big_orders_get_ten_percent_off (10%) picks a random basket size each run. We'll come back to this one, because it's not a test problem.

test_kitchen_is_open_now passed all 100 times. We ran this at 11:28 in the morning, India time.

Step 3: Run it somewhere else

Most CI machines run on UTC. At the same moment, in UTC, it was 06:00:

$ TZ=UTC python flake_rate.py 20
20 runs, random order each time

test                                      failed  rate
test_add_a_wrap                               14    70%
test_big_orders_get_ten_percent_off            1     5%
test_delivery_estimate                         5    25%
test_kitchen_is_open_now                      20   100%
test_new_basket_is_empty                      13    65%

The kitchen test failed every time. On a developer's laptop in India it passes during the day. On the CI server it fails until 11:00 UTC, which is 16:30 in India. Depending on when people look, it's "always green", "always red" or "flaky". It's the same problem as the Part 14 test: it depends on the real clock.

Step 4: Reproduce a failure

A flaky failure you can't repeat is hard to fix. pytest-randomly prints the seed it used for the order at the start of each run:

$ pytest -p randomly
Using --randomly-seed=2365621843
...
FAILED tests/test_basket.py::test_delivery_estimate - assert None == 25
FAILED tests/test_basket.py::test_new_basket_is_empty - AssertionError: asser...
FAILED tests/test_basket.py::test_add_a_wrap - assert 2430 == 180
========================= 3 failed, 2 passed in 0.17s ==========================

Run it again with that seed, and you get the same order and the same failures:

$ pytest -p randomly --randomly-seed=2365621843 -v
Using --randomly-seed=2365621843
tests/test_basket.py::test_big_orders_get_ten_percent_off PASSED         [ 20%]
tests/test_basket.py::test_delivery_estimate FAILED                      [ 40%]
tests/test_basket.py::test_new_basket_is_empty FAILED                    [ 60%]
tests/test_basket.py::test_kitchen_is_open_now PASSED                    [ 80%]
tests/test_basket.py::test_add_a_wrap FAILED                             [100%]
...
========================= 3 failed, 2 passed in 0.17s ==========================

Now the cause is visible: the big-orders test ran first and left 15 wraps in the shared basket, so "add a wrap" saw ₹2,430 instead of ₹180. pytest-randomly also resets Python's random module with the same seed before each test, so even the delivery-time delay repeated exactly.

Step 5: The tempting shortcut: retries

The usual reaction to flaky tests is to retry failures automatically. We tried it, 20 times, with --reruns 2:

$ pytest -p randomly --reruns 2 -q
2 failed, 3 passed, 4 rerun in 0.41s
1 failed, 4 passed, 2 rerun in 0.23s
1 failed, 4 passed, 3 rerun in 0.36s
...
green runs with reruns: 1 / 20

Retries made 1 run in 20 green. A rerun happens in the same process, so the basket is still full from the earlier test, and the retry fails the same way.

Where retries do turn a suite green, they make things worse in a different way: the failures are still happening, but nobody sees them. The test still fails some of the time, and when it's a real bug, it's now hidden. If you use retries at all, record every retried failure and count them, so flakiness stays visible.

Step 6: Fix each cause

Shared state. Each test gets its own basket. That meant changing the code, not just the tests: a module-level basket that lives for the whole process was the real problem. The basket becomes a class, and a fixture gives every test a new one:

class Basket:
    def __init__(self):
        self._items: list[str] = []

    def add(self, dish: str) -> None:
        self._items.append(dish)

    def items(self) -> list[str]:
        return list(self._items)

    def total(self) -> int:
        """Food total in rupees. Orders of 10 or more dishes get 10% off."""
        food = sum(PRICES[d] for d in self._items)
        if len(self._items) > 10:
            food = food * 90 // 100
        return food

@pytest.fixture
def basket():
    return Basket()  # a new, empty basket for every test

Timing. Don't sleep and hope. Wait for the thing you need, with a time limit. The service sets an event when the answer arrives:

class EtaService:
    def __init__(self):
        self.minutes: int | None = None
        self.ready = threading.Event()

    def start(self) -> None:
        def work():
            time.sleep(random.uniform(0.05, 0.15))  # the maps service takes 50-150 ms
            self.minutes = 25
            self.ready.set()
        threading.Thread(target=work, daemon=True).start()

def test_delivery_estimate():
    eta = EtaService()
    eta.start()
    assert eta.ready.wait(timeout=2), "the maps service didn't answer within 2 seconds"
    assert eta.minutes == 25

The test now continues the moment the answer arrives, after 50 to 150 ms, which on average is no slower than the old sleep. It only fails if the service really takes more than 2 seconds.

The clock. Pass the time in. The function already accepted now. The test just didn't use it:

@pytest.mark.parametrize("hour, is_open", [(10, False), (11, True), (22, True), (23, False)])
def test_kitchen_hours(hour, is_open):
    assert is_kitchen_open(datetime(2026, 11, 12, hour, 30)) is is_open

That's also a better test: it checks both sides of opening and closing time, instead of whatever time it happens to be.

Randomness. Replace the random basket size with the sizes that matter: either side of the discount boundary, and a big one:

@pytest.mark.parametrize("n", [9, 10, 11, 20])
def test_big_order_discount(basket, n):
    for _ in range(n):
        basket.add("Paneer wrap")
    expected = 180 * n * 90 // 100 if n >= 10 else 180 * n
    assert basket.total() == expected

Step 7: The flaky test that was a real bug

With the test fixes in place, the 100-run measurement on UTC:

$ TZ=UTC python flake_rate.py 100
test                                      failed  rate
test_add_a_wrap                                0     0%
test_big_order_discount[10]                  100   100%
test_big_order_discount[11]                    0     0%
test_big_order_discount[20]                    0     0%
test_big_order_discount[9]                     0     0%
test_delivery_estimate                         0     0%
test_kitchen_hours[10-False]                   0     0%
...

FAILED tests/test_basket.py::test_big_order_discount[10] - assert 1800 == 1620

The 10% "flaky" test wasn't flaky at all. The code says if len(self._items) > 10, but the rule, in its own comment, is "10 or more dishes get 10% off". An order of exactly 10 dishes was charged ₹1,800 instead of ₹1,620. The random basket size picked exactly 10 about one time in eleven, so the bug showed up as a test that "sometimes fails".

This is why flaky tests deserve attention instead of re-runs: some of them are real bugs that only happen sometimes. Race conditions, time zones, boundary values and rare data all look like flakiness from the outside.

The fix is one character:

-        if len(self._items) > 10:
+        if len(self._items) >= 10:

$ TZ=UTC python flake_rate.py 100
test                                      failed  rate
test_add_a_wrap                                0     0%
test_big_order_discount[10]                    0     0%
test_big_order_discount[11]                    0     0%
test_big_order_discount[20]                    0     0%
test_big_order_discount[9]                     0     0%
test_delivery_estimate                         0     0%
test_kitchen_hours[10-False]                   0     0%
test_kitchen_hours[11-True]                    0     0%
test_kitchen_hours[22-True]                    0     0%
test_kitchen_hours[23-False]                   0     0%
test_new_basket_is_empty                       0     0%

And 30 more runs on India time: 0 failures. Eleven tests, 130 runs, no failures.

Common causes, and their fixes

Cause	What it looks like	Fix
Shared state	Passes alone, fails in the full suite, or in a different order	Fresh data per test; no module-level state; run in random order to catch it
Timing	sleep() followed by an assertion	Wait for a condition or an event, with a time limit
The real clock	Fails at night, at month end, or on a CI server in another time zone	Pass the time in, or freeze it in the test
Unseeded randomness	Fails "sometimes", with different data each time	Test the boundaries explicitly; if you use random data, log the seed
External services	Fails when a sandbox or network is slow	Fake it in most tests (Part 5), check the contract (Part 7)
Asynchronous UI	Reads the page before it has updated	Retrying assertions or explicit waits (Parts 11 and 12)
Parallel runs	Fails only with -n auto or several CI jobs	Unique names for files, ports, database rows and users per test
A real bug	Any of the above	Find out before you fix the test

A team policy that works

1. Measure. Track each test's pass rate across runs, not just whether the latest build was green.
2. Quarantine, with an owner and a date. Move a known flaky test out of the blocking suite, keep running it, and give it an owner and a deadline. A quarantine without a deadline is a deletion.
3. Find the cause before fixing the test. Run it with the failing seed, at the failing time, in the failing environment. One in four of our flaky tests was a real bug.
4. Don't let retries hide failures. If you retry, report every retried failure.
5. Run in random order in CI, so shared state is found when it's added, not months later.

Common mistakes

• Re-running until green. It hides real bugs, and in our run it didn't even work: 1 green run in 20.
• Raising sleeps. sleep(0.1) becomes sleep(1). The test gets slower and is still flaky on a bad day.
• Testing with the real clock. The test passes where it was written and fails at night or on UTC.
• Random test data without a seed. You can't reproduce the failure, and you may be hiding a boundary bug.
• Deleting the test. The flakiness might be the only sign of a real race or boundary bug.
• Running in one fixed order. Shared state stays hidden until someone adds a test in the wrong place.

What's next

In Part 17 of Testing in Practice we look at property-based testing with Hypothesis: instead of choosing examples, describe what must always be true, and let the tool search for the inputs that break it. In a quick check on the buggy code, it found the 10-dish bug in all five runs, in under a second each.

ProvaLab's test management system keeps the history of every test across runs, so a test that passes, fails and passes again on the same code shows up as a pattern, not as a red build someone re-ran. Results arrive from pytest, Cypress, Playwright and other frameworks through ProvaLab's plugins, and you can re-run just the failed tests of a run from TMS, with the re-run recorded as a run of its own. See it in the two-minute product tour.