Handling Flaky Tests: Stop Non-Deterministic CI Failures

Previously in this course, we covered monitoring pipeline health to keep our build times predictable. In this lesson, we address the "silent killer" of developer productivity: the flaky test.

A flaky test is a test that returns different results (pass or fail) for the same code state. If your pipeline fails randomly, developers stop trusting the status badge, and eventually, they start ignoring real failures. This lesson shows you how to bring stability back to your test suite.

Identifying the Flaky Test

A flaky test isn't just "bad luck." It is usually the result of non-determinism, such as relying on timing, network latency, or shared state. Before you can fix a test, you must confirm it is truly flaky.

If you have a test that fails once in every ten runs, it is a classic candidate for isolation. Use these signals to detect them:

1. The "Rerun" Trap: If a developer clicks "Re-run" on a failed job and it passes without any code changes, you have a flake.
2. Timing Sensitivity: Tests that use sleep() or rely on external API responses that fluctuate in latency are prime suspects.
3. Shared Resource Contention: If multiple tests write to the same database or file path, they may interfere with each other.

To master the art of narrowing down these issues, I highly recommend reviewing isolating failing code segments to ensure you have a minimal reproduction case.

Implementing Retries

While fixing the root cause is the gold standard, some external dependencies (like flaky third-party APIs) are out of your control. In these cases, implementing a retry strategy is an acceptable stop-gap.

The "Retry" Pattern

Most test runners provide built-in flags for retries. If you are using Pytest, you can install pytest-rerunfailures.

1. Install the plugin: pip install pytest-rerunfailures
2. Execute with retries: pytest --reruns 3

In your GitHub Actions workflow, you can update your test step:

YAML
- name: Run Tests
  run: |
    pytest --reruns 3 --reruns-delay 2

The --reruns-delay 2 flag is critical; it forces the runner to wait 2 seconds before trying again, giving the external system (or the event loop) time to clear the contention.

Stabilizing Your Test Suite

Retries are a crutch, not a cure. To achieve long-term stability, follow these principles:

• Remove Time-based Dependencies: Never use time.sleep(). Instead, use "polling" or "wait-for" patterns that check for a condition every 100ms until a timeout is reached.
• Isolate Test Data: Ensure every test creates its own unique database record or file. If two tests share a user ID, they will eventually collide.
• Mock External Services: If your test suite calls a real API, it will eventually fail when that API is down. Use local mocks or service virtualization to simulate consistent responses.

As you build out these more complex testing scenarios, you might find that you need to implement advanced error handling within your test environment to catch these non-deterministic issues before they hit the pipeline.

Hands-on Exercise

1. Identify: Look at your GitHub Actions history. Find a run that failed but passed upon a manual retry.
2. Isolate: Move that specific test into a separate file and run it 10 times locally in a loop: for i in {1..10}; do pytest test_flaky.py || break; done.
3. Retry: Implement the --reruns flag in your .github/workflows/main.yml and commit the change to verify the pipeline now recovers from that specific flake.

Common Pitfalls

• The "Retry Everything" Mentality: If you set your retries to 10, you are hiding systemic bugs. A test should fail fast if it is broken. Limit retries to 2 or 3.
• Ignoring the Log: Just because a test passed on the second attempt doesn't mean you ignore the failure log of the first. Read the error—it usually tells you why it was non-deterministic (e.g., "Connection Timeout").
• Global State Pollution: If your test suite leaves behind files or processes, the next test in the queue will inherit a "dirty" environment. Always clean up in a teardown block.

FAQ

Q: Should I ever delete a flaky test? A: If a test is flaky and provides no business value, delete it. If it is critical, quarantine it (mark it as skip) until you have the time to rewrite it properly.

Q: Does adding retries make my build slower? A: Yes. Every retry adds time to your pipeline. This is the "stability tax" you pay for having non-deterministic tests.

Recap

Handling flaky tests is about moving from a state of "it works on my machine" to a robust, repeatable CI/CD process. By detecting patterns, using sensible retry limits, and focusing on data isolation, you transform your pipeline from an annoying source of noise into a reliable engineering tool.

Up next: Now that we have a stable testing foundation, we'll look at Container Registry Integration to store our images safely.