Implementing Circuit Breakers: A Guide to System Resilience

Previously in this course, we explored the theory behind Designing for Failure: Resilience and Fault Tolerance Basics. While understanding why we need resilience is vital, today we’re moving to the "how." We are going to build a functional circuit breaker to prevent your services from overwhelming failing dependencies, a practice that builds directly on the patterns discussed in API resilience with circuit breakers: stop cascading failures.

The Circuit Breaker Pattern from First Principles

In a distributed system, when a downstream service—like a database or an external API—starts failing, your application often continues to hammer it with requests. This creates a "thundering herd" effect, where your service wastes resources waiting for timeouts on a service that is already dead.

A circuit breaker acts as a state machine that sits between your code and the external dependency. It has three primary states:

1. Closed: Everything is normal. Requests pass through to the dependency. The breaker tracks failure counts.
2. Open: The dependency is failing too often. The breaker "trips," and all requests are immediately rejected (fail-fast) without calling the dependency.
3. Half-Open: After a cooldown period, the breaker allows a limited number of "test" requests. If they succeed, the breaker resets to Closed. If they fail, it returns to Open.

A Concrete Implementation

Let's write a simple, thread-safe Circuit Breaker in Python. While production systems often use libraries like Hystrix or Resilience4j, building one from scratch is the best way to understand the underlying mechanics.

PYTHON
import time

class CircuitBreaker:
    def __init__(self, failure_threshold=3, recovery_timeout=10):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.failures = 0
        self.state = "CLOSED"
        self.last_failure_time = None

    def call(self, func, *args, **kwargs):
        if self.state == "OPEN":
            if time.time() - self.last_failure_time > self.recovery_timeout:
                self.state = "HALF-OPEN"
            else:
                raise Exception("Circuit is OPEN - Request rejected")

        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            raise e

    def _on_success(self):
        self.failures = 0
        self.state = "CLOSED"

    def _on_failure(self):
        self.failures += 1
        self.last_failure_time = time.time()
        if self.failures >= self.failure_threshold:
            self.state = "OPEN"

Configuring Failure Thresholds

The failure_threshold and recovery_timeout are not arbitrary numbers; they are tuning parameters for your system's SLA (Service Level Agreement).

• Low threshold: Makes your system highly sensitive. Good for critical dependencies where even one failure is unacceptable.
• High threshold: Prevents "flapping" (switching states too frequently due to transient network blips).
• Recovery Timeout: Too short, and you risk slamming a recovering service; too long, and you stay in a degraded state unnecessarily.

Hands-on Exercise

Integrate the CircuitBreaker class above into your project's service layer.

1. Create a mock dependency function that randomly raises an Exception.
2. Wrap that function in your CircuitBreaker.call() method.
3. Run a loop of 10 requests. Observe how the breaker trips after the 3rd failure.
4. Wait for the recovery_timeout and verify that the system allows a request again.

Common Pitfalls

• Global State: If you are running multiple instances of your service, a local in-memory circuit breaker only tracks the health of the dependency from the perspective of that specific instance. For distributed protection, you might need a distributed circuit breaker.
• Ignoring Exceptions: Only trip the breaker for actual service failures (like 5xx errors or timeouts). Do not trip it for client-side errors (4xx), as those are usually logic errors, not dependency health issues.
• Lack of Fallback: A circuit breaker that just returns an error is only half the solution. You must implement a "fallback" method—like returning cached data or a default value—to maintain graceful degradation.

FAQ

Q: Should I use a circuit breaker for every HTTP call? A: No. Use them for external dependencies where failure is possible and latency is a concern. Using them for local function calls adds unnecessary complexity.

Q: Can I use circuit breakers to handle database slowness? A: Yes, but ensure your failure detection logic accounts for timeouts, not just connection errors.

Recap

Circuit breakers are essential for building resilient distributed systems. By managing states—Closed, Open, and Half-Open—we stop cascading failures and give struggling dependencies the room they need to recover. Always pair your breaker with a clear fallback strategy.

Up next: We will discuss how to ensure your requests are processed safely even when retries happen, by learning about Idempotency in Distributed Systems.