Imagine this scenario. It is 2:00 AM. Your phone goes off because your API gateway is throwing 504 Gateway Timeouts. You look at the dashboards and see that your order service is completely unresponsive.
You trace the issue back to a single third-party payment gateway. That gateway is having a minor outage, causing its response times to spike from 200 milliseconds to 15 seconds. Because your order service waits for the payment gateway to respond, its thread pool fills up instantly.
Every thread is stuck waiting for a socket connection that will never finish. The API gateway keeps routing traffic to the order service, which cannot accept new connections. The failure cascades up the stack, and now your entire front-end application is dead.
This is the classic cascading failure in distributed systems. When one service slows down, it drags down everything that depends on it.
You cannot prevent downstream services from failing. Network issues happen. Third-party APIs go down. Databases lock up. What you can do is prevent their failures from destroying your entire system (and exhausting your error budgets). That is where the circuit breaker pattern comes in.
The Danger of Naive Retries
When a network call fails, the immediate instinct of many engineers is to retry it—though doing so safely requires designing API idempotency keys to avoid duplicate side effects.
// The dangerous approach
async function fetchUserWithRetry(userId: string, retries = 3) {
for (let i = 0; i < retries; i++) {
try {
return await httpClient.get(`/users/${userId}`);
} catch (error) {
if (i === retries - 1) throw error;
}
}
}If the downstream service is down because it is overloaded, retries make the problem worse. In these scenarios, moving the work to fault-tolerant background job queues can help buffer the load. If 1,000 clients are hitting your service, and you retry three times, you just turned 1,000 failing requests into 4,000 requests.
This is called a retry storm. It acts like a self-inflicted Distributed Denial of Service (DDoS) attack. Instead of helping the downstream service recover, you keep hitting it while it is down, ensuring it stays down.
How the Circuit Breaker Works
An electrical circuit breaker stops the flow of electricity when it detects a fault, like a short circuit. It protects the rest of the house from catching fire.
In software, a circuit breaker wraps a network call. It monitors the failures. If the failure rate crosses a certain threshold, the circuit breaker trips. It stops making calls to the broken service and returns an error or a fallback response immediately.
A software circuit breaker operates in three states: Closed, Open, and Half-Open.
State Transitions:
[ Closed ] -> (Failures > Threshold) -> [ Open ]
[ Open ] -> (Timeout Expires) -> [ Half-Open ]
[ Half-Open ] -> (Success) -> [ Closed ]
[ Half-Open ] -> (Failure) -> [ Open ]
The Closed State
In this state, the circuit breaker is closed, meaning electricity (traffic) flows freely. All requests go through to the downstream service. The breaker keeps track of the number of successes and failures within a rolling time window. If the failure rate stays below your limit, the breaker remains closed.
The Open State
If the failure rate crosses your limit (for example, 50% of requests fail over a 10-second window), the breaker trips and enters the Open state.
In this state, all requests fail immediately without even trying to call the downstream service. This gives the downstream service time to recover and prevents your system from wasting threads and memory waiting for timeouts.
The Half-Open State
After a configured period (for example, 30 seconds), the breaker enters the Half-Open state.
It allows a limited number of trial requests to pass through to the downstream service. If these trial requests succeed, the breaker assumes the downstream service is healthy again. It resets the failure count and returns to the Closed state.
If any of the trial requests fail, the breaker assumes the service is still broken. It resets the timer and goes back to the Open state.
Implementing a Circuit Breaker in TypeScript
Let us build a simple, production-grade circuit breaker in TypeScript. This implementation will manage states, track failures, handle timeouts, and trigger fallbacks.
First, we define our states and configuration options.
type BreakerState = 'CLOSED' | 'OPEN' | 'HALF-OPEN';
interface BreakerOptions {
failureThreshold: number; // Number of failures before tripping
recoveryTimeout: number; // Time in ms to wait before testing the service again
requestTimeout: number; // Time in ms to wait before timing out a request
}Now, let us write the CircuitBreaker class.
class CircuitBreaker {
private state: BreakerState = 'CLOSED';
private failureCount = 0;
private lastStateChange: number = Date.now();
private options: BreakerOptions;
constructor(options: BreakerOptions) {
this.options = options;
}
public async execute<T>(action: () => Promise<T>, fallback: () => T): Promise<T> {
this.checkState();
if (this.state === 'OPEN') {
return fallback();
}
try {
const result = await this.executeWithTimeout(action);
this.onSuccess();
return result;
} catch (error) {
this.onFailure();
return fallback();
}
}
private async executeWithTimeout<T>(action: () => Promise<T>): Promise<T> {
return new Promise<T>((resolve, reject) => {
const timer = setTimeout(() => {
reject(new Error('Request timed out'));
}, this.options.requestTimeout);
action()
.then((result) => {
clearTimeout(timer);
resolve(result);
})
.catch((err) => {
clearTimeout(timer);
reject(err);
});
});
}
private checkState(): void {
if (this.state === 'OPEN') {
const now = Date.now();
const timeSinceTrip = now - this.lastStateChange;
if (timeSinceTrip > this.options.recoveryTimeout) {
this.transitionTo('HALF-OPEN');
}
}
}
private onSuccess(): void {
this.failureCount = 0;
if (this.state === 'HALF-OPEN') {
this.transitionTo('CLOSED');
}
}
private onFailure(): void {
this.failureCount++;
if (this.state === 'HALF-OPEN' || this.failureCount >= this.options.failureThreshold) {
this.transitionTo('OPEN');
}
}
private transitionTo(newState: BreakerState): void {
this.state = newState;
this.lastStateChange = Date.now();
// In a real system, you would emit events or log this change
console.log(`Circuit Breaker transitioned to: ${newState}`);
}
}How to Use This Implementation
Here is how you wrap a network call with this breaker.
const paymentBreaker = new CircuitBreaker({
failureThreshold: 3,
recoveryTimeout: 10000, // 10 seconds
requestTimeout: 2000 // 2 seconds
});
async function processPayment(amount: number) {
const action = async () => {
const response = await fetch('https://api.paymentgateway.com/charge', {
method: 'POST',
body: JSON.stringify({ amount })
});
return response.json();
};
const fallback = () => {
return {
success: false,
message: 'Payment system is currently unavailable. Please try again later.'
};
};
return paymentBreaker.execute(action, fallback);
}If the payment gateway fails three times in a row, the breaker trips. For the next ten seconds, any call to processPayment returns the fallback message instantly. No network calls are made. No threads are blocked.
Fallback Strategies
When a circuit is open, you must decide how to handle the failure. The fallback strategy depends heavily on the type of data you are fetching.
1. Silent Fail
If the failing service is not critical to the core user experience, you can fail silently. For example, if the recommendation engine on an e-commerce site goes down, you can return an empty list or hide the recommendations section entirely. The user can still buy products.
const fallback = () => [];2. Cached Data
If you are fetching data that does not change often, you can return the last cached version. If the user profile service goes down, you can serve the cached profile from Redis or local memory. It might be slightly stale, but it is better than a crash.
const fallback = async () => {
return await redis.get(`user:profile:${userId}`) || defaultProfile;
};3. Static Defaults
You can return a generic, pre-configured response. If the tax calculation service goes down during checkout, you could fall back to a flat tax rate estimate and adjust it later.
Real-World Engineering Gotchas
Implementing the pattern in code is only half the battle. You have to configure it correctly for production.
Timeout Alignment
Your request timeout must be shorter than the recovery timeout. If your request timeout is 5 seconds, and your recovery timeout is 2 seconds, the breaker will try to test the downstream service in the Half-Open state before the previous timeout has finished. Keep your request timeouts tight. If a service normally responds in 100 milliseconds, set the timeout to 500 milliseconds, not 5 seconds.
Memory Leaks and State in Microservices
If you run multiple instances of your service (for example, five Kubernetes pods), each pod has its own local circuit breaker instance. If one pod trips its breaker, the other four pods might still send traffic to the failing service.
You might think you need a shared, distributed circuit breaker using Redis. Do not do this.
A distributed circuit breaker adds network latency and introduces another dependency that can fail. If Redis goes down, your circuit breakers stop working. Local, in-memory circuit breakers in each pod are usually good enough. The traffic will naturally trip the breakers across all pods quickly anyway.
Thread Pool Isolation
A circuit breaker prevents network calls, but it does not prevent your application server from running out of threads if you have too many concurrent requests waiting for the breaker to trip.
To solve this, combine the circuit breaker with the bulkhead pattern. Isolate your thread pools. Assign a maximum of 10 threads to the payment gateway service. Even if that service fails and the breaker has not tripped yet, only those 10 threads will block. The rest of your application threads remain free to handle other requests.
Monitoring and Alerts
A tripped circuit breaker is a symptom of a larger problem. If a breaker enters the Open state, your operations team needs to know immediately.
You should expose metrics from your circuit breaker. Track:
- Current state (Closed = 0, Open = 1, Half-Open = 2)
- Failure count
- Number of requests rejected while the circuit was open
If you use Prometheus, you can expose these as gauge metrics. Set up an alert that triggers if the state gauge remains at 1 for more than two minutes. This means a dependency is down, and the system is actively failing fast.
Summary of Best Practices
To make the most of this pattern, keep these guidelines in mind:
- Do not wrap everything. Use circuit breakers only for network calls, database queries, and file system operations. Do not wrap local, CPU-bound operations.
- Keep timeouts short. A circuit breaker is useless if it spends minutes waiting for socket timeouts.
- Write test cases for the Half-Open state. It is the most common place for race conditions to hide.
- Graceful degradation is the goal. Always design a fallback. A broken feature should look like a simplified user interface, not an application crash.
