Designing Resilient Connection Error Handling in Redis Task Queues

Background workers are the unsung heroes of modern web applications. They handle the slow, heavy tasks like processing payments, resizing images, and sending emails, keeping your web servers fast and responsive. Offloading these tasks is a crucial strategy for optimizing Core Web Vitals in Next.js, as it prevents long tasks from blocking the main thread and degrading user experience. Before these tasks even reach the queue, they typically originate from user input that has been sanitized using robust React form validation with Zod. Once validated, many teams choose Redis to back these task queues because it is fast, simple, and supports blocking list operations out of the box.

But running a Redis-backed queue in production reveals a frustrating class of bugs. You deploy your workers, everything runs smoothly for a few weeks, and then jobs suddenly stop processing. When you check your server logs, there are no error messages. The worker process is still running, consuming almost zero CPU, but doing absolutely nothing. The queue just keeps growing.

This is the classic zombie worker state. It almost always traces back to unhandled network drops between your worker and your Redis instance. If you rely on the default settings of your Redis client library, a brief network blip can leave your workers permanently disconnected but convinced they are still waiting for work.

Why Connections Die Silently

To understand why workers hang, you have to look at how operating systems handle TCP connections. When a client connects to Redis, they establish a TCP socket. If no data is moving back and forth, the connection is silent.

In a web application making standard cache queries, this is rarely an issue. The app sends a query, receives a response, and repeats the process. If the network drops, the next query fails immediately, the app throws an error, and you can catch it and retry.

Task queues work differently. They rely on long-lived, idle connections. A worker connects to Redis and waits for a job to appear. If your application is quiet, that connection might sit idle for minutes or hours.

Problems arise when intermediate network devices get involved. Firewalls, NAT gateways, and cloud load balancers (like AWS ELB) track active connections in translation tables. To save memory, these devices silently drop inactive connections from their tables after a period of inactivity, often between 5 and 15 minutes.

When this happens, the firewall simply stops forwarding packets for that connection. It does not send a TCP RST (reset) packet to the client or the server. The connection is dead, but neither the worker nor the Redis server knows it yet. The worker's operating system believes the socket is still open, waiting for data that will never arrive.

Configuring TCP Keepalives

The first line of defense against silent connection drops is TCP keepalives. When you enable keepalives, your operating system periodically sends empty probe packets across the connection to verify the remote host is still there. If the remote host fails to respond to a set number of probes, the OS declares the connection dead, closes the socket, and alerts your application.

By default, the Linux kernel sets the keepalive idle time to 7200 seconds (two hours). If a firewall drops your connection after 10 minutes of inactivity, your worker will sit in a zombie state for two hours before the OS notices.

You must configure keepalives at the client library level to use much shorter intervals. Most modern Redis clients allow you to pass socket options directly. Here is how you configure keepalives using the ioredis library in Node.js:

const Redis = require('ioredis');
 
const redis = new Redis({
  host: '127.0.0.1',
  port: 6379,
  keepAlive: 10000, // Send a probe every 10 seconds
});

With this configuration, the client tells the operating system to send a probe after 10 seconds of inactivity. If the network path is broken, the connection fails quickly, allowing your client library's reconnection logic to kick in.

The Danger of Infinite Blocking Pops

Many queue designs use blocking commands like BRPOP or BLPOP to fetch jobs. These commands tell Redis: "If the queue is empty, block this connection and wait until an item is pushed."

A common mistake is setting the timeout parameter of these commands to zero, which tells Redis to block indefinitely:

// Naive implementation that blocks forever
while (true) {
  try {
    const [queue, jobData] = await redis.brpop('task_queue', 0);
    await processJob(jobData);
  } catch (err) {
    console.error('Worker error:', err);
  }
}

If the underlying TCP connection drops while the client is blocked on BRPOP queue 0, the client will never write to the socket. It is waiting to read. Because it never writes, it may not detect the broken socket even with keepalives enabled, depending on how the OS handles keepalive probes on blocked sockets.

To prevent this, never block indefinitely. Set a reasonable timeout, such as 5 or 30 seconds.

while (true) {
  try {
    // Block for a maximum of 5 seconds
    const result = await redis.brpop('task_queue', 5);
    
    if (!result) {
      // No jobs in the last 5 seconds, loop again
      continue;
    }
 
    const [queue, jobData] = result;
    await processJob(jobData);
  } catch (err) {
    console.error('Connection lost or error occurred. Waiting to retry...', err);
    // Sleep briefly to avoid tight loops on persistent failures
    await new Promise(resolve => setTimeout(resolve, 5000));
  }
}

When you set a timeout, the client regularly completes the command cycle. If the connection is dead, the next loop iteration will attempt to write a new BRPOP command to the socket. The write operation will fail immediately, triggering the client's error handler and initiating a reconnect.

Reconnection Loops and the Thundering Herd

When a network drop occurs, your workers will try to reconnect. If you have dozens of worker processes running across a cluster, a brief network hiccup can cause all of them to disconnect at the same time.

If all workers immediately try to reconnect as fast as possible, they will overwhelm your Redis server. This is the thundering herd problem. The sudden spike in CPU and connection limits can crash a recovering Redis instance.

You need to design a reconnection strategy that uses exponential backoff and randomized jitter. Instead of retrying every second, increase the delay between attempts and add a random offset to prevent the workers from syncing up.

We can calculate the retry delay using a simple formula:

delay = min(maxDelay, baseDelay * 2^attempt)

Then, we add random jitter to spread the connection attempts over time:

jitter = random(0, jitterFactor * delay) finalDelay = delay + jitter

Here is how to implement this strategy in ioredis:

const redis = new Redis({
  host: '127.0.0.1',
  port: 6379,
  retryStrategy(times) {
    // Stop retrying after 100 failed attempts
    if (times > 100) {
      return null; // Returning null stops reconnection attempts
    }
 
    // Calculate exponential backoff (cap at 10 seconds)
    const delay = Math.min(times * 100, 10000);
    
    // Add random jitter between 0 and 1000ms
    const jitter = Math.random() * 1000;
    
    return delay + jitter;
  }
});

This ensures that your workers stagger their connection attempts, giving your network and Redis server room to stabilize.

The Offline Queue Memory Leak

Most Redis client libraries try to be helpful when a connection drops. They do not want your application to lose commands, so they buffer any commands sent during a disconnect in an internal memory array. Once the connection is restored, they replay the buffered commands.

For standard web apps—such as those built using our guide on getting started with Next.js—this buffer is useful. But for a queue worker running a continuous loop, it is a memory leak waiting to happen.

Consider what happens in our worker loop when the connection drops. The loop catches the error, sleeps for a few seconds, and then calls BRPOP again. If the library buffers commands, that BRPOP call is pushed to an in-memory queue. The loop continues to run, calling BRPOP over and over, piling up commands in memory.

If your Redis instance is offline for a few hours, the worker process will consume all available system memory and crash with an Out Of Memory (OOM) error.

You must disable the offline queue for your worker clients. In ioredis, you do this by setting enableOfflineQueue to false:

const redis = new Redis({
  host: '127.0.0.1',
  port: 6379,
  enableOfflineQueue: false, // Do not buffer commands when disconnected
});

With the offline queue disabled, any command sent while disconnected fails immediately. Your worker loop will catch the error, log it, sleep, and try again without consuming extra memory.

Implementing the Reliable Queue Pattern

Standard BRPOP is a destructive operation. The moment a worker pulls a job from the queue, Redis deletes it. If your worker crashes, loses power, or loses its network connection mid-job, that task is gone forever.

To build a resilient system, you need an acknowledgment pattern. The worker should move the job to a temporary "processing" list and only remove it once the work is complete.

Historically, developers used the RPOPLPUSH command for this. In modern Redis versions (6.2 and newer), you should use LMOVE (or the blocking version BLMOVE).

The workflow looks like this:

1. Use BLMOVE to atomically pop a task from task_queue and push it to a worker-specific processing_queue.
2. Process the task in your application.
3. Once successful, remove the task from processing_queue.

Here is a Node.js implementation of this pattern:

const workerId = `worker:${process.pid}`;
const mainQueue = 'task_queue';
const processingQueue = `processing:${workerId}`;
 
async function workerLoop() {
  while (true) {
    try {
      // Move task from main queue to processing queue atomically
      // Blocks for up to 5 seconds
      const jobData = await redis.blmove(
        mainQueue,
        processingQueue,
        'RIGHT',
        'LEFT',
        5
      );
 
      if (!jobData) {
        continue;
      }
 
      // Process the job
      await processJob(JSON.parse(jobData));
 
      // Work is done, remove the job from the processing list
      await redis.lrem(processingQueue, 1, jobData);
 
    } catch (err) {
      console.error('Error processing job:', err);
      // Wait before retrying to avoid spinning CPU on network failures
      await new Promise(resolve => setTimeout(resolve, 5000));
    }
  }
}

If the worker crashes mid-job, the task remains in the processing queue. You can run a separate cleanup process (a sweeper) that runs periodically, checks for stale processing queues, and pushes those tasks back to the main queue for other workers to pick up.

Handling Graceful Shutdowns

When you deploy new code or scale down your servers, your orchestration tool (like Kubernetes, ECS, or PM2) sends a SIGTERM signal to your worker processes. You have a short window, usually 30 seconds, to finish what you are doing before the system sends a SIGKILL and terminates the process.

If you do not handle this signal, the worker will die mid-job, leaving tasks stranded in your processing queue.

You must catch these termination signals, stop polling for new work, finish the active job, and close the Redis connection cleanly.

let isRunning = true;
let activeJobsCount = 0;
 
// Listen for termination signals
process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);
 
function shutdown() {
  console.log('Received shutdown signal. Stopping queue poll...');
  isRunning = false;
}
 
async function run() {
  while (isRunning) {
    try {
      const result = await redis.blmove(
        'task_queue',
        `processing:${workerId}`,
        'RIGHT',
        'LEFT',
        5
      );
 
      if (!result) continue;
 
      activeJobsCount++;
      try {
        await processJob(JSON.parse(result));
        await redis.lrem(`processing:${workerId}`, 1, result);
      } finally {
        activeJobsCount -= 1;
      }
 
    } catch (err) {
      console.error('Worker loop error:', err);
      await new Promise(resolve => setTimeout(resolve, 5000));
    }
  }
 
  // Wait for active jobs to finish before exiting
  await waitForJobsToComplete();
  await redis.quit();
  console.log('Clean shutdown complete.');
  process.exit(0);
}
 
async function waitForJobsToComplete() {
  return new Promise((resolve) => {
    const check = () => {
      if (activeJobsCount === 0) {
        resolve();
      } else {
        console.log(`Waiting for ${activeJobsCount} active jobs to finish...`);
        setTimeout(check, 1000);
      }
    };
    check();
  });
}
 
run();

This code intercepts the shutdown signal and flips the isRunning flag. The loop stops polling for new tasks. The process then waits for activeJobsCount to hit zero, closes the Redis client connection cleanly, and exits.

Building error handling into your queues requires moving away from default configurations. By combining TCP keepalives, blocking timeouts, exponential backoff, and proper signal handling, you can prevent silent failures and keep your background jobs processing reliably.