Building Resilient WebSocket Gateway Engines for Real-Time Event Streaming

Most developers start building real-time features by spinning up a basic socket server. It works fine in development with a few hundred users. But when you push that setup to production and hit tens of thousands of concurrent connections, the architecture starts to buckle. The server runs out of file descriptors, network hiccups drop thousands of clients at once, and a sudden wave of reconnections crashes your database.

WebSockets change the rules of scaling. With standard HTTP (for more on how this compares to other protocols, see gRPC vs REST in distributed systems), if a server instance dies, your load balancer routes the next request to another healthy node. The system is stateless. But with WebSockets, connections are stateful and persistent. A client is pinned to a specific server instance. If that server goes down, the connection breaks, and the client state is lost.

Building a resilient WebSocket gateway requires designing for failure, similar to the strategies used when designing resilient connection error handling in Redis task queues. You need to manage connection state, handle sudden drops, protect downstream services from traffic spikes, and route messages across a distributed cluster.

The Stateful Connection Problem

In a typical web application, horizontal scaling is simple. You add more app servers behind a load balancer. Since each request contains all the context it needs (like a token in the header), any server can handle any request.

WebSockets break this model. Once a client establishes a connection, a long-lived TCP socket remains open between the client and a specific gateway instance.

[Client A] -> [Load Balancer] -> [Gateway Server 1] (Holds Socket A)
[Client B] -> [Load Balancer] -> [Gateway Server 2] (Holds Socket B)

If Gateway Server 1 needs to send a message to Client B, it cannot do so directly because Client B's socket lives on Gateway Server 2. You need a way to route messages between these instances.

To solve this, you need a pub/sub backplane. While you might use a type-safe event bus in TypeScript for in-process event propagation, a distributed system requires a shared broker. Redis, NATS, or RabbitMQ work well here. When Client B connects to Gateway Server 2 and subscribes to a channel, the gateway subscribes to that same channel on the pub/sub backplane. When a backend service publishes a message for Client B, it sends it to the broker. The broker broadcasts it to the correct gateway, which then pushes it down the active WebSocket connection.

Using NATS for this layer is often more efficient than Redis. NATS is built for high-throughput, low-latency messaging and handles clustering natively. It doesn't write messages to disk by default, which keeps CPU usage low on the messaging layer.

Mitigating the Reconnection Storm

When you deploy updates to your gateway fleet, you have to restart the servers. If you have 50,000 users connected to a single node and that node restarts, those 50,000 clients will disconnect simultaneously.

If your client-side code is written to reconnect immediately upon disconnection, you will trigger a thundering herd. Fifty thousand clients will hit your load balancer, attempt to complete a TLS handshake, run authentication checks against your database, and load user sessions all at the same time. This will crash your authentication service and database before the gateway even finishes initialization.

To prevent this, you must enforce jittered exponential backoff on the client side. Instead of reconnecting immediately, the client should wait a random amount of time.

Here is a simple client-side reconnection strategy:

function connectWithRetry(attempt = 0) {
  const baseDelay = 1000; // 1 second
  const maxDelay = 30000; // 30 seconds
  const factor = 2;
  
  // Calculate exponential delay
  let delay = baseDelay * Math.pow(factor, attempt);
  delay = Math.min(delay, maxDelay);
  
  // Add random jitter (up to 30% of the delay)
  const jitter = delay * 0.3 * Math.random();
  const finalDelay = delay + jitter;
 
  setTimeout(() => {
    const ws = new WebSocket('wss://api.example.com/stream');
    
    ws.onclose = () => {
      connectWithRetry(attempt + 1);
    };
    
    ws.onopen = () => {
      console.log('Connected');
      // Reset attempts on successful connection
    };
  }, finalDelay);
}

On the gateway side, you should also rate-limit incoming connection upgrades. If the gateway is receiving more connections than it can handle, it should return an HTTP 429 Too Many Requests status code. This stops the client from attempting a full TLS handshake and database lookup, protecting your internal infrastructure.

Heartbeats and Half-Open Connections

TCP connections can fail silently. If a user walks into an elevator or loses cell signal, their phone's operating system might not send a FIN packet to close the connection. The gateway server still thinks the connection is active. It will keep holding the file descriptor open and trying to send data to a dead socket.

These are called half-open connections. If you do not clean them up, your server will eventually hit its file descriptor limit and refuse new connections.

To detect and clean up dead sockets, you need an application-level ping/pong mechanism. Do not rely on TCP keepalives alone; they take too long to detect failures (often hours by default).

The gateway should periodically send a ping frame to the client. The client must respond with a pong frame within a set window. If the client fails to respond, the gateway terminates the connection and frees up resources.

Here is a robust Go implementation using the gorilla/websocket library that handles ping/pong deadlines:

package main
 
import (
	"log"
	"net/http"
	"time"
 
	"github.com/gorilla/websocket"
)
 
const (
	// Time allowed to write a message to the peer.
	writeWait = 10 * time.Second
 
	// Time allowed to read the next pong message from the peer.
	pongWait = 60 * time.Second
 
	// Send pings to peer with this period. Must be less than pongWait.
	pingPeriod = (pongWait * 9) / 10
)
 
var upgrader = websocket.Upgrader{
	ReadBufferSize:  1024,
	WriteBufferSize: 1024,
}
 
func handleConnection(w http.ResponseWriter, r *http.Request) {
	conn, err := upgrader.Upgrade(w, r, nil)
	if err != nil {
		log.Println("Upgrade error:", err)
		return
	}
	defer conn.Close()
 
	// Configure read deadlines
	conn.SetReadLimit(512 * 1024) // 512KB max message size
	conn.SetReadDeadline(time.Now().Add(pongWait))
	
	// When we receive a pong, extend the read deadline
	conn.SetPongHandler(func(string) error {
		conn.SetReadDeadline(time.Now().Add(pongWait))
		return nil
	})
 
	// Start write loop
	ticker := time.NewTicker(pingPeriod)
	defer ticker.Stop()
 
	go func() {
		for range ticker.C {
			conn.SetWriteDeadline(time.Now().Add(writeWait))
			if err := conn.WriteMessage(websocket.PingMessage, nil); err != nil {
				log.Println("Ping write error:", err)
				return
			}
		}
	}()
 
	// Read loop
	for {
		_, _, err := conn.ReadMessage()
		if err != nil {
			if websocket.IsUnexpectedCloseError(err, websocket.CloseGoingAway, websocket.CloseAbnormalClosure) {
				log.Printf("Read error: %v", err)
			}
			break
		}
	}
}

Setting write deadlines is critical. If a client is slow or has disconnected, the write call will block. Without a deadline, the goroutine handling that connection will hang indefinitely, leaking memory.

Managing Backpressure and Slow Clients

What happens when your backend publishes 1,000 events per second, but a client is on a spotty mobile network and can only download 50 events per second?

If your gateway keeps queueing messages in memory for that client, the memory footprint of the gateway will grow until the operating system terminates the process due to an out-of-memory error. This is a backpressure problem.

To handle slow clients, you must set a maximum buffer size per connection. If the buffer fills up, you have to make a choice:

1. Drop the oldest messages in the queue to make room for new ones.
2. Drop the newest messages.
3. Terminate the connection.

For real-time dashboards or stock tickers, dropping old messages is usually the best approach. On the client side, you can process these high-frequency updates by offloading heavy computations with Web Workers to keep the main thread responsive. The client only cares about the latest state. For chat applications where message delivery is guaranteed, you should terminate the connection, forcing the client to reconnect and sync missed messages from a database.

Here is a conceptual model of a buffered channel writer that drops old messages when the channel fills up:

type Client struct {
	sendChan chan []byte
}
 
func (c *Client) QueueMessage(msg []byte) {
	select {
	case c.sendChan <- msg:
		// Message queued successfully
	default:
		// Buffer is full. Evict the oldest message to make room.
		select {
		case <-c.sendChan:
			// Removed one message from the queue
		default:
		}
		// Try queueing the new message again
		c.sendChan <- msg
	}
}

This pattern keeps memory usage bounded. It ensures that one slow user cannot consume all the server's RAM and impact other connected clients.

Session Resumption and Sequence Numbers

When a client loses connection for a few seconds due to a network switch (like moving from Wi-Fi to cellular data), they do not want to reload the entire application state. They just want the messages they missed during the brief offline window.

To implement session resumption:

1. Every message sent to a client must include a monotonically increasing sequence number seq.
2. The client tracks the highest sequence number it has successfully processed.
3. When reconnecting, the client sends this sequence number to the gateway (e.g., resume_session?last_seq=1402).
4. The gateway checks a cache (like Redis) for messages matching that session ID with sequence numbers greater than 1402.
5. The gateway replays those missed messages to the client before resuming the live stream.

You must set a time-to-live limit on these cached sessions. Storing missed messages for 2 to 5 minutes is usually enough to cover typical network switches. If a client is offline for longer, expire the session and force them to perform a full state sync from your primary database.

Load Balancing and Sticky Sessions

When deploying a WebSocket gateway behind a load balancer, you need to decide how to distribute connections.

Layer 4 (TCP) load balancing is fast and efficient. The load balancer does not look at the HTTP headers; it simply forwards the TCP packets to a gateway server. This reduces CPU usage on the load balancer. However, Layer 4 load balancing makes it harder to implement smart routing or authentication at the load balancer level.

Layer 7 (HTTP/HTTPS) load balancing allows the load balancer to inspect the incoming handshake request. You can terminate TLS at the load balancer and read cookies or headers to route requests.

If you use Layer 7 load balancing, make sure to disable sticky sessions unless you have a specific architectural reason to keep them. Sticky sessions can cause uneven distribution of connections. If one gateway node restarts and all its clients try to reconnect, a round-robin load balancing strategy will distribute them evenly across the remaining nodes. Sticky sessions might force them all back onto the same server, creating a hotspot.

By separating connection termination from application logic, using a high-performance pub/sub broker like NATS, and enforcing strict client-side retry policies, you can build a WebSocket gateway that handles millions of events without falling over.