For the past decade, the recipe for AI progress was simple: shrink the transistors, pack more tensor cores onto a die, and run the clock faster. If a model needed to train faster, you threw more raw FLOPS at it. But we have reached the end of this straightforward scaling trajectory. The bottleneck in modern AI training and inference is no longer the speed at which a single GPU can compute matrix multiplications. As detailed in our analysis of modern AI chip architectures, the bottleneck is the wire.
When you scale an AI cluster to tens of thousands of GPUs, the compute units spend an unacceptable amount of time sitting idle. They are waiting for weight updates, synchronization barriers, and collective communication steps to finish. Nvidia realized this shift earlier than most. Their acquisition of Mellanox in 2019 was not a side bet on networking hardware; it was a fundamental pivot in their architecture strategy. Today, Nvidia is building a system where the network itself acts as the primary compute engine, and individual GPUs are merely execution units hanging off a massive, distributed backplane.
The Physical Wall of Monolithic Silicon
To understand this strategy, you have to look at the physical limits of chip manufacturing. Silicon fabrication relies on photolithography, which projects a circuit design onto a wafer. The equipment used for this process has a hard physical limit known as the reticle limit, typically around 858 square millimeters. You cannot print a single monolithic chip larger than this size without encountering catastrophic yield drops and soaring costs.
Nvidia's Hopper architecture pushed right up against this limit. The H100 GPU is a monolithic die measuring 814 square millimeters, leaving almost no room for physical expansion. To deliver more compute in the Blackwell generation, Nvidia had to abandon the monolithic approach. The Blackwell B200 is not a single large chip. It consists of two separate dies, manufactured at the reticle limit, placed side-by-side on a silicon interposer, and connected by a high-bandwidth link capable of 10 TB/s.
This multi-die design makes the two chips behave as a single unified processor. But this design choice also highlights the core challenge of modern hardware: packaging and interconnects have become the defining constraints of system performance. If you cannot move data between the dies fast enough, the extra tensor cores are useless. The moment data leaves the silicon package to travel across a circuit board, the energy cost and latency skyrocket.
NVLink and the Scale-Up Domain
Within a single server rack, Nvidia uses NVLink to bridge the communication gap. NVLink is a proprietary, high-speed interconnect designed specifically for GPU-to-GPU communication, bypassing the slow PCIe bus. In the Blackwell architecture, NVLink 5 provides 1.8 TB/s of bidirectional bandwidth per GPU.
The real magic happens when you combine this with the NVSwitch. In a standard server, you might have eight GPUs connected in a hybrid mesh. In Nvidia's GB200 NVL72 rack design, they use NVSwitch chips to link 72 GPUs into a single logical system with a unified memory pool. Every GPU in the rack can access the memory of any other GPU at full NVLink speeds.
This scale-up domain relies on copper cabling. At these frequencies, sending signals over copper traces on a standard printed circuit board causes massive signal degradation. Nvidia resolved this by using custom copper cables to connect the GPUs directly to the NVSwitch blades. This passive copper backplane eliminates the need for active transceivers, saving thousands of watts per rack while maintaining sub-microsecond latency.
But copper has physical length limits. You cannot run a copper NVLink cable across a data center. The high-frequency signals degrade after a few meters. When you need to scale beyond 72 GPUs to build a cluster of 10,000 or 100,000 nodes, you must transition from the scale-up domain of NVLink to the scale-out domain of traditional data center networks.
The Scale-Out Bottleneck: Lossless vs. Lossy Networks
In the scale-out domain, the network behaves differently. Traditional enterprise networks run on Ethernet. Ethernet was designed for reliability over long distances and variable conditions. It is a lossy protocol. If a switch buffer overflows, the switch drops the packet, and the sender retransmits it. For web traffic or database queries, this packet loss is normal and handled easily by TCP.
For AI training, packet loss is a disaster. AI training runs are highly synchronized. During the backward pass of a training step, GPUs must share their gradient updates using collective communication patterns like AllReduce. In these patterns, every GPU waits for all other GPUs to finish sending their data before moving to the next step.
If a single packet is dropped in a cluster of 10,000 GPUs, the entire training run stalls. The GPUs sit idle, consuming hundreds of kilowatts of power while waiting for the TCP stack to detect the loss, request a retransmission, and deliver the missing packet. This is the tail latency problem. The speed of the entire cluster is dictated by the slowest packet in the network.
To prevent this, Nvidia has pushed InfiniBand as the default interconnect for AI training. InfiniBand is a credit-based, lossless network. A sending node will not transmit data unless the receiving switch has explicitly declared that it has buffer space to accept it. This guarantees that packets are never dropped due to buffer overflows.
However, InfiniBand is expensive, requires specialized optical cabling, and demands unique engineering expertise to manage. Many hyperscalers, who are already shifting to custom ARM cloud data center CPUs to optimize general compute costs, also want to run AI workloads on standard Ethernet infrastructure. To address this, Nvidia introduced the Spectrum-X platform, which combines their Spectrum Ethernet switches with BlueField-3 Data Processing Units (DPUs).
The BlueField DPU acts as a smart network interface card (SmartNIC) inside the server. It handles packet processing, encryption, and storage virtualization, offloading these tasks from the CPU and GPU. More importantly, it works with the switch to run custom congestion control algorithms. If a path through the network starts to clog, the switch and the DPU dynamically reroute the packets or throttle the senders before buffers overflow. This brings Ethernet performance close to the lossless behavior of InfiniBand, making large-scale Ethernet clusters viable for LLM training.
Collective Communications and NCCL
The software layer that coordinates this hardware dance is the Nvidia Collective Communications Library (NCCL). When a developer writes code in PyTorch to train a model across multiple GPUs, PyTorch calls NCCL to handle the data exchange.
NCCL implements optimized algorithms for collective operations like:
AllReduce: Sums up tensors across all GPUs and distributes the result back to everyone.AllGather: Collects data from all GPUs and distributes the combined array to all nodes.ReduceScatter: Sums up data across GPUs and splits the result across the nodes.
These operations are not simple broadcasts. They are structured as rings or trees to minimize the hop count between switches. For example, in a ring topology, each GPU only sends data to its immediate neighbor and receives data from its other neighbor. The data circulates around the ring until everyone has the complete update.
If the physical network topology does not match the logical ring used by NCCL, the performance drops. If a packet must hop through three different switches to reach the next GPU in the ring, it adds latency. Nvidia's network strategy ensures that their switches, DPUs, and NCCL software are co-designed. The software knows the exact physical layout of the network and routes the collective operations along the paths of least resistance.
The Energy Cost of Moving Data
The ultimate driver of this architectural shift is energy physics. Computing a 16-bit floating-point operation on a modern GPU core takes a fraction of a picojoule. Moving those same 16 bits of data across a copper trace to local HBM memory takes tens of picojoules. Sending that data across an optical network link to another server rack takes hundreds of picojoules.
Compute (FP16) | [0.1 pJ]
Memory Access | [10 pJ] =======> 100x Energy Cost
Network Link | [100 pJ] =========================================> 1000x Energy Cost
We are spending orders of magnitude more energy moving data than processing it. If we continue to build faster GPUs without fixing the interconnects, we will build systems that consume megawatts of power just to move numbers back and forth between idle processors. This has led some to explore alternative paradigms like etching models in silicon to bypass general-purpose GPU bottlenecks.
This energy math is forcing the industry toward optical integration. Silicon photonics, where optical transceivers are integrated directly onto the chip package alongside the GPU and memory, is the next logical step. Instead of converting electrical signals to optical signals at the edge of the server tray, the chip itself will emit light. This co-packaged optics approach will drastically reduce the power required to send data across the scale-out network, bringing the latency of the optical network closer to the latency of the local board.
Nvidia's dominance is not secure simply because they make the fastest silicon dies. It is secure because they control the entire stack: the GPU, the NVLink bus, the NVSwitch, the InfiniBand switches, the DPUs, and the NCCL software. They have built an integrated system where the network is designed to behave like a backplane, and the backplane is designed to behave like a chip. Until competitors can match this system-level integration, focusing solely on building a faster GPU core will not be enough to win the AI infrastructure race.
