Silicon design has hit a hard physical limit. Compute performance scaled exponentially for decades, matching Moore's Law. Memory bandwidth did not. This performance gap created the memory wall. In modern AI workloads, especially Large Language Models (LLMs), memory bandwidth determines performance, not raw compute capacity.
To understand why, calculate the arithmetic intensity of autoregressive LLM inference. Arithmetic intensity defines the ratio of execution operations to memory access, measured in FLOPs per byte. During the generation phase of an LLM, the model processes one token at a time. The system must load every weight of the model from memory to the compute cores to generate a single token.
For a model with N parameters, computing the next token requires approximately 2 * N floating-point operations. If we use FP16 precision, each parameter occupies 2 bytes of memory. Therefore, we must read 2 * N bytes of data. This means the arithmetic intensity is:
Intensity = (2 * N FLOPs) / (2 * N Bytes) = 1 FLOP/Byte
If a chip offers 300 TeraFLOPs of compute performance but only 1.5 Terabytes per second (TB/s) of memory bandwidth, the maximum throughput for batch size 1 is constrained by the memory speed. The compute cores sit idle most of the time, waiting for weights to arrive from memory. To fully utilize the 300 TeraFLOPs, the system needs an arithmetic intensity that matches the chip's compute-to-bandwidth ratio. For this hypothetical chip:
Required Intensity = 300 TeraFLOPs / 1.5 TB/s = 200 FLOPs/Byte
At batch size 1, the arithmetic intensity is 1. The hardware runs at 0.5% of its peak compute capacity. This is the memory-bound regime. To shift the workload into the compute-bound regime, hardware architects must either increase memory bandwidth or increase batch size. Increasing batch size processes multiple tokens simultaneously, reusing the loaded weights across different inputs, which raises the arithmetic intensity. But larger batch sizes increase memory capacity requirements and latency, creating a different set of engineering constraints.
High Bandwidth Memory and Packaging
Standard DDR memory cannot handle the bandwidth demands of AI workloads, a constraint worsened by the global DRAM shortage. Graphics Double Data Rate (GDDR) memory, used in traditional GPUs, offers higher speeds but still relies on narrow buses (typically 256-bit or 384-bit) running at high clock rates. This design causes high power consumption and limited capacity.
AI accelerators solve this with High Bandwidth Memory (HBM). Instead of placing memory chips elsewhere on a printed circuit board, HBM stacks DRAM dies vertically using Through-Silicon Vias (TSVs). TSVs are microscopic vertical physical connections that pass through the silicon dies.
The stacked memory die connects to the compute die through a silicon interposer. This packaging method is called 2.5D integration. TSMC's Chip-on-Wafer-on-Substrate (CoWoS) is the dominant packaging technology for this layout. The interposer acts as a high-density routing layer, allowing a bus width of 1024 bits per HBM stack. HBM3e, the current standard, runs at pin speeds up to 9.6 Gbps, delivering over 1.2 TB/s of bandwidth per stack. A chip layout with eight HBM3e stacks achieves over 9.6 TB/s of aggregate memory bandwidth.
Moving data across the interposer requires significantly less energy than moving data across a motherboard. Physical distance dictates energy cost. On-chip SRAM access requires roughly 0.1 picojoules per bit (pJ/bit). HBM access across the interposer requires about 1 to 2 pJ/bit. Standard off-package LPDDR5 or DDR5 access requires 10 to 20 pJ/bit. By keeping memory physically close to the compute cores, architects reduce thermal output and power consumption.
But 2.5D packaging introduces thermal and mechanical stress. The compute die and the HBM stacks sit close together on the interposer, creating a concentrated heat source. The materials expand at different rates when heated, which can crack the microbumps connecting the dies to the interposer. Cooling these packages requires advanced liquid cooling systems or custom vapor chambers.
Compute Cores: Systolic Arrays vs. Vector Processors
AI workloads consist mostly of matrix multiplications. Computing C = A * B requires multiplying rows by columns and accumulating the results. Traditional CPUs use Scalar or Vector (SIMD) units for these operations. Vector processors load data into registers, perform a single instruction on multiple data points, and write the results back to registers.
For large matrix operations, register file access becomes a bottleneck. Loading inputs, weights, and intermediate values repeatedly from the register file consumes time and power. AI accelerators use two main architectural alternatives: Systolic Arrays and Tensor Cores.
Systolic Arrays
Google's Tensor Processing Unit (TPU) popularized the systolic array. A systolic array is a grid of processing elements (PEs) that process data like blood pumping through a heart. In a weight-stationary systolic array, the weights of the matrix are loaded into the PEs and remain fixed. The input data flows horizontally through the grid, while the accumulated results flow vertically.
Each PE contains a multiply-accumulate (MAC) unit and local registers. When an input value meets a weight inside a PE, the MAC unit multiplies them, adds the result to the value coming from the top neighbor, and passes the sum to the bottom neighbor. The input value is passed to the right neighbor.
This design minimizes register file reads. A value read from memory once passes through dozens of PEs, maximizing data reuse. The control logic is simple because data flows deterministically on every clock cycle.
Tensor Cores
NVIDIA's Tensor Cores use a different approach. Instead of a rigid grid, Tensor Cores are specialized execution units within the Streaming Multiprocessor (SM). They execute warp-level matrix instructions. For example, a Tensor Core can execute a matrix multiply-accumulate operation on small matrices (such as 4x4 or 8x8) in a single clock cycle.
Modern Tensor Cores support mixed-precision execution. They multiply low-precision inputs (FP8 or FP16) to speed up compute, then accumulate the results in higher precision (FP32) to maintain model accuracy. This design balances throughput and mathematical precision.
To double compute density, modern architectures implement structured sparsity. NVIDIA's Ampere and Hopper architectures use a 2:4 sparsity pattern. The hardware assumes that out of every four values in a weight matrix, at least two are zero. The compiler compresses the matrix, removing the zeros. The Tensor Cores then skip calculations involving these zero values, doubling execution speed without losing model accuracy.
Alternative Architectures: SRAM-Only and Spatial Dataflow
Some hardware designers bypass the HBM bottleneck entirely by building architectures without external memory, similar to etching models in silicon to bypass general-purpose GPU limits.
Wafer-Scale Integration
Cerebras Systems designed the Wafer-Scale Engine (WSE). Instead of cutting a silicon wafer into hundreds of small chips, Cerebras uses the entire wafer as a single giant chip. The WSE-3 contains 900,000 AI-optimized compute cores on a single piece of silicon.
Instead of HBM, the WSE uses only on-chip SRAM. The chip contains 44 Gigabytes of SRAM distributed across the compute cores. Because SRAM is integrated directly into the silicon fabric, it provides massive performance advantages:
- Memory bandwidth: Over 21 Petabytes per second (PB/s).
- Latency: Sub-nanosecond access times.
- Energy: Minimal power required to move data, as distances are measured in micrometers.
The tradeoff is physical scale and cost. Fitting 44 GB of SRAM on a single wafer limits the size of the model that can run on a single chip. A 70-billion parameter model in FP16 requires 140 GB of memory, which exceeds the capacity of a single wafer. Running large models requires clustering multiple wafer-scale systems, shifting the bottleneck from memory bandwidth to network interconnects.
Deterministic Spatial Architectures
Another approach is the Language Processing Unit (LPU) by Groq. Backed by a major Groq Neocloud pivot, this architecture avoids traditional GPU dynamic hardware scheduling, branch predictors, and instruction caches to manage execution flow.
Because execution is deterministic, there are no cache misses, pipeline stalls, or synchronization overheads. This predictability allows the compiler to optimize data movement across a cluster of chips without needing dynamic network handshakes. This design delivers high throughput and low latency for sequential workloads like LLM token generation.
Interconnects and Cluster Scaling
No single chip can train or run inference on trillion-parameter models. Systems must scale out across clusters of thousands of accelerators. At this scale, the network interconnect becomes the primary bottleneck.
Scaling uses two primary pathways: scale-up and scale-out.
Scale-Up Interconnects
Scale-up refers to connecting accelerators within the same server node. NVIDIA uses NVLink for this purpose. NVLink is a proprietary, high-bandwidth interface that connects GPUs directly. In a standard Hopper node, eight GPUs connect via NVLink switches, providing 1.8 TB/s of bidirectional bandwidth per GPU. This allows the cluster of GPUs to share memory pools with minimal latency, behaving like a single massive GPU.
Scale-Out Interconnects
Scale-out refers to connecting different server nodes across a data center network. This scaling path relies on InfiniBand or Ultra Ethernet.
InfiniBand is the traditional choice for high-performance computing (HPC). It uses credit-based flow control to prevent packet loss, ensuring low latency and high throughput. However, InfiniBand hardware is expensive and relies on proprietary designs.
The industry is developing Ultra Ethernet as an alternative. Ultra Ethernet Consortium (UEC) modifies standard Ethernet to handle AI workloads. Standard Ethernet uses TCP/IP, which handles packet loss by retransmitting data. This introduces latency spikes (called tail latency) that stall AI training runs. Ultra Ethernet uses packet striping, multi-path routing, and selective retransmission to achieve low latency over standard physical infrastructure.
When scaling clusters, software engineers use different parallelization strategies to manage bandwidth limits:
- Tensor Parallelism: Splits individual layer calculations across multiple GPUs. This requires high-bandwidth scale-up interconnects (like NVLink) because GPUs must exchange intermediate activation states multiple times per layer.
- Pipeline Parallelism: Splits layers sequentially across GPUs. GPU 1 computes layers 1-10, then passes the output to GPU 2 for layers 11-20. This requires lower bandwidth but introduces idle periods (bubbles) where GPUs wait for upstream data.
- Data Parallelism: Replicates the entire model on every GPU, processing different batches of data. The GPUs must average their calculated gradients at the end of each step, which requires efficient scale-out networking.
Future Trends: Optical Interconnects and Analog Compute
As electrical wires hit physical limits, chip designers are turning to optical interconnects. Copper wires suffer from high resistance and signal degradation at high frequencies. Optical fibers use light to transmit data, offering higher bandwidth, lower latency, and lower power consumption over longer distances. Integrating silicon photonics directly onto the compute package allows chips to communicate optically, bypassing copper traces.
Another emerging field is Analog In-Memory Compute (AIMC). Instead of moving data from memory to compute units, AIMC performs calculations directly inside the memory array. By applying voltages to non-volatile memory cells (such as Phase Change Memory or Resistive RAM), the hardware uses Ohm's Law and Kirchhoff's Current Law to perform matrix multiplication instantly. This approach eliminates data movement entirely, reducing energy consumption by orders of magnitude. However, analog compute suffers from noise, precision loss, and manufacturing variations, limiting its use to low-precision edge AI applications.
