GPUs prioritize throughput over latency. While a CPU (like those powering modern ARM cloud data centers) uses complex branch predictors and large caches to execute a single thread as fast as possible, a GPU relies on massive parallelism. If a memory read takes 400 clock cycles, the GPU does not stall its execution pipeline. It switches execution to another thread group.
To understand this process, you must look at the execution unit: the warp (NVIDIA) or wavefront (AMD, which is also investing in alternative architectures like etching models in silicon). A warp consists of 32 threads executing the same instruction in lockstep. When an instruction requires data from global memory, the GPU initiates a hardware process that spans the execution core, the cache hierarchy, the routing network, and the physical memory interface.
The Warp Scheduler and Scoreboard
At the core of the Streaming Multiprocessor (SM) is the warp scheduler. Each SM contains multiple schedulers, and each scheduler manages a pool of warps. Every cycle, a scheduler selects an eligible warp and issues an instruction to the execution units.
An eligible warp is one that has all its dependencies resolved. To track these dependencies, the SM uses a hardware structure called a scoreboard. The scoreboard tracks register usage. When an instruction is issued, the scoreboard marks the target registers as pending.
When a global memory load instruction, such as LDG, is decoded, the warp scheduler checks the scoreboard. If the source registers are ready, the instruction is sent to the Load/Store (LDST) unit. The scoreboard immediately marks the destination registers as busy.
Because global memory access takes hundreds of cycles, the scheduler marks this warp as blocked. The scheduler then searches its pool for another warp whose next instruction has no pending registers in the scoreboard. The context switch is instantaneous because each warp has its own dedicated registers in the physical register file. No register state needs to be saved to or restored from external memory.
The scoreboard manages Read-After-Write (RAW) and Write-After-Write (WAW) hazards. For memory reads, the latency is variable. The register remains locked in the scoreboard until the data returns from the memory subsystem and is written to the register file.
Address Generation and the Coalescing Unit
Once the LDST unit receives the load instruction, it must resolve the target memory addresses. Each of the 32 threads in the warp calculates a 64-bit virtual memory address. These calculations happen in parallel using the thread's local registers.
These 32 virtual addresses are sent to the Coalescing Unit. The Coalescing Unit is a dedicated hardware block within the LDST unit. Its job is to group these 32 individual addresses into the fewest possible physical memory transactions.
Memory transactions between the SM and the cache hierarchy occur in fixed-size segments, typically 32-byte, 64-byte, or 128-byte blocks. The Coalescing Unit analyzes the address range of the active threads.
The coalescing process depends on the size of the data type requested by the instruction. For example, if a warp executes a 32-bit load (LDG.E.32), each thread requests 4 bytes of data. The total data requested by the 32 threads is 128 bytes. If these addresses are contiguous and aligned to a 128-byte boundary, the coalescing unit generates a single 128-byte transaction.
If the warp executes a 64-bit load (LDG.E.64), each thread requests 8 bytes. The total data is 256 bytes. The coalescing unit must split this into two 128-byte transactions, even if the addresses are perfectly contiguous.
If the threads access memory with a stride, where thread 0 accesses address addr, thread 1 accesses addr + 8, and thread 2 accesses addr + 16, the coalescing unit calculates the span of the addresses. If the span covers multiple 128-byte cache lines, the hardware issues multiple requests. This increases the load on the L1-to-L2 interconnect.
If the threads access non-contiguous memory, or if the alignment is offset, the Coalescing Unit must generate multiple transactions. If thread 0 reads address 0x1000 and thread 1 reads address 0x2000, the hardware cannot merge them. The Coalescing Unit splits the request into multiple independent memory transactions. This is memory divergence. It reduces effective memory bandwidth because the hardware must fetch full cache lines even if only a fraction of the data is used by the threads.
The L1 Cache and Miss Status Holding Registers (MSHR)
The coalesced memory transactions are sent to the L1 Data Cache. The L1 cache is local to the SM and operates at high clock frequencies.
The L1 cache controller looks up the requested address tags. If the data is present, which is an L1 hit, the controller reads the data from the cache storage and sends it to the register file.
If the data is missing, which is an L1 miss, the request must be forwarded to the L2 cache. To manage this without blocking the SM pipeline, the L1 cache uses Miss Status Holding Registers (MSHRs).
An MSHR is a table that tracks outstanding cache misses. When an L1 miss occurs, the controller allocates an entry in the MSHR. This entry stores the target physical address, the destination register ID, the warp ID, and a bitmask indicating which threads in the warp need the data.
If another warp subsequently requests the same memory block while the first request is still in flight, the L1 controller detects this. It does not send a duplicate request to the L2 cache. Instead, it merges the new request into the existing MSHR entry.
The MSHR acts as a deduplication table. If the MSHR table becomes full because too many unique memory requests are in flight, the SM cannot issue any more memory instructions. The pipeline stalls until an outstanding transaction returns and frees an MSHR slot.
The Crossbar Interconnect and Address Hashing
Requests that miss the L1 cache must travel to the L2 cache. Unlike the L1 cache, which is private to each SM, the L2 cache is a shared resource. The L2 cache is divided into multiple independent partitions, or slices, distributed across the GPU die.
To route requests from the SMs to the L2 partitions, the GPU uses a high-bandwidth routing network called the Crossbar Interconnect (XBar).
Before a request enters the XBar, the physical address is processed by an address hashing function. This function maps the physical address to a specific L2 partition. The hashing algorithm is designed to distribute sequential memory addresses across all available L2 partitions. This prevents a situation where multiple SMs attempt to access the same L2 partition at the same time, which would cause congestion.
The XBar uses a credit-based flow control mechanism to manage traffic. Each L2 partition has an input buffer queue. The partition sends credits to the SMs based on the available slots in its queue. An SM can only transmit a request if it has a credit for the target L2 partition. This prevents buffer overflows and limits latency spikes within the routing fabric.
This routing can lead to port conflicts. A port conflict occurs when multiple SMs send requests to the same L2 partition in the same clock cycle. The XBar arbiters resolve this using round-robin or priority-based scheduling. The winning request goes through, while the losing requests are buffered. If the buffers fill up, backpressure propagates to the SMs, slowing down instruction issue rates.
This bottleneck is called partition camping. It happens when an application's memory access pattern maps many concurrent requests to a single L2 partition, leaving other partitions idle.
L2 Cache Processing and Memory Controllers
Once the request traverses the XBar, it arrives at the target L2 partition. The L2 cache controller performs a tag lookup.
If the request hits in the L2 cache, the data is retrieved and sent back through the XBar to the requesting SM.
If the request misses in the L2 cache, it must be fetched from external DRAM or HBM. Each L2 partition is directly connected to a memory controller channel. The L2 controller forwards the miss request to its associated memory controller.
The memory controller manages the physical interface to the memory chips. It contains a command queue and scheduling logic designed to optimize physical memory access.
The controller translates the flat physical address into hardware-specific coordinates: Channel, Rank, Bank Group, Bank, Row, and Column.
Memory Controller Scheduling and DRAM Mechanics
The memory controller's primary goal is to maximize the utilization of the data bus. Physical memory chips have strict timing constraints. Switching between reading and writing, or opening and closing rows, introduces idle cycles on the data bus.
The controller's command scheduler reorders requests in the command queue to minimize these idle cycles. It tracks physical timing parameters such as Row Cycle Time (tRC), Row-to-Column Delay (tRCD), and Precharge Time (tRP).
Modern memory systems like GDDR6 and HBM use Bank Groups to increase data transfer rates. A DRAM chip contains multiple banks grouped together.
Accessing banks within the same bank group requires a longer delay between commands due to shared internal data paths. Accessing banks in different bank groups allows the controller to overlap commands, reducing the delay.
The memory controller scheduler tracks which bank groups are active. It prioritizes requests that alternate between different bank groups. This scheduling optimization helps keep the physical data bus saturated.
The scheduler uses page management policies to handle rows in the DRAM banks. A DRAM bank is organized into rows and columns. To read data, a row must be copied into a temporary storage area called the Row Buffer. This is done using an Activate (ACT) command.
Under an Open Page policy, the controller keeps the row open in the Row Buffer after a read operation. If the next request in the queue targets the same row, the controller can read the data immediately using a Column Address Strobe (CAS) command. This is a Row Buffer Hit.
If the next request targets a different row in the same bank, it is a Row Buffer Conflict. The controller must close the current row using a Precharge (PRE) command, wait for the precharge time (tRP), open the new row using an Activate (ACT) command, wait for the activation time (tRCD), and then issue the CAS command. This sequence adds significant latency.
The memory controller scheduler attempts to group requests targeting the same DRAM rows together, even if they arrived out of order, to maximize Row Buffer Hits.
The Physical Interface and Data Return
Once the scheduler determines the optimal command sequence, it sends the commands across the physical interface (PHY) to the memory devices.
In systems using GDDR6, the interface uses high-speed differential signaling. GDDR6 transfers data on both the rising and falling edges of the clock. It also uses techniques like Write Data Link Training and Command/Address parity to ensure signal integrity at high frequencies.
In systems using High Bandwidth Memory (HBM), the GPU is connected to the memory stacks via a silicon interposer. HBM uses a very wide interface, typically 1024 bits per stack, running at lower clock speeds than GDDR6. This wide bus allows HBM to transfer large amounts of data in parallel without the high power consumption associated with high-frequency GDDR6 signaling.
The DRAM chip retrieves the data from the storage cells, places it in the output buffers, and drives it onto the data bus. The data travels back across the PHY to the memory controller.
The Return Path and Warp Rescheduling
The returning data follows the reverse path back to the execution units.
First, the memory controller receives the data and forwards it to the associated L2 partition. The L2 cache updates its data arrays and marks the cache line as valid.
Next, the L2 controller sends the data across the Crossbar Interconnect to the SM that initiated the request.
The data arrives at the SM's L1 cache controller. The L1 cache updates its data array.
The L1 controller uses the transaction ID to look up the corresponding entry in the MSHR.
The MSHR retrieves the metadata for the outstanding request: the destination register ID, the warp ID, and the active thread mask.
The LDST unit writes the data directly into the SM's physical Register File.
Once the write-back is complete, the LDST unit signals the dependency scoreboard. The scoreboard clears the pending flag for the target register.
The warp scheduler detects that the register is no longer blocked. It changes the status of the warp from blocked to eligible.
