How Post-Training Quantization Shrinks LLMs to Run on Laptops

Running a modern large language model requires massive computational resources. When a model is trained, its weights are stored as high-precision floating-point numbers. Loading an 8-billion parameter model at full precision requires sixteen gigabytes of memory.

This memory requirement makes running models on consumer devices impossible without modifications. Most standard laptops simply do not have the unified memory needed to load the weights. This is where post-training quantization becomes essential.

By compressing the model weights after training completes, developers can significantly reduce the memory footprint. This process allows consumer hardware to run advanced models locally. Let us explore the mechanics of this compression technique.

The Memory Bandwidth Bottleneck

When executing an LLM, the primary bottleneck is not CPU or GPU compute speed. Instead, the constraint is memory bandwidth. The processor must load every single parameter from memory into the cache for every single token generated.

If a model uses 16-bit floating-point numbers (FP16), each parameter consumes two bytes. To generate text, a chip must transfer billions of bytes across the memory bus continuously. This transfer speed limits token generation rates.

By shrinking the size of each parameter, we reduce the volume of data that must travel across the bus. This optimization directly increases execution speed. It allows hardware to run AI models on laptops with limited memory.

The Core Math of Quantization

Post-training quantization works by mapping continuous floating-point values to a smaller set of discrete integer buckets. The most common targets are 8-bit (INT8) and 4-bit (INT4) representations.

Mathematically, this mapping relies on a scale factor S and a zero-point offset Z. The scale factor projects the range of floating-point values onto the integer range. The zero-point ensures that the floating-point value of zero maps exactly to an integer.

We can express the quantization formula as:

q = round(x / S) + Z

Where x is the original float weight, and q is the resulting integer. To retrieve the weight during computation, we apply the dequantization formula:

dequantized_x = S * (q - Z)

During execution, the processor loads the quantized integer weights. It dequantizes them back to floating-point numbers on the fly before performing matrix multiplications. This dynamic calculation keeps the memory footprint small while utilizing standard hardware.

Because the weights are stored at lower precision, the model consumes far less storage. An 8-billion parameter model shrinks from sixteen gigabytes down to just four gigabytes when quantized to 4-bit precision.

Walkthrough: Quantizing a Weight Tensor

To see this math in action, let us quantize a tiny weight tensor to 4-bit unsigned integers. A 4-bit integer can store values from 0 to 15. Suppose we have the following four weights:

Weights (x) = [-1.5, 0.2, 1.8, 3.0]

First, we find the minimum and maximum values in our tensor: x_min = -1.5 and x_max = 3.0. We calculate the scale factor S using the 4-bit target range (15):

S = (x_max - x_min) / 15 = (3.0 - (-1.5)) / 15 = 0.3

Next, we calculate the zero-point offset Z, which must be an integer:

Z = round(-x_min / S) = round(1.5 / 0.3) = 5

Now we apply the quantization formula to each weight:

• For x_0 = -1.5: q_0 = round(-1.5 / 0.3) + 5 = -5 + 5 = 0
• For x_1 = 0.2: q_1 = round(0.2 / 0.3) + 5 = 1 + 5 = 6
• For x_2 = 1.8: q_2 = round(1.8 / 0.3) + 5 = 6 + 5 = 11
• For x_3 = 3.0: q_3 = round(3.0 / 0.3) + 5 = 10 + 5 = 15

Our quantized 4-bit weights are [0, 6, 11, 15]. When the model runs, it dequantizes these back to floats:

• For q_0 = 0: dequantized_x_0 = 0.3 * (0 - 5) = -1.5 (Exact match)
• For q_1 = 6: dequantized_x_1 = 0.3 * (6 - 5) = 0.3 (Original was 0.2, difference of 0.1)
• For q_2 = 11: dequantized_x_2 = 0.3 * (11 - 5) = 1.8 (Exact match)
• For q_3 = 15: dequantized_x_3 = 0.3 * (15 - 5) = 3.0 (Exact match)

This approximation introduces minor errors, but it compresses the required memory storage by 75% compared to 16-bit representations.

Symmetric vs Asymmetric Mapping

Quantization implementations generally choose between symmetric and asymmetric mapping. Symmetric quantization maps the range of weights symmetrically around zero. This choice simplifies calculations because the zero-point offset is always zero.

Asymmetric quantization, on the other hand, maps the minimum and maximum weights directly to the minimum and maximum integer values. This method is more precise when the weight distribution is skewed or not centered around zero.

While asymmetric mapping reduces representation error, it introduces extra computational overhead. The processor must track the non-zero offset during dequantization. Developers select the method based on the target hardware capabilities.

Weight-Only vs Activation Quantization

In many local deployments, developers only quantize the static weights of the model. The active intermediate representations (activations) generated during processing remain in higher precision (FP16). This pattern is called weight-only quantization.

Quantizing activations is significantly harder because their dynamic ranges change with every prompt. Some activation channels contain outlier values that are much larger than the average. These outliers skew the scale factor, reducing representation accuracy.

To solve this, advanced techniques like AWQ (Activation-aware Weight Quantization) protect critical channels. AWQ identifies which weights correspond to the largest activation channels and skips quantizing those channels, preserving model accuracy.

GGUF vs GPTQ: Formats Explained

Two dominant file formats have emerged for local LLM execution. GPTQ is a quantization method designed primarily for GPU execution. It utilizes calibration data to adjust the remaining weights, minimizing the accuracy loss caused by precision reduction.

GGUF, developed by the llama.cpp community, is designed for hybrid CPU and GPU execution. It stores the model weights and structured metadata in a single file. This format allows seamless memory offloading between the system RAM and GPU VRAM.

For laptop deployments, GGUF is highly popular. It allows users to run models that exceed their GPU memory capacity by splitting the layers between the system processor and the graphics card.

Comparing Precision Formats

The table below summarizes the trade-offs of the different precision formats for an 8-billion parameter model.

As the precision level decreases, the memory requirements drop significantly. This drop allows standard laptops to host model files that would otherwise trigger out-of-memory errors.

The Cost: Accuracy and Perplexity

Reducing precision inevitably introduces quantization noise. When weights are rounded to fit into discrete buckets, the fine details of the model's representations are lost. This loss is measured using a metric called perplexity.

Higher perplexity indicates that the model is less certain about its predictions. Quantizing from FP16 to INT8 introduces almost zero measurable perplexity increase. The model performs identically to the original uncompressed version.

Stepping down to INT4 increases perplexity slightly. In practice, this degradation is minor. The model remains highly coherent and retains its core reasoning capabilities. Going below 3-bit, however, causes accuracy to degrade rapidly.

For most developers and local use cases, the minor loss in precision is an excellent trade-off. The ability to run models locally offsets the slight reduction in response quality.

Conclusion

Post-training quantization is the technology that made local AI practical. By mapping complex floats to compact integers, it bypasses the physical memory limits of consumer hardware.

As quantization algorithms improve, we will see even smaller representations with lower accuracy loss. For now, formats like GGUF and GPTQ enable developers to prototype and deploy language models directly on their personal laptops.