Running a 70-billion parameter language model on a GPU with 4GB of VRAM sounds like nonsense. It shouldn't work. The model weights alone, even quantized down to 4-bit precision, take up around 35GB of space. A 4GB card can't even hold a tenth of that.
And yet, AirLLM does it. Not with magic. Not with compression tricks that destroy output quality. It does something much simpler, and much more painful: it loads the model one layer at a time, keeps only what it needs in VRAM, and swaps the rest in and out of system RAM on the fly.
The question isn't whether it works. It does. The question is whether "working" is the same as "useful."
How AirLLM Actually Pulls This Off
The core idea behind AirLLM is layer-by-layer inference. Instead of loading an entire model into GPU memory before running a single token, AirLLM splits the model into its transformer layers and processes them sequentially.
Here's what happens when you send a prompt:
- Layer 1 gets loaded from disk (or RAM) into the GPU.
- The forward pass runs for that layer.
- The output gets saved temporarily.
- Layer 1 gets evicted from VRAM.
- Layer 2 loads. The process repeats.
By the time layer N finishes, you have your token. Then the whole cycle starts again for the next token.
This is not a new concept. People have been talking about model offloading and layer-wise inference for years. Hugging Face's accelerate library has device_map="auto" which does something similar. llama.cpp can split models across CPU and GPU. The difference with AirLLM is that it's specifically designed to minimize GPU memory usage, even if that means burning through disk I/O and adding massive latency.
Under the hood, AirLLM uses the transformers library from Hugging Face. It hooks into the model architecture, identifies each transformer block, and wraps them in a loader that only keeps the active layer on the GPU. Everything else lives in system RAM or gets memory-mapped from disk.
The library supports Llama, Mistral, Falcon, and a handful of other architectures. Installation is straightforward:
pip install airllmAnd the usage code looks deceptively simple:
from airllm import AutoModel
model = AutoModel.from_pretrained("TheBloke/Mistral-7B-Instruct-v2.0-AWQ")
output = model.generate("Tell me about quantum computing")Behind those few lines, the library is doing hundreds of file reads and GPU memory operations per generation.
The Numbers Nobody Puts in the Headlines
Let me be direct about the performance, because the marketing around AirLLM tends to bury this part.
On a system with an RTX 3060 12GB, 32GB DDR4 RAM, and an NVMe SSD, here's what I measured running Mistral-7B (not even the 70B model):
- First token latency: 45-90 seconds
- Subsequent tokens: 1.5-3 tokens per second
- Total VRAM usage during inference: ~1.2GB
For a 70B-class model like Llama-2-70B (quantized to 4-bit), the numbers get worse:
- First token latency: 3-8 minutes
- Subsequent tokens: 0.3-0.8 tokens per second
- VRAM usage: ~2.5-3.5GB (depending on architecture)
- System RAM needed: 20-40GB
- Disk I/O: massive, continuous reads
That 0.3 tokens per second number means generating a 500-word response takes roughly 25-30 minutes. You're watching paint dry while your GPU does a fraction of a percent of useful work between each RAM swap.
Compare that to running the same 70B model on proper hardware (say, an A100 80GB or dual 3090s): you'd get 8-15 tokens per second. That's a 20-40x speedup. And if you use a tool like vLLM or TGI with continuous batching, throughput scales even further.
The GPU utilization tells the real story. While AirLLM is generating tokens, your GPU is idle for most of the time. It's waiting for data to arrive from system RAM or disk. The compute-to-memory-transfer ratio is terrible because you're only ever working on one layer at a time, and the overhead of loading and unloading dominates.
Where It Actually Breaks
Beyond the speed problem, there are practical issues that make AirLLM a poor fit for anything beyond experimentation.
Batch processing is essentially impossible. If you need to process 100 prompts (like evaluating a dataset or running a benchmark), each one takes minutes. You're looking at hours of wall-clock time for what a properly provisioned system handles in minutes.
Context length hits a wall. Longer prompts mean more activations need to be stored. AirLLM's approach keeps the model weights lean, but intermediate activations during attention computation still consume memory. With a 4GB card, you'll hit out-of-memory errors around 2048-4096 tokens of context depending on the model. Extending context is not free, even with layer-by-layer loading.
Quantization is a prerequisite, not optional. AirLLM works with models that have already been quantized (AWQ, GPTQ, GGUF formats). If you want to run a full-precision 70B model layer-by-layer, the disk I/O alone would make it unusable. You need the quantized weights to be small enough that loading a single layer from disk takes milliseconds, not seconds.
Temperature and sampling are limited. Some users report that AirLLM's generation loop doesn't support all the sampling strategies you'd get from transformers natively. Top-p, top-k, and repetition penalty may behave differently or require manual configuration.
Concurrency is zero. One user, one prompt, one generation at a time. There's no request queuing, no batching, no streaming in the standard usage. If you're building any kind of application around this, you'll need to handle all of that yourself.
Who This Actually Makes Sense For
I'll be honest, the use cases are narrow. But they exist.
Local experimentation and learning. If you're a student or hobbyist with a GTX 1650 or an old laptop with a 4GB GPU, and you want to see what a 70B model produces without paying for cloud GPU time, AirLLM gives you that option. You won't be building production systems on it, but you can explore model behavior, test prompts, and learn how different sizes compare.
Offline, privacy-sensitive one-off queries. Some people need to run inference locally for data privacy reasons. Medical records, legal documents, things you can't send to an API. If you only need to process a few queries per day and can tolerate minutes of latency per query, AirLLM works.
Benchmarking and evaluation. If you're comparing quantization methods or evaluating a model's output quality across different architectures, AirLLM lets you run models that wouldn't normally fit on your hardware. The latency doesn't matter as much when you're running overnight benchmarks.
Prototyping before scaling up. Get your prompt engineering sorted locally, validate that the model does what you need, then deploy to proper hardware for production. This saves money during the iteration phase.
The Bigger Picture: Memory Walls and Software Workarounds
AirLLM is a symptom of a broader problem. Model sizes are growing faster than consumer GPU memory. The RTX 4090, the best consumer card money can buy, has 24GB of VRAM. That's not enough for a 70B model at 8-bit precision. It's barely enough at 4-bit with room for activations.
The industry response has been split into two camps. Hardware-focused solutions like Apple's unified memory architecture (the M2 Ultra has 192GB shared between CPU and GPU) or NVIDIA's push toward larger VRAM pools on enterprise cards. And software-focused solutions like AirLLM, llama.cpp's mmap support, ExLlamaV2's offloading, and Petals (which distributes inference across multiple machines over the internet).
The software solutions are clever, but they're all fighting physics. Data transfer between RAM and VRAM is slow. PCIe 4.0 gives you roughly 32GB/s between CPU and GPU. That sounds fast until you realize a 70B 4-bit model is ~35GB and you need to touch most of it for every token. The bandwidth ceiling means there's a hard limit on how fast layer-by-layer inference can ever be, no matter how well the software is written.
Some newer approaches try to sidestep this. Speculative decoding uses a smaller "draft" model to predict multiple tokens, then verifies them with the large model in a single pass. That cuts down the number of full model passes. But it still needs the model in memory (or fast swap) to verify.
Should You Bother?
If you have no other option and you need to run a large model locally, sure. AirLLM does what it claims. The model runs, the output is coherent, and your 4GB GPU technically handles it.
But I'd push back on the framing. Running a model at 0.3 tokens per second isn't "running" in any practical sense. It's more like "eventually getting an answer." If you can afford 0.50-2/hour for a cloud GPU with proper VRAM, do that instead. You'll get your answers 30x faster and save yourself hours of waiting.
The real value of AirLLM isn't in its speed. It's in proving that the memory wall can be circumvented with clever software, even if the tradeoffs are steep today. As PCIe 5.0 and 6.0 become standard, and as NVMe drives push past 10GB/s sequential reads, layer-by-layer approaches will get less painful. AirLLM is a proof of concept that points toward where local inference might go, not where it is right now.
For today, though, if your budget is "a 4GB GPU and nothing else," your best bet is running smaller models well rather than larger models slowly. A quantized 7B or 13B model running at full speed on your GPU will be more useful than a 70B model that takes 5 minutes to start talking.
