Local Inference

AirLLM: Running 70B Parameter Models on a Single 4GB GPU

AirLLM claims you can run 70B models on consumer GPUs with just 4GB VRAM. Here's how it works, where it breaks, and whether it's actually useful for real workloads.