GPU memory for local AI — parameters, precision, and quantization
I bought a used 3090 specifically to run 13B models locally. The math for whether a given model fits on a given card is the difference between smooth inference and swapping to disk at three tokens a second.
Large language models are measured in parameters — the individual weights the network learned during training. A '7B model' has 7 billion parameters. Each parameter is a number, and the amount of VRAM the model consumes depends on how many bytes are used to store each number.
At full 32-bit precision (fp32), each parameter is 4 bytes. At half-precision fp16 or bf16, each is 2 bytes. Modern inference almost always uses fp16/bf16 as the baseline, because the quality loss versus fp32 is negligible for most uses and memory drops in half.
So the baseline rule: model size in billions × 2 = VRAM required in GB, for fp16 inference of the weights alone. A 7B model is ~14 GB, a 13B model is ~26 GB, a 70B model is ~140 GB.
What quantization does
Quantization stores each weight in fewer bits by rounding to a smaller set of allowed values. 8-bit quantization uses 1 byte per weight, cutting memory in half again versus fp16. 4-bit quantization uses half a byte per weight, cutting it in half once more.
So a 7B model at 4-bit is ~3.5 GB, a 13B is ~7 GB, a 70B is ~35 GB. This is why 4-bit quantization is the difference between running Llama-70B on two consumer cards versus needing datacenter hardware.
The quality cost of quantization is smaller than you'd expect for modern quant methods (GPTQ, AWQ, GGUF Q4_K_M). 4-bit typically retains 95%+ of the fp16 model's benchmark scores. 3-bit and below start to visibly degrade. 2-bit is a curiosity for the largest models where even a lossy version beats nothing.
Context and KV cache
The weights aren't the only thing in VRAM. The KV cache — the running state of the conversation — grows with context length and with the model's hidden dimension. For a 13B model with 4K context, budget another 1-2 GB. For 32K context, budget 8-16 GB more. This is the sneaky reason a model that 'fits' at short context blows past your VRAM at long context.
What fits where
8 GB card (RTX 3070, 4060): 7B models at 4-bit comfortably, 13B at 4-bit with small context.
12 GB card (RTX 3060 12GB, 4070): 13B at 4-bit with decent context. 7B at 8-bit.
16 GB card (RTX 4060 Ti 16GB, 4080 Super, Apple M-series unified memory 16 GB): 13B at 8-bit, 30-34B at 4-bit with small context.
24 GB card (RTX 3090, 4090): 34B at 4-bit with room, 13B at fp16, 70B at 2-3 bit.
48 GB (RTX 6000 Ada, dual 3090): 70B at 4-bit with real context.
80 GB (H100, A100): 70B at fp16.
Apple Silicon unified memory does this differently — the whole system RAM is available to the GPU, so a 128 GB Mac Studio can run 70B fp16 or Llama-405B at 3-bit, slowly. Bandwidth is lower than a dedicated GPU so token speed is lower, but 'runs at all' beats 'fits nowhere' for big models.
Try the converters mentioned in this article
FAQ
Is 1 GB in VRAM the same as 1 GB on disk?+
Very close. Model files (GGUF, safetensors) are compressed slightly on disk but expand to roughly their listed size in memory. Add ~10% overhead for framework buffers.
Why is dual-GPU not always 2× the VRAM?+
Because splitting a model across cards requires the framework to shard it, which adds inter-GPU communication overhead. Tensor parallelism works well within a machine over NVLink, less well over PCIe.
This article was written by The Turbo Unit Converter engineering desk (Applied metrology & engineering units) and last reviewed on 2026-07-22 against NIST SP 811 and the BIPM SI Brochure. Read our full editorial policy.