How much VRAM do you need to run an LLM?

At 4-bit quantization (Q4_K_M), a model needs about 0.6 GB of VRAM per billion parameters, plus 1 to 2 GB for the KV cache and runtime overhead at a short context of 4 to 8K tokens.

• 8B model: about 5 to 7 GB
• 24B model: about 15 to 17 GB
• 32B model: about 20 to 22 GB
• 70B model: about 42 to 45 GB, beyond a single 24 GB card

Mixture-of-experts models need memory for all their weights, not only the active ones: Qwen3 30B A3B activates 3B parameters per token but still needs about 20 GB.

Figures are for short contexts. Long prompts and many concurrent requests grow the KV cache, so keep 10 to 20% headroom.

The KV cache grows linearly with the number of tokens in flight. A 32B model that fits in 22 GB at 4K tokens can run out of memory at 32K tokens, or with eight long requests at once. If you need long contexts on a 24 GB card, prefer a 24B or MoE model, or a smaller quantization.

• 8 GB (RTX 4060 Ti): 8 to 9B models, embeddings, vision at 8B.
• 16 GB (RTX 4070 Ti Super): gpt-oss 20B, Mistral Small 24B with care.
• 24 GB (RTX 3090 / 4090, L4, RTX PRO 4000 Blackwell in Hetzner's GEX45): every 32B model in the catalog.
• 96 GB (RTX PRO 6000 Blackwell): 70B models and long contexts.

gpuos shows a VRAM gauge with your node's real memory before every deployment, so you see the fit before anything downloads.