LLM Roofline
NVIDIA · PCIE · Backend: vllm

NVIDIA RTX 5090 (32 GB) — LLM Sizing & Speed Matrix

GDDR7 512-bit bus @ 1,792 GB/s; single-card speed king for 24B-32B models and massive prefill.

Reference Pricing
$2,499 Street
Usable Memory
31 GiB
Memory Bandwidth
1792 GB/s (82% eff.)
FP16 Compute
209.5 TFLOPS
Roofline Ridge
92.7 FLOP/B
Power (Load / Idle)
575W / 45W

Buy or Rent NVIDIA RTX 5090 (32 GB)

Vendor & Cloud Links
Deploy on Vast.ai Spot GPU
Vast.ai · On-Demand / Spot GPU
↗
Launch Pod on RunPod
RunPod · On-Demand / Spot GPU
↗
RTX 5090 32 GB on Amazon
Amazon · Direct Hardware
↗
NVIDIA RTX GPUs at B&H Photo
B&H Photo · Direct Hardware
↗

All 32 Open-Weight Models on NVIDIA RTX 5090 (32 GB) (Q4_K_M @ 32k)

Model Status Weights Total @ 32k Decode TTFT (4k) Max Safe Ctx
GLM-4 32B 0414 COMFORTABLE 18.6 GiB 22.1 GiB 66.7 tok/s 2.07s 32k
Qwen3 30B-A3B COMFORTABLE 17.6 GiB 21.6 GiB 245.8 tok/s 0.28s 40k
Gemma 3 27B IT COMFORTABLE 15.8 GiB 20.2 GiB 73.2 tok/s 1.75s 128k
Mistral Small 3.1 24B COMFORTABLE 13.8 GiB 20.3 GiB 72.7 tok/s 1.52s 101k
Codestral 22B COMFORTABLE 12.8 GiB 21.4 GiB 69.2 tok/s 1.46s 32k
Qwen3 14B COMFORTABLE 8.5 GiB 15.0 GiB 101.2 tok/s 0.98s 40k
Qwen2.5 14B Instruct COMFORTABLE 8.5 GiB 16.0 GiB 94.2 tok/s 0.99s 32k
DeepSeek-R1 Distill Qwen 14B COMFORTABLE 8.5 GiB 16.0 GiB 94.2 tok/s 0.99s 112k
Phi-4 14B COMFORTABLE 8.5 GiB 13.0 GiB 118.1 tok/s 0.97s 16k
Gemma 3 12B IT COMFORTABLE 7.0 GiB 10.6 GiB 146.6 tok/s 0.82s 128k
Qwen3 8B COMFORTABLE 4.7 GiB 10.5 GiB 148.4 tok/s 0.56s 40k
Llama 3.1 8B Instruct COMFORTABLE 4.6 GiB 9.9 GiB 158.7 tok/s 0.55s 128k
Qwen2.5 7B Instruct COMFORTABLE 4.4 GiB 7.4 GiB 223.4 tok/s 0.52s 32k
Gemma 3 4B IT COMFORTABLE 2.5 GiB 4.4 GiB 410.3 tok/s 0.31s 128k
Qwen3 32B TIGHT 18.9 GiB 28.3 GiB 50.9 tok/s 2.14s 40k
Qwen2.5 Coder 32B TIGHT 18.9 GiB 28.3 GiB 50.9 tok/s 2.09s 32k
DeepSeek-R1 Distill Qwen 32B TIGHT 18.9 GiB 28.3 GiB 50.9 tok/s 2.09s 43k
QwQ 32B Reasoning TIGHT 18.9 GiB 28.3 GiB 50.9 tok/s 2.09s 40k
DeepSeek-R1 671B OOM 386.4 GiB 390.3 GiB OOM — OOM
DeepSeek-V3 671B OOM 386.4 GiB 390.3 GiB OOM — OOM
Llama 3.1 405B Instruct OOM 233.8 GiB 252.5 GiB OOM — OOM
Llama 4 Maverick 400B-A17B OOM 231.5 GiB 239.0 GiB OOM — OOM
Qwen3 235B-A22B OOM 135.4 GiB 142.6 GiB OOM — OOM
Mixtral 8x22B Instruct OOM 81.2 GiB 89.8 GiB OOM — OOM
Command A 111B OOM 63.9 GiB 69.1 GiB OOM — OOM
Llama 4 Scout 109B-A17B OOM 62.8 GiB 70.2 GiB OOM — OOM
Qwen2.5 72B Instruct OOM 41.9 GiB 53.7 GiB OOM — OOM
DeepSeek-R1 Distill Llama 70B OOM 40.7 GiB 52.5 GiB OOM — OOM
Llama 3.3 70B Instruct OOM 40.7 GiB 52.5 GiB OOM — OOM
Llama 3.1 70B Instruct OOM 40.7 GiB 52.5 GiB OOM — OOM
Llama-3.3 Nemotron Super 49B OOM 28.7 GiB 36.7 GiB OOM — 5k
Mixtral 8x7B Instruct OOM 26.9 GiB 32.2 GiB OOM — 22k

Interactive NVIDIA RTX 5090 (32 GB) Simulator

32.3B Dense · GQA
1792 GB/s · 209.5 TFLOPS
3. Weight Quantization 4.85 bpw
4. KV Cache
32,768 tok
Compute Roofline
COMFORTABLE FIT Memory-Bandwidth Bound
Max Safe Ctx: 32k
Memory Allocation Footprint 22.10 / 31.00 GiB usable
Weights: 18.60 GiB KV Cache: 1.91 GiB Overhead: 1.59 GiB Headroom: 8.90 GiB
Decode Speed
66.7
tok/s (batch=1)
Prefill Speed
2012
prompt tok/s
TTFT Latency
2.07s
for 4,096 tok
Active Params
32.3B
of 32.3B total
llama.cpp Server Command 
llama-server -hf zai-org/GLM-4-32B-0414:Q4_K_M -ngl 99 -fa on -c 32768 --port 8080
Context KV Cache Total VRAM Decode Status
4k 0.24 GiB 20.01 GiB 72.6 tok/s COMFORTABLE
8k 0.48 GiB 20.67 GiB 71.7 tok/s COMFORTABLE
16k 0.95 GiB 21.15 GiB 70.0 tok/s COMFORTABLE
32k 1.91 GiB 22.10 GiB 66.7 tok/s COMFORTABLE

Buy-vs-Cloud-vs-API Monthly TCO Break-Even

USE SERVERLESS API
Own Hardware/mo
$57.65
69.3h active + power
Cloud GPU/mo
$70.97
@$0.89/hr spot/pod
Serverless API/mo
$9.50
API cheaper

Recommended Hardware & Cloud Deployment

Contextual to fit status

Hardware FAQ — NVIDIA RTX 5090 (32 GB)

How much usable VRAM / GPU memory does NVIDIA RTX 5090 (32 GB) actually provide for LLMs?

NVIDIA RTX 5090 (32 GB) provides 31 GiB usable VRAM after subtracting 1.0 GiB driver/display overhead.

What is the largest Q4_K_M model NVIDIA RTX 5090 (32 GB) can run at 32k context?

With Q4_K_M weights and FP16 KV cache at 32k context, the largest fitting model in our verified catalog on NVIDIA RTX 5090 (32 GB) is GLM-4 32B 0414 (32.3B total).