LLM Roofline
NVIDIA · PCIE · Backend: vllm

NVIDIA RTX PRO 6000 Blackwell (96 GB) — LLM Sizing & Speed Matrix

Single-slot/workstation 96 GB GDDR7 ECC powerhouse; fits 70B Q8_0 + 128k context in one card.

Reference Pricing
$8,499 Street
Usable Memory
95 GiB
Memory Bandwidth
1792 GB/s (84% eff.)
FP16 Compute
250 TFLOPS
Roofline Ridge
112.9 FLOP/B
Power (Load / Idle)
600W / 45W

Buy or Rent NVIDIA RTX PRO 6000 Blackwell (96 GB)

Vendor & Cloud Links
Launch Pod on RunPod
RunPod · On-Demand / Spot GPU
↗
Deploy on Vast.ai Spot GPU
Vast.ai · On-Demand / Spot GPU
↗
NVIDIA Workstation GPUs at B&H
B&H Photo · Direct Hardware
↗

All 32 Open-Weight Models on NVIDIA RTX PRO 6000 Blackwell (96 GB) (Q4_K_M @ 32k)

Model Status Weights Total @ 32k Decode TTFT (4k) Max Safe Ctx
Command A 111B COMFORTABLE 63.9 GiB 69.1 GiB 21.0 tok/s 5.54s 256k
Llama 4 Scout 109B-A17B COMFORTABLE 62.8 GiB 70.2 GiB 78.1 tok/s 0.90s 164k
Qwen2.5 72B Instruct COMFORTABLE 41.9 GiB 53.7 GiB 27.0 tok/s 3.67s 32k
DeepSeek-R1 Distill Llama 70B COMFORTABLE 40.7 GiB 52.5 GiB 27.7 tok/s 3.57s 128k
Llama 3.3 70B Instruct COMFORTABLE 40.7 GiB 52.5 GiB 27.7 tok/s 3.57s 128k
Llama 3.1 70B Instruct COMFORTABLE 40.7 GiB 52.5 GiB 27.7 tok/s 3.57s 128k
Llama-3.3 Nemotron Super 49B COMFORTABLE 28.7 GiB 36.7 GiB 40.2 tok/s 2.52s 128k
Mixtral 8x7B Instruct COMFORTABLE 26.9 GiB 32.2 GiB 107.9 tok/s 0.68s 32k
Qwen3 32B COMFORTABLE 18.9 GiB 28.3 GiB 52.1 tok/s 1.72s 40k
Qwen2.5 Coder 32B COMFORTABLE 18.9 GiB 28.3 GiB 52.1 tok/s 1.68s 32k
DeepSeek-R1 Distill Qwen 32B COMFORTABLE 18.9 GiB 28.3 GiB 52.1 tok/s 1.68s 128k
QwQ 32B Reasoning COMFORTABLE 18.9 GiB 28.3 GiB 52.1 tok/s 1.68s 40k
GLM-4 32B 0414 COMFORTABLE 18.6 GiB 22.1 GiB 68.4 tok/s 1.67s 32k
Qwen3 30B-A3B COMFORTABLE 17.6 GiB 21.6 GiB 251.7 tok/s 0.23s 40k
Gemma 3 27B IT COMFORTABLE 15.8 GiB 20.2 GiB 75.0 tok/s 1.41s 128k
Mistral Small 3.1 24B COMFORTABLE 13.8 GiB 20.3 GiB 74.5 tok/s 1.22s 128k
Codestral 22B COMFORTABLE 12.8 GiB 21.4 GiB 70.9 tok/s 1.17s 32k
Qwen3 14B COMFORTABLE 8.5 GiB 15.0 GiB 103.7 tok/s 0.79s 40k
Qwen2.5 14B Instruct COMFORTABLE 8.5 GiB 16.0 GiB 96.5 tok/s 0.80s 32k
DeepSeek-R1 Distill Qwen 14B COMFORTABLE 8.5 GiB 16.0 GiB 96.5 tok/s 0.80s 128k
Phi-4 14B COMFORTABLE 8.5 GiB 13.0 GiB 121.0 tok/s 0.78s 16k
Gemma 3 12B IT COMFORTABLE 7.0 GiB 10.6 GiB 150.1 tok/s 0.66s 128k
Qwen3 8B COMFORTABLE 4.7 GiB 10.5 GiB 152.0 tok/s 0.46s 40k
Llama 3.1 8B Instruct COMFORTABLE 4.6 GiB 9.9 GiB 162.6 tok/s 0.45s 128k
Qwen2.5 7B Instruct COMFORTABLE 4.4 GiB 7.4 GiB 228.8 tok/s 0.42s 32k
Gemma 3 4B IT COMFORTABLE 2.5 GiB 4.4 GiB 420.3 tok/s 0.26s 128k
Mixtral 8x22B Instruct TIGHT 81.2 GiB 89.8 GiB 41.9 tok/s 1.98s 56k
DeepSeek-R1 671B OOM 386.4 GiB 390.3 GiB OOM — OOM
DeepSeek-V3 671B OOM 386.4 GiB 390.3 GiB OOM — OOM
Llama 3.1 405B Instruct OOM 233.8 GiB 252.5 GiB OOM — OOM
Llama 4 Maverick 400B-A17B OOM 231.5 GiB 239.0 GiB OOM — OOM
Qwen3 235B-A22B OOM 135.4 GiB 142.6 GiB OOM — OOM

Interactive NVIDIA RTX PRO 6000 Blackwell (96 GB) Simulator

111B Dense · GQA
1792 GB/s · 250 TFLOPS
3. Weight Quantization 4.85 bpw
4. KV Cache
32,768 tok
Compute Roofline
COMFORTABLE FIT Memory-Bandwidth Bound
Max Safe Ctx: 256k
Memory Allocation Footprint 69.11 / 95.00 GiB usable
Weights: 63.93 GiB KV Cache: 2.75 GiB Overhead: 2.44 GiB Headroom: 25.89 GiB
Decode Speed
21.0
tok/s (batch=1)
Prefill Speed
744
prompt tok/s
TTFT Latency
5.54s
for 4,096 tok
Active Params
111.0B
of 111B total
llama.cpp Server Command 
llama-server -hf CohereLabs/c4ai-command-a-03-2025:Q4_K_M -ngl 99 -fa on -c 32768 --port 8080
Context KV Cache Total VRAM Decode Status
4k 1.00 GiB 66.52 GiB 21.6 tok/s COMFORTABLE
8k 1.25 GiB 67.61 GiB 21.5 tok/s COMFORTABLE
16k 1.75 GiB 68.11 GiB 21.4 tok/s COMFORTABLE
32k 2.75 GiB 69.11 GiB 21.0 tok/s COMFORTABLE
64k 4.75 GiB 71.11 GiB 20.4 tok/s COMFORTABLE
128k 8.75 GiB 75.11 GiB 19.3 tok/s COMFORTABLE

Buy-vs-Cloud-vs-API Monthly TCO Break-Even

BUY LOCAL
Own Hardware/mo
$181.02
216.8h active + power
Cloud GPU/mo
$461.23
@$1.85/hr spot/pod
Serverless API/mo
$275.00
Payback: 22.3 mo

Recommended Hardware & Cloud Deployment

Contextual to fit status

Hardware FAQ — NVIDIA RTX PRO 6000 Blackwell (96 GB)

How much usable VRAM / GPU memory does NVIDIA RTX PRO 6000 Blackwell (96 GB) actually provide for LLMs?

NVIDIA RTX PRO 6000 Blackwell (96 GB) provides 95 GiB usable VRAM after subtracting 1.0 GiB driver/display overhead.

What is the largest Q4_K_M model NVIDIA RTX PRO 6000 Blackwell (96 GB) can run at 32k context?

With Q4_K_M weights and FP16 KV cache at 32k context, the largest fitting model in our verified catalog on NVIDIA RTX PRO 6000 Blackwell (96 GB) is Command A 111B (111B total).