LLM Roofline
NVIDIA Cloud · SXM · Backend: vllm

Cloud NVIDIA H100 SXM (80 GB) — LLM Sizing & Speed Matrix

3.35 TB/s HBM3 cloud benchmark standard; ideal for high-batch vLLM serving & 70B Q4/Q6.

Reference Pricing
~$2.39/hr Cloud
Usable Memory
79.4 GiB
Memory Bandwidth
3350 GB/s (84% eff.)
FP16 Compute
989 TFLOPS
Roofline Ridge
253.0 FLOP/B
Power (Load / Idle)
700W / 0W

Buy or Rent Cloud NVIDIA H100 SXM (80 GB)

Vendor & Cloud Links
Deploy on DigitalOcean GPU Droplets
DigitalOcean · On-Demand / Spot GPU
↗
Launch Pod on RunPod
RunPod · On-Demand / Spot GPU
↗
Deploy on Vast.ai Spot GPU
Vast.ai · On-Demand / Spot GPU
↗
Run Serverless / GPU on Novita AI
Novita AI · On-Demand / Spot GPU
↗
Deploy on Vultr Cloud GPU
Vultr · On-Demand / Spot GPU
↗

All 32 Open-Weight Models on Cloud NVIDIA H100 SXM (80 GB) (Q4_K_M @ 32k)

Model Status Weights Total @ 32k Decode TTFT (4k) Max Safe Ctx
Command A 111B COMFORTABLE 63.9 GiB 69.1 GiB 39.3 tok/s 1.35s 197k
Llama 4 Scout 109B-A17B COMFORTABLE 62.8 GiB 70.2 GiB 146.1 tok/s 0.24s 81k
Qwen2.5 72B Instruct COMFORTABLE 41.9 GiB 53.7 GiB 50.5 tok/s 0.90s 32k
DeepSeek-R1 Distill Llama 70B COMFORTABLE 40.7 GiB 52.5 GiB 51.7 tok/s 0.88s 118k
Llama 3.3 70B Instruct COMFORTABLE 40.7 GiB 52.5 GiB 51.7 tok/s 0.88s 118k
Llama 3.1 70B Instruct COMFORTABLE 40.7 GiB 52.5 GiB 51.7 tok/s 0.88s 118k
Llama-3.3 Nemotron Super 49B COMFORTABLE 28.7 GiB 36.7 GiB 75.2 tok/s 0.63s 128k
Mixtral 8x7B Instruct COMFORTABLE 26.9 GiB 32.2 GiB 201.8 tok/s 0.19s 32k
Qwen3 32B COMFORTABLE 18.9 GiB 28.3 GiB 97.5 tok/s 0.44s 40k
Qwen2.5 Coder 32B COMFORTABLE 18.9 GiB 28.3 GiB 97.5 tok/s 0.43s 32k
DeepSeek-R1 Distill Qwen 32B COMFORTABLE 18.9 GiB 28.3 GiB 97.5 tok/s 0.43s 128k
QwQ 32B Reasoning COMFORTABLE 18.9 GiB 28.3 GiB 97.5 tok/s 0.43s 40k
GLM-4 32B 0414 COMFORTABLE 18.6 GiB 22.1 GiB 127.8 tok/s 0.42s 32k
Qwen3 30B-A3B COMFORTABLE 17.6 GiB 21.6 GiB 470.6 tok/s 0.08s 40k
Gemma 3 27B IT COMFORTABLE 15.8 GiB 20.2 GiB 140.3 tok/s 0.36s 128k
Mistral Small 3.1 24B COMFORTABLE 13.8 GiB 20.3 GiB 139.2 tok/s 0.32s 128k
Codestral 22B COMFORTABLE 12.8 GiB 21.4 GiB 132.5 tok/s 0.31s 32k
Qwen3 14B COMFORTABLE 8.5 GiB 15.0 GiB 193.8 tok/s 0.21s 40k
Qwen2.5 14B Instruct COMFORTABLE 8.5 GiB 16.0 GiB 180.4 tok/s 0.22s 32k
DeepSeek-R1 Distill Qwen 14B COMFORTABLE 8.5 GiB 16.0 GiB 180.4 tok/s 0.22s 128k
Phi-4 14B COMFORTABLE 8.5 GiB 13.0 GiB 226.1 tok/s 0.21s 16k
Gemma 3 12B IT COMFORTABLE 7.0 GiB 10.6 GiB 280.6 tok/s 0.18s 128k
Qwen3 8B COMFORTABLE 4.7 GiB 10.5 GiB 284.2 tok/s 0.14s 40k
Llama 3.1 8B Instruct COMFORTABLE 4.6 GiB 9.9 GiB 303.9 tok/s 0.13s 128k
Qwen2.5 7B Instruct COMFORTABLE 4.4 GiB 7.4 GiB 427.7 tok/s 0.13s 32k
Gemma 3 4B IT COMFORTABLE 2.5 GiB 4.4 GiB 785.6 tok/s 0.09s 128k
DeepSeek-R1 671B OOM 386.4 GiB 390.3 GiB OOM — OOM
DeepSeek-V3 671B OOM 386.4 GiB 390.3 GiB OOM — OOM
Llama 3.1 405B Instruct OOM 233.8 GiB 252.5 GiB OOM — OOM
Llama 4 Maverick 400B-A17B OOM 231.5 GiB 239.0 GiB OOM — OOM
Qwen3 235B-A22B OOM 135.4 GiB 142.6 GiB OOM — OOM
Mixtral 8x22B Instruct OOM 81.2 GiB 89.8 GiB OOM — OOM

Interactive Cloud NVIDIA H100 SXM (80 GB) Simulator

111B Dense · GQA
3350 GB/s · 989 TFLOPS
3. Weight Quantization 4.85 bpw
4. KV Cache
32,768 tok
Compute Roofline
COMFORTABLE FIT Memory-Bandwidth Bound
Max Safe Ctx: 197k
Memory Allocation Footprint 69.11 / 79.40 GiB usable
Weights: 63.93 GiB KV Cache: 2.75 GiB Overhead: 2.44 GiB Headroom: 10.29 GiB
Decode Speed
39.3
tok/s (batch=1)
Prefill Speed
3117
prompt tok/s
TTFT Latency
1.35s
for 4,096 tok
Active Params
111.0B
of 111B total
vLLM Production Serve Command 
vllm serve CohereLabs/c4ai-command-a-03-2025 --max-model-len 32768 --quantization fp8
Context KV Cache Total VRAM Decode Status
4k 1.00 GiB 66.52 GiB 40.4 tok/s COMFORTABLE
8k 1.25 GiB 67.61 GiB 40.2 tok/s COMFORTABLE
16k 1.75 GiB 68.11 GiB 39.9 tok/s COMFORTABLE
32k 2.75 GiB 69.11 GiB 39.3 tok/s COMFORTABLE
64k 4.75 GiB 71.11 GiB 38.2 tok/s COMFORTABLE
128k 8.75 GiB 75.11 GiB 36.1 tok/s TIGHT

Buy-vs-Cloud-vs-API Monthly TCO Break-Even

USE SERVERLESS API
Own Hardware/mo
$0.00
110.5h active + power
Cloud GPU/mo
$303.57
@$2.39/hr spot/pod
Serverless API/mo
$275.00
API cheaper

Recommended Hardware & Cloud Deployment

Contextual to fit status

Hardware FAQ — Cloud NVIDIA H100 SXM (80 GB)

How much usable VRAM / GPU memory does Cloud NVIDIA H100 SXM (80 GB) actually provide for LLMs?

Cloud NVIDIA H100 SXM (80 GB) provides 79.4 GiB usable VRAM after subtracting 0.6 GiB driver/display overhead.

What is the largest Q4_K_M model Cloud NVIDIA H100 SXM (80 GB) can run at 32k context?

With Q4_K_M weights and FP16 KV cache at 32k context, the largest fitting model in our verified catalog on Cloud NVIDIA H100 SXM (80 GB) is Command A 111B (111B total).