Llama 3 · Llama-3.1 · HF: meta-llama/Llama-3.1-405B-Instruct
Llama 3.1 405B Instruct — VRAM, KV Cache & Hardware Matrix
Serverless API Rate
$0.80 in / $1.20 out per 1M
Parameters
405.9B Dense
Attention
GQA (8 KV / 128 Q)
Layers × Hidden
126 × 16384
Max Native Context
128k tokens
MoE Experts
N/A (Dense)
Quantization Memory Footprint
| Quantization | BPW | Weights Only | Total @ 8k Ctx | Total @ 32k Ctx | Quality |
|---|---|---|---|---|---|
| FP16 | 16.00 | 771.17 GiB | 777.91 GiB | 789.72 GiB | Lossless Reference |
| Q8_0 | 8.50 | 409.68 GiB | 416.42 GiB | 428.23 GiB | Near-Lossless |
| Q6_K | 6.57 | 316.66 GiB | 323.40 GiB | 335.21 GiB | High Precision |
| Q5_K_M | 5.69 | 274.25 GiB | 280.98 GiB | 292.80 GiB | Balanced High |
| Q4_K_M | 4.85 | 233.76 GiB | 240.50 GiB | 252.31 GiB | Recommended 4-bit |
| MLX_4BIT | 4.50 | 216.89 GiB | 223.63 GiB | 235.44 GiB | Apple Silicon Optimized |
| IQ4_XS | 4.25 | 204.84 GiB | 211.58 GiB | 223.39 GiB | Compact 4-bit |
| Q3_K_M | 3.91 | 188.45 GiB | 195.19 GiB | 207.00 GiB | Aggressive Compression |
Hardware Compatibility & Speed Leaderboard (Q4_K_M @ 32k Context)
Interactive Llama 3.1 405B Instruct Simulator
405.9B Dense · GQA
819 GB/s · 58 TFLOPS
3. Weight Quantization
4.85 bpw
4. KV Cache
32,768 tok
Compute Roofline
COMFORTABLE FIT Memory-Bandwidth Bound
Max Safe Ctx: 128k
Memory Allocation Footprint 252.31 / 399.36 GiB usable
Weights: 233.76 GiB
KV Cache: 15.75 GiB
Overhead: 2.80 GiB
Headroom: 147.05 GiB
Decode Speed
2.4
tok/s (batch=1)
Prefill Speed
41
prompt tok/s
TTFT Latency
100.94s
for 4,096 tok
Active Params
405.9B
of 405.9B total
llama.cpp Server Command
llama-server -hf meta-llama/Llama-3.1-405B-Instruct:Q4_K_M -ngl 99 -fa on -c 32768 --port 8080| Context | KV Cache | Total VRAM | Decode | Status |
|---|---|---|---|---|
| 4k | 1.97 GiB | 237.40 GiB | 2.5 tok/s | COMFORTABLE |
| 8k | 3.94 GiB | 240.50 GiB | 2.5 tok/s | COMFORTABLE |
| 16k | 7.88 GiB | 244.44 GiB | 2.5 tok/s | COMFORTABLE |
| 32k | 15.75 GiB | 252.31 GiB | 2.4 tok/s | COMFORTABLE |
| 64k | 31.50 GiB | 268.06 GiB | 2.2 tok/s | COMFORTABLE |
| 128k | 63.00 GiB | 299.56 GiB | 2.0 tok/s | COMFORTABLE |
Buy-vs-Cloud-vs-API Monthly TCO Break-Even
USE SERVERLESS APIOwn Hardware/mo
$203.05
2092.9h active + power
Cloud GPU/mo
$8399.76
@$3.49/hr spot/pod
Serverless API/mo
$58.00
Payback: 233.3 mo
Recommended Hardware & Cloud Deployment
Contextual to fit statusEngineering FAQ — Llama 3.1 405B Instruct
How much VRAM does Llama 3.1 405B Instruct require in Q4_K_M?
Llama 3.1 405B Instruct weights occupy 233.76 GiB in Q4_K_M. Including 32k FP16 KV cache and runtime buffers, total memory is 252.31 GiB.
What is the cheapest local hardware that runs Llama 3.1 405B Instruct?
Apple Mac Studio M3 Ultra (512 GB) (512 GB) runs Llama 3.1 405B Instruct Q4_K_M at ~2.4 tok/s decode and 100.94s TTFT.
How does Llama 3.1 405B Instruct scale KV cache at long context?
Llama 3.1 405B Instruct uses GQA with 8 KV heads across 126 layers (128 head_dim).