Roofline Memory, Bandwidth & TCO Equations

1. Binary Memory & OS Wired Ceilings

All capacities and tensor footprints are computed in binary gibibytes (1 GiB = 1024³ bytes). Model weights include a 2% container alignment & unquantized norm/embedding overhead:

weights_gib = (params_total_b * 1e9 * (bpw / 8.0) * 1.02) / (1024 ** 3)

• Discrete / Cloud GPUs: Usable VRAM subtracts 1.0 GiB display driver buffer (0.6 GiB headless cloud) plus 0.35 GiB per extra GPU for NCCL/PCIe peer buffers.
• Apple Silicon UMA: macOS caps wired GPU memory around min(78%, RAM - 6 GiB) by default, unlockable up to min(88%, RAM - 4.5 GiB) via sudo sysctl iogpu.wired_limit_mb=....
• AMD Strix Halo / UMA Mini-PCs: BIOS/GTT defaults reserve 96 GiB usable on 128 GB systems (110 GiB tuned via Linux amdgpu.gttsize / TTM).

2. Architecture-Aware KV Cache (GQA, MLA & Hybrid Sliding Window)

Naive calculators fail wildly on modern attention variants. We extract layer counts, KV heads, head dimensions, MLA latent ranks, and sliding-window patterns straight from Hugging Face config.json:

# Standard GQA / MHA
kv_bytes = 2 * effective_layer_tokens * num_kv_heads * head_dim * kv_precision_bytes * batch_size

# DeepSeek Multi-Head Latent Attention (MLA: R1 / V3)
kv_bytes = effective_layer_tokens * (kv_lora_rank + qk_rope_head_dim) * kv_precision_bytes * batch_size

# Hybrid Sliding-Window (e.g. Gemma 3, Cohere Command A)
effective_layer_tokens = global_layers * ctx_tokens + local_layers * min(ctx_tokens, sliding_window)

3. MoE Batch Union, Decode Bandwidth & Prefill TTFT

During autoregressive decode, every generated token reads both active weights and the entire active KV cache across the memory bus. For Mixture-of-Experts models at concurrency B > 1, routed experts form a probabilistic union:

eff_params_b = min(total_b, shared_b + routed_pool_b * (1 - (1 - k_active / n_experts) ** batch_size))
step_bytes   = eff_params_b * 1e9 * (bpw / 8.0) * 1.02 + kv_bytes_total
decode_tok_s = min(eff_bw_bytes_s / step_bytes * batch_size, eff_tflops * 1e12 / (2 * active_b * 1e9))

# Prompt Prefill (TTFT): Compute-Roofline bound including O(N * L * d) attention FLOPs
prefill_tok_s = (eff_tflops * 1e12) / (2 * active_b * 1e9 + 2 * num_layers * num_q_heads * head_dim * prompt_len)
ttft_s        = 0.035 + prompt_len / prefill_tok_s

4. Buy vs. Cloud GPU vs. Serverless API TCO

Monthly ownership cost combines straight-line 36-month depreciation net of conservative resale value (35% Apple/NVIDIA, 25% Mini-PC) plus active compute and idle wall-plug electricity ($/kWh). Cloud GPU estimates multiply exact required active hours by a 1.15x container boot/session factor; Serverless API costs pull directly from live OpenRouter prompt/completion rates.