Cloud vs. Rig TCO
Plot cumulative 36-month API spending against hourly RunPod/H100 rentals and owned RTX/Mac hardware with electricity and resale value.
Verify whether Llama 3.3 70B, Qwen 32B, Mistral 24B, or DeepSeek R1 fits inside your NVIDIA GPU or Apple Silicon unified memory across 7 quantization formats and KV-cache tiers.
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 \ ollama run qwen2.5-coder:32b-instruct-q4_k_m --ctx-size 8192
| Quantization | Memory Needed | Speed Est. | Placement |
|---|---|---|---|
| FP16 / BF16 (16-bit)Reference Lossless | 66.9 GB | 1 tok/s | RAM Offload |
| Q8_0 (8-bit)Near-Lossless | 36.43 GB | 2.5 tok/s | RAM Offload |
| Q6_K (6-bit K-Quant)Studio Fidelity | 28.55 GB | 3.8 tok/s | RAM Offload |
| Q5_K_M (5-bit Medium)Minimal Loss | 25.06 GB | 4.8 tok/s | RAM Offload |
| Q4_K_M (4-bit Sweet Spot)Recommended Default | 21.6 GB | 32.9 tok/s | Pure GPU VRAM |
| IQ3_XS (3.3-bit I-Quant)Noticeable Degradation | 15.71 GB | 52 tok/s | Pure GPU VRAM |
| IQ2_XXS (2.1-bit Extreme)Emergency Fit Only | 10.84 GB | 79.9 tok/s | Pure GPU VRAM |
| Model | Best Pure-VRAM Quant | Memory | Est. Speed |
|---|---|---|---|
| Llama 3.1 8B InstructUltra-fast RAG extraction, IDE autocomplete & classification | FP16 | 17.46 GB | 45 tok/s |
| Qwen 2.5 Coder 14BMid-tier local VS Code copilot balancing speed & repo accuracy | Q8_0 | 17.27 GB | 46.1 tok/s |
| Mistral Small 24BClean instruction following on 24 GB single-GPU workstations | Q6_K | 21.21 GB | 33 tok/s |
| Gemma 2 / 3 27BMultilingual & editorial writing in single 24–32 GB GPUs | Q5_K_M | 21.72 GB | 33.3 tok/s |
| Qwen 2.5 32B / DeepSeek-R1 32BGold-standard local coding & reasoning on RTX 3090/4090/5090 & 48GB Macs | Q4_K_M | 21.6 GB | 32.9 tok/s |
| Mixtral 8x7B MoE (46.7B Total)Sparse Mixture-of-Experts: needs 47B RAM footprint, computes at 13B speed | IQ3_XS | 21.55 GB | 117.5 tok/s |
| Llama 3.3 70B / DeepSeek-R1 70BGPT-4 class private local reasoning for 48 GB–128 GB rigs | IQ2_XXS | 21.87 GB | 33.3 tok/s |
| DeepSeek R1 671B MoE (Full Frontier)Cluster or Mac Studio Ultra / dual EPYC server deployment (37B active per token) | Needs Offload / Cluster | 408.95 GB | OOM |
| Qwen3-30B-A3B (MoE — 3.3B Active)Ultra-fast 30B Sparse MoE decoding at 3.3B speed; effortless fit on 24 GB cards | Q4_K_M | 19.77 GB | 352.8 tok/s |
| Qwen3-32B (Dense Dual-Thinking)2026 dense local reasoning & coding benchmark with dual thinking/non-thinking toggle | Q4_K_M | 21.78 GB | 32.6 tok/s |
| Llama 4 Scout 109B (MoE — 17B Active)Multimodal 109B MoE (17B active) built for 64–128 GB Macs, Dual-3090, or H100 pods | Needs Offload / Cluster | 68.03 GB | 6 tok/s |
Because LLM autoregressive decoding reads every weight tensor per generated token, choosing a quantization level directly bounds both fit probability and tokens-per-second throughput.
| GGUF Quant | Effective Bits/Weight | Perplexity Impact | Recommended Use Case |
|---|---|---|---|
| IQ3_M / Q3_K_M | ~3.66–3.90 bpw | Noticeable degradation on complex code | Fitting 70B models onto dual-16 GB / single-32 GB cards |
| Q4_K_M (Sweet Spot) | ~4.85 bpw | Minimal loss vs FP16 baseline | Default Ollama tier; ideal 24 GB GPU fit for 24B–32B models |
| Q5_K_M / Q6_K | ~5.70–6.57 bpw | Near-lossless reasoning & SQL | 32 GB–64 GB RTX 5090 / Apple M-Series Pro/Max coding rigs |
| Q8_0 (FP8 Equiv) | 8.50 bpw | Indistinguishable from BF16 | Mac Studio Ultra / 96 GB–128 GB high-accuracy evaluation |
Total GPU Memory = (Model_Parameters * Bits_Per_Weight / 8) + (2 * Layers * KV_Heads * Head_Dim * Context_Length * KV_Bytes * Batch_Users / 1024^3) + Runtime_Buffer. Decode tok/s scales with memory bandwidth divided by active per-token weight footprint.
Primary Technical & US Grid Benchmarks
Plot cumulative 36-month API spending against hourly RunPod/H100 rentals and owned RTX/Mac hardware with electricity and resale value.
Calculate daily, monthly, and annual kWh bills using 80+ PSU efficiency, duty cycle, optional AC cooling load, and 50 US state EIA rates.
Combine dual-GPU draw, CPU peak boost, NVMe arrays, and ATX 3.1 transient spikes to hit the 50–70% PSU efficiency sweet spot.