LIVE TELEMETRYGPT-5$1.25 / $10.00Sonnet 5.5$2.00 / $10.00Gemini 3.8$0.75 / $3.75DS V4.1$0.15 / $0.60RTX 50901,792 GB/sM4 Max546 GB/sUS Avg$0.182/kWh
TELEMETRY CONSOLE NAVIGATION

Guncelkal Telemetry Index

Search VRAM specs, API costs, or Dev tools

AI & LLM EconomicsGQA KV Cache + CLI Export

Can My PC or Mac Run This Local LLM? (VRAM Checker)

Verify whether Llama 3.3 70B, Qwen 32B, Mistral 24B, or DeepSeek R1 fits inside your NVIDIA GPU or Apple Silicon unified memory across 7 quantization formats and KV-cache tiers.

Hardware & Model Configuration

8,192 tokens
1 stream
64 GB
KV Cache Precision:
CLI LAUNCH SNIPPET
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 \
  ollama run qwen2.5-coder:32b-instruct-q4_k_m --ctx-size 8192
Total Required
21.6 GB
Safe cap: 22.8 GB
Est. Speed
32.9 tok/s
Decode bandwidth bound
KV Cache
1 GB
Q8_0 @ 8K ctx
Max Context Fit
18,022
Tokens before spill
VRAM ALLOCATION & KV-CACHE MAP21.6 GB / 22.8 GB Usable
Safe Pool (22.8 GB)Physical 24 GB
Weights: 19.7 GB KV Cache: 1 GB Runtime: 0.9 GB
Status: Tight Fit (Near Memory Ceiling — Cap Context)

All 7 GGUF / Quantization Tiers on NVIDIA RTX 4090 (24 GB)

QuantizationMemory NeededSpeed Est.Placement
FP16 / BF16 (16-bit)Reference Lossless66.9 GB1 tok/sRAM Offload
Q8_0 (8-bit)Near-Lossless36.43 GB2.5 tok/sRAM Offload
Q6_K (6-bit K-Quant)Studio Fidelity28.55 GB3.8 tok/sRAM Offload
Q5_K_M (5-bit Medium)Minimal Loss25.06 GB4.8 tok/sRAM Offload
Q4_K_M (4-bit Sweet Spot)Recommended Default21.6 GB32.9 tok/sPure GPU VRAM
IQ3_XS (3.3-bit I-Quant)Noticeable Degradation15.71 GB52 tok/sPure GPU VRAM
IQ2_XXS (2.1-bit Extreme)Emergency Fit Only10.84 GB79.9 tok/sPure GPU VRAM

All 11 Open-Weight LLMs Ranked for NVIDIA RTX 4090 (24 GB) (Click Row to Switch)

ModelBest Pure-VRAM QuantMemoryEst. Speed
Llama 3.1 8B InstructUltra-fast RAG extraction, IDE autocomplete & classificationFP1617.46 GB45 tok/s
Qwen 2.5 Coder 14BMid-tier local VS Code copilot balancing speed & repo accuracyQ8_017.27 GB46.1 tok/s
Mistral Small 24BClean instruction following on 24 GB single-GPU workstationsQ6_K21.21 GB33 tok/s
Gemma 2 / 3 27BMultilingual & editorial writing in single 24–32 GB GPUsQ5_K_M21.72 GB33.3 tok/s
Qwen 2.5 32B / DeepSeek-R1 32BGold-standard local coding & reasoning on RTX 3090/4090/5090 & 48GB MacsQ4_K_M21.6 GB32.9 tok/s
Mixtral 8x7B MoE (46.7B Total)Sparse Mixture-of-Experts: needs 47B RAM footprint, computes at 13B speedIQ3_XS21.55 GB117.5 tok/s
Llama 3.3 70B / DeepSeek-R1 70BGPT-4 class private local reasoning for 48 GB–128 GB rigsIQ2_XXS21.87 GB33.3 tok/s
DeepSeek R1 671B MoE (Full Frontier)Cluster or Mac Studio Ultra / dual EPYC server deployment (37B active per token)Needs Offload / Cluster408.95 GBOOM
Qwen3-30B-A3B (MoE — 3.3B Active)Ultra-fast 30B Sparse MoE decoding at 3.3B speed; effortless fit on 24 GB cardsQ4_K_M19.77 GB352.8 tok/s
Qwen3-32B (Dense Dual-Thinking)2026 dense local reasoning & coding benchmark with dual thinking/non-thinking toggleQ4_K_M21.78 GB32.6 tok/s
Llama 4 Scout 109B (MoE — 17B Active)Multimodal 109B MoE (17B active) built for 64–128 GB Macs, Dual-3090, or H100 podsNeeds Offload / Cluster68.03 GB6 tok/s

GGUF Quantization Bit-Width & Quality Trade-Offs

Because LLM autoregressive decoding reads every weight tensor per generated token, choosing a quantization level directly bounds both fit probability and tokens-per-second throughput.

Browse all GPU + Model pairings →
GGUF QuantEffective Bits/WeightPerplexity ImpactRecommended Use Case
IQ3_M / Q3_K_M~3.66–3.90 bpwNoticeable degradation on complex codeFitting 70B models onto dual-16 GB / single-32 GB cards
Q4_K_M (Sweet Spot)~4.85 bpwMinimal loss vs FP16 baselineDefault Ollama tier; ideal 24 GB GPU fit for 24B–32B models
Q5_K_M / Q6_K~5.70–6.57 bpwNear-lossless reasoning & SQL32 GB–64 GB RTX 5090 / Apple M-Series Pro/Max coding rigs
Q8_0 (FP8 Equiv)8.50 bpwIndistinguishable from BF16Mac Studio Ultra / 96 GB–128 GB high-accuracy evaluation

Telemetry Methodology & Benchmark Sources

Read full equations →

Total GPU Memory = (Model_Parameters * Bits_Per_Weight / 8) + (2 * Layers * KV_Heads * Head_Dim * Context_Length * KV_Bytes * Batch_Users / 1024^3) + Runtime_Buffer. Decode tok/s scales with memory bandwidth divided by active per-token weight footprint.

Primary Technical & US Grid Benchmarks

  • llama.cpp GGUF Quantization Bit-Width Reference
  • Grouped-Query Attention (GQA) KV-Cache Specification
  • Apple Silicon Metal Unified Wired Memory Limits
  • NVIDIA GDDR6X / GDDR7 Memory Bandwidth Tables

Engineering & Pricing FAQ

Why can Apple Silicon Macs not allocate 100% of unified RAM to Metal LLM weights?
macOS reserves roughly 20–25% of unified memory for the OS kernel, window server, and CPU apps by default (known as the wired memory ceiling) unless overridden via sysctl iogpu.wired_limit_mb.
Why does decoding speed collapse when even 2 GB spills to system RAM?
Transformer decode is strictly memory-bandwidth bound. GPU VRAM runs at 360 to 1,792 GB/s, whereas spilling layers over PCIe 4.0/5.0 to system DDR5 (~50–75 GB/s) forces every generated token to wait on slow bus transfers.
How much memory does q8_0 KV-cache quantization save at 32K–64K context?
Switching KV cache precision from fp16 to q8_0 cuts context memory by 50% with negligible perplexity impact on GQA models, often reclaiming 4 GB–8 GB of headroom so long repository sessions stay fully resident on-card.
Why do MoE models like Qwen3-30B-A3B decode much faster than dense 32B models?
Sparse Mixture-of-Experts (MoE) architectures still require total VRAM for all ~30B parameters, but only activate ~3B weights per token pass—meaning decode throughput matches a tiny 3B model rather than a 32B dense matrix multiply.
What is the best quantization tier for coding and reasoning?
Q4_K_M (~4.85 bits/weight) offers the industry sweet spot retaining most coding accuracy in 24 GB cards; if you have 32 GB to 64 GB, stepping up to Q6_K or Q8_0 recovers near-lossless perplexity.

Continue in Console Workflow

AI & LLM Economics

Cloud vs. Rig TCO

Plot cumulative 36-month API spending against hourly RunPod/H100 rentals and owned RTX/Mac hardware with electricity and resale value.

Open Workbench
Hardware & Power

kWh Power Cost Lab

Calculate daily, monthly, and annual kWh bills using 80+ PSU efficiency, duty cycle, optional AC cooling load, and 50 US state EIA rates.

Open Workbench
Hardware & Power

PSU Wattage & Spike

Combine dual-GPU draw, CPU peak boost, NVMe arrays, and ATX 3.1 transient spikes to hit the 50–70% PSU efficiency sweet spot.

Open Workbench