RTX 4090 24 GB Running Qwen 32B
Fits comfortably at Q4_K_M (~21 GB total with 8K Q8 KV cache) at ~48–60 tok/s.
Explore pre-computed memory breakdowns and bandwidth-limited decode speed for popular GPUs and Apple Silicon Macs, or test custom weights in our VRAM Checker.
Fits comfortably at Q4_K_M (~21 GB total with 8K Q8 KV cache) at ~48–60 tok/s.
Even at Q4_K_M (~43 GB), a 70B dense model spills onto system DDR RAM unless quantized to IQ2_XXS or paired in 2x 3090 NVLink.
Total ~45 GB fits inside 48 GB dual-3090 VRAM with zero system RAM spill.
32 GB capacity unlocks Q6_K studio coding fidelity instead of settling for 4-bit weights.
Q6_K leaves ~3 GB headroom for a 16K code repository context window.
Runs 8B at Q8_0 (near-lossless) around ~45–55 tok/s on modest 170W TDP.
8B Q5_K_M fits silently on under 35W at ~18–22 tok/s.
Fits cleanly below macOS 18 GB wired limit with zero swap thrashing.
29 GB weights + context fits neatly inside the 36 GB macOS wired GPU pool.
46 GB total sits cleanly inside M4 Max 64 GB unified wired allocation.
Near-lossless Q8_0 ~77 GB fits comfortably under macOS 102 GB ceiling.
Uniquely enables single-box DeepSeek R1 671B MoE at IQ2 (~186 GB) if wired cap is tuned.
24B at Q5_K_M delivers ~70+ tok/s on RTX 4090.
Requires 8K–16K context control so KV cache stays within 24 GB.
Fits wholly in 32 GB RTX 5090 for blazing ~150+ tok/s MoE output.
3.35 TB/s HBM3 memory saturates concurrent API serving cleanly.
Leaves over 7 GB spare VRAM for long-context retrieval agents.
32B Q8_0 + 32K context uses ~38 GB — well within M4 Max 64 GB wired allocation.