Realistic GPU/Memory Sizing for Local LLMs

Mon, 25 May 2026 00:00:00 +0000

Decision-first: Budget file size + KV(context) + overhead, not file size — and on unified memory, subtract OS + co-resident workloads first. “Barely fits” means doesn’t fit. Size memory by total params, speed by active params.

Scope & freshness: General sizing principles (version-independent); worked numbers from 2026-05 on a GB10 (128 GB unified) + a 64 GB Apple-Silicon Mac. Re-measure resident sizes for your model/quant/context.

Resident size is bigger than the file#

The single most common sizing mistake is equating the model file size with how much memory it needs at runtime. Resident footprint is:

Running Local LLMs on the NVIDIA GB10 (DGX Spark / ASUS Ascent GX10)

Mon, 25 May 2026 00:00:00 +0000

Decision-first: On a GB10, pick low-active MoE models (A3B-class), serve GGUF (not MLX) via LM Studio, run one model at a time behind an OOM guard, and monitor GPU via DCGM but read the model footprint from system RAM (no framebuffer metrics). Dense 70B is unusable (~2-3 tok/s).

Scope & freshness: GB10 / Grace-Blackwell, 128 GB unified, DCGM 4.5.3 + driver 580-class, as of 2026-05-25. Re-check the DCGM profiling/framebuffer gaps after a driver/DCGM bump (≥585).

Gpu-Memory-Sizing on Agent Zone

Realistic GPU/Memory Sizing for Local LLMs

Resident size is bigger than the file#

Running Local LLMs on the NVIDIA GB10 (DGX Spark / ASUS Ascent GX10)