The Number Nobody Tells You to Check When Buying a GPU for Local AI

If you have spent any time researching GPUs for running AI models locally, you have probably heard the same advice repeated everywhere: get as much VRAM as possible. That advice is not wrong. VRAM capacity determines whether a model loads at all, and if your GPU does not have enough, the model either refuses to run or gets partially offloaded to system RAM at a fraction of the speed. VRAM is the hard gate.

But here is what most guides skip over entirely. Once your GPU has enough VRAM to hold the model, the number that determines how fast the model actually generates text is not VRAM size. It is memory bandwidth, and the difference between GPUs on this metric is enormous, sometimes three to four times faster on paper-comparable hardware.

Understanding this distinction will save you from two categories of mistake that are genuinely common right now. The first is buying a GPU with plenty of VRAM but mediocre bandwidth and wondering why inference feels sluggish. The second is spending two to three times more money on a flashy system spec-sheet because it promises large unified memory, only to discover that the bandwidth connecting that memory to the processor makes it feel slower than a used GPU you could have bought for a third of the price.

Why Memory Bandwidth Controls Inference Speed

Before getting into the numbers, it helps to understand exactly what happens inside a GPU every time an LLM generates a single token.

The GPU does not load the model once and then keep it in fast registers forever. For every token it generates, it reads the relevant portions of the model weights from VRAM into the compute cores, runs the attention and feed-forward calculations for that token, and then moves on to the next one. This cycle repeats for every single token in the output. The model weights are large, the cycle is continuous, and the speed of the memory connection between VRAM and compute cores is what sets the pace.

This is why LLM inference is described as memory-bound rather than compute-bound. Houtini’s July 2026 GPU guide describes it precisely that LLM inference is memory-bandwidth-bound, meaning the GPU’s compute units are frequently idle, waiting for data to arrive from VRAM rather than being limited by raw compute throughput. This has been confirmed in peer-reviewed research across multiple architectures, with arXiv papers on LLM inference consistently documenting that decoding becomes memory-bound during autoregressive generation, where weight movement from memory dominates the latency profile. Adding more compute cores to a memory-bound workload does almost nothing. Adding more bandwidth changes everything.

Here is a concrete way to think about the difference. VRAM capacity is like the size of your desk, determining how much material you can have in front of you at once. Memory bandwidth is how fast you can reach into the filing cabinet and pull out whatever you need next. You could have a desk the size of a warehouse floor, but if fetching each document takes fifteen minutes, you will not get much work done regardless of how much space you have.

The Math That Makes This Real

The relationship between memory bandwidth and inference speed is direct enough to calculate approximately. For a model with N parameters at Q4 quantization, which uses roughly 0.5 bytes per parameter, each token generation requires the GPU to read approximately the full model weights from VRAM. A 13B parameter model at Q4 quantization requires reading roughly 6 to 7 GB of data per token generated. Hardware Corner’s benchmark data shows this playing out exactly as the theory predicts across real hardware.

Working through the bandwidth math for a 13B model at Q4 quantization:

• At 960 GB/s (RTX 3090, RX 7900 XTX): approximately 19 milliseconds/token, or roughly 50 tokens/second
• At 504 GB/s (mid-range desktop GPU): approximately 36 milliseconds/token, or roughly 28 tokens/second
• At 256 GB/s (integrated APU memory): approximately 70 milliseconds/token, or roughly 14 tokens/second

That is a 3.5 times difference in speed from hardware that might cost similar amounts of money depending on what you buy. The AI Knowledge Library’s June 2026 GPU comparison puts the RTX 5090’s bandwidth advantage in context: at 1,792 GB/s it is 78% faster than the RTX 4090’s 1,008 GB/s. Hardware Corner’s benchmarks show the RTX 5090 delivering 102.7 tokens per second on Qwen3 14B at Q4 quantization with a 16K context window, making the bandwidth advantage visible in real-world output speed.

GPU Comparison Table: Ranked by Memory Bandwidth (July 2026)

This table covers the GPUs worth evaluating for local LLM inference in July 2026, sorted by memory bandwidth. Prices reflect approximate current ranges from Amazon and major retailers. The last two entries are included specifically to illustrate the bandwidth trap described in the section that follows. Hardware Corner’s LLM benchmark database and Houtini’s July 2026 GPU guide were the primary sources for confirmed bandwidth figures.

2 Expensive Machines That Will Disappoint You

The bandwidth table above reveals two categories of hardware that get marketed heavily toward the local AI crowd and consistently underdeliver on the metric that actually matters for inference speed. Both deserve a direct warning.

The AMD Ryzen AI Max Mini PC Trap

Mini PCs built around the AMD Ryzen AI Max+ 395 processor are being marketed aggressively in mid-2026 at prices ranging from $2,000 to $4,000. The headline specification is unified memory up to 96GB, which sounds compelling when you are used to discrete GPUs topping out at 24 to 32GB for consumer hardware. The problem is that this unified memory is DDR5 system RAM shared between the CPU, the integrated Radeon 8060S GPU, and the rest of the system, with a total memory bandwidth of approximately 256 GB/s. Ajit Singh’s April 2026 benchmark analysis confirms that on Apple Silicon, MLX achieves 30 to 50% faster throughput than Ollama, and the Apple M-series chips reach their performance partly because their unified memory bandwidth is dramatically higher than DDR5 systems. The Ryzen AI Max+ 395 does not have that advantage.

At 256 GB/s, the Ryzen AI Max+ 395 will generate tokens on a 13B model at roughly 14 tokens per second in theoretical maximum. A used RTX 3090 with 936 GB/s bandwidth will generate those same tokens at roughly 50 tokens per second, nearly 3.5 times faster, and currently sells for approximately $1,000 to $1,500 on the used market in July 2026 based on live eBay and resale data. The mini PC costs two to four times more and runs AI inference at roughly one-third the speed.

There are legitimate reasons to buy one of these systems. They are quiet, compact, power-efficient, and capable general-purpose computers. If your workload is mostly CPU-bound tasks, light productivity, and occasional small model inference, they are reasonable machines. If your primary use case is running local LLMs at speed, the bandwidth limitation means you are paying a significant premium for a machine that will feel slow specifically on the task you bought it for.

The NVIDIA DGX Spark Trap

NVIDIA’s DGX Spark was announced at a $3,000 price point and marketed as a personal AI supercomputer. The specification that gets the most attention is 128GB of unified LPDDR5X memory via the Grace-Blackwell SoC. 128GB is a genuinely useful amount of capacity for running very large models that do not fit on consumer hardware. Houtini’s July 2026 analysis notes that the MI300X, AMD’s enterprise datacenter card, delivers 5,300 GB/s of bandwidth on paper but costs over $20,000 and ships in rackmount form factors. The DGX Spark sits at a consumer price point with consumer bandwidth.

The DGX Spark’s LPDDR5X memory delivers approximately 273 GB/s of bandwidth. An RTX 3090 available used for approximately $1,000 to $1,500 delivers 936 GB/s. For any model that fits in 24GB, the used RTX 3090 will generate tokens roughly 3.4 times faster than the DGX Spark at less than a third of the price.

The DGX Spark makes genuine sense for one specific situation: you need to load a model larger than 60GB that simply does not fit on any consumer discrete GPU. At that point the bandwidth penalty is unavoidable because no alternative exists. For everything up to approximately 32B parameters at Q4 quantization, which covers the majority of models most developers actually use day-to-day, a much cheaper discrete GPU with higher bandwidth will outperform it on the metric that determines whether the experience feels fast or frustrating.

The Value Picks Worth Buying Right Now

RTX 3090 (Used, Around $1,000 to $1,500)

The RTX 3090 is the best value GPU for local LLM inference in mid-2026 if you do not need more than 24GB of VRAM. According to live July 2026 market data from ResalePrices.com and GPUPoet, used RTX 3090 units are currently averaging approximately $1,250, with a range of roughly $1,000 to $1,500 depending on condition, seller, and AIB model. The RTX 5080, which costs over $1,200 new, delivers 960 GB/s, a 2.6% bandwidth advantage over the RTX 3090. The RTX 4090 delivers 1,008 GB/s, which is meaningfully faster, but commands $1,800 or more.

Houtini’s analysis notes one practical consideration that the RTX 3090 draws 350W under load, which is real power consumption requiring a capable power supply and adequate case airflow. Budget 200W of PSU headroom above the GPU’s rated TDP. A system that browns out mid-inference does not fail quietly.

RX 7900 XTX (Around $800 to $1,000)

AMD’s RX 7900 XTX matches the RTX 6000 Ada’s 960 GB/s bandwidth with a full 384-bit memory bus and 24GB of GDDR6, at a fraction of the price of the professional card. The AI Knowledge Library’s GPU comparison confirms that ROCm 7.2, released in March 2026, achieved full software parity with CUDA on Linux for the primary inference frameworks. llama.cpp, vLLM, and LM Studio all work correctly with ROCm in mid-2026 without the driver friction that plagued AMD GPU inference in earlier years. If you are on Windows rather than Linux, the software story is still slightly less mature, but the gap has narrowed considerably from where it was twelve months ago.

2 RX 7900 XTX cards running in parallel via ROCm gives you 48GB of total VRAM at under $2,000, which is a setup that handles virtually every open model available for local inference at the time of writing.

Radeon AI PRO R9700 (Around $1,350)

AMD’s R9700 is the workstation card for users who want 32GB of VRAM with ECC memory reliability at a price point below the professional NVIDIA alternatives. Bandwidth is more modest at 645 GB/s due to the 256-bit memory bus, which is a real trade-off relative to the RTX 3090 and RX 7900 XTX. But the 32GB capacity is the sweet spot for running 30B to 34B models fully in VRAM without quantization compromises. CoreLab’s LLM GPU guide notes the 32GB buffer as the practical ceiling for most developers in 2026 running coding assistant and reasoning-heavy workloads with extended context.

Quick Decision Guide

The One Thing Worth Remembering

VRAM capacity tells you whether a model can run on your GPU at all. Memory bandwidth tells you whether you will enjoy using it. Both numbers matter, but they answer different questions, and conflating them is what leads people to spend $3,000 on a system with 128GB of unified memory and then discover it generates tokens more slowly than a card they could have bought for $800 used.

Check the bandwidth number before you buy. Hardware Corner’s GPU LLM benchmark database gives you tested tokens-per-second figures across models and context lengths for most cards worth considering. The math from bandwidth to inference speed is direct enough that theoretical limits translate to real-world experience with reasonable accuracy.

The used GPU market in mid-2026 is unusually good for buyers. New Blackwell cards have pushed prices down on the Ampere and Ada generation hardware that was premium twelve months ago. An RTX 3090 at approximately $1,000 to $1,500 on the used market, with 936 GB/s of bandwidth and 24GB of VRAM, is a legitimately excellent local inference machine. It will outperform systems costing two to three times more on the workload you bought it for.

Your future self running a 32B parameter model at 50 tokens per second will be considerably happier than the version who bought the flashy spec sheet and is watching a loading spinner.