DollarWise

How to Pick the Right GPU for AI Training: VRAM Guide (2026)

October 22, 2026 · 6 min read · By the DollarWise team
How to Pick the Right GPU for AI Training: VRAM Guide (2026)
On this page

You have a model to train and a budget to protect. Pick a GPU with too little VRAM and your training run dies with an out-of-memory error at 2 AM. Pick one with far too much and you burn money on silicon that sits idle. VRAM — the memory on the graphics card itself — is the single most important spec for AI training, and choosing it is really just arithmetic. This guide walks through the math, the real-world numbers for popular models, and how to get the card you need without buying one.

Note: this article focuses on training workloads (fine-tuning and training from scratch). If you are choosing a card mainly for running models, the VRAM math is much lighter.

The VRAM formula for training

Training a model loads far more than just the weights into VRAM. A useful back-of-the-envelope formula for full-parameter training is:

VRAM needed ≈ 4 bytes per parameter × model size

So a 7B-parameter model needs roughly 28GB of VRAM for full training in mixed precision (fp16 weights at 2 bytes/param, gradients at 2 bytes/param, plus Adam optimizer states at about 8 bytes/param — the total lands near 14–16 bytes per parameter, or roughly 4x the weight size in practice).

Where the bytes go:

  • Model weights — the parameters themselves. 2 bytes/param in fp16, 4 in fp32.
  • Gradients — one gradient per parameter, another 2 bytes/param in mixed precision.
  • Optimizer states — Adam keeps two running averages per parameter (about 8 bytes/param). This is the silent VRAM killer.
  • Activations — intermediate values saved for backpropagation. These scale with batch size and sequence length, and for long-context training they can dwarf everything else.

The practical takeaway: training needs roughly 3–4x the VRAM of inference for the same model. A 7B model that runs fine in 14GB at inference time needs 48–60GB to fully fine-tune.

Here is a realistic map of what each tier handles in 2026:

16GB (RTX 4080, RTX 5080, A4000): QLoRA fine-tuning of 7B models, LoRA on 8B models, training small models from scratch (under ~3B parameters). Tight but workable for learning and experiments.

24GB (RTX 4090, RTX 5090, A5000, L4): The sweet spot for hobbyists and indie builders. QLoRA/LoRA fine-tuning of models up to ~13B, full fine-tuning of models up to ~3B, comfortable Stable Diffusion / Flux fine-tuning. Most individual AI projects live happily here.

48GB (RTX A6000, L40S): LoRA fine-tuning of 30–70B models with quantization, full fine-tuning of 7B models, serious image/video model training. The professional workstation tier.

80GB (A100 80GB, H100): Full fine-tuning of 7B–13B models, training mid-size models from scratch, multi-GPU setups for 70B+. This is data-center territory — almost nobody buys these for personal use.

Three techniques that shrink your VRAM bill

Before you rent a bigger card, know the techniques that make a small card act like a big one:

Advertisement

1. QLoRA (quantized LoRA). Quantize the base model to 4-bit and train only small adapter layers. A 70B model — which needs ~280GB for full training — fine-tunes in about 40–48GB with QLoRA. Quality loss is minimal for most tasks, and it is the single biggest VRAM saver available.

2. Gradient checkpointing. Trade compute for memory: instead of storing all activations, recompute them during the backward pass. Saves 30–50% of activation memory at the cost of ~20–30% slower training. One line of config, often worth it.

3. Smaller batch sizes with gradient accumulation. VRAM-hungry activations scale with batch size. Cut the batch size, accumulate gradients over more steps, and get the same effective batch size with a fraction of the memory.

With these three, a 24GB card punches far above its weight — which is why renting a mid-tier GPU beats buying a flagship for most people.

Rent vs. buy: the 2026 math

A RTX 4090 (24GB) costs around $1,600–$2,000 to buy. Renting the same card on a community cloud costs roughly $0.30–$0.50/hour. That means you can rent a 4090 for about 3,500–5,000 hours before you match the purchase price — and rental gives you access to A100s and H100s you could never justify buying.

The rule of thumb:

  • Training runs under ~200 hours total: rent, always. Even an A100 80GB at ~$1.50–$2/hour costs less than a month of a financed workstation.
  • Daily, ongoing experimentation: a local 24GB card pays for itself in convenience within months.
  • One-off fine-tunes or paper experiments: rent the exact card the math demands, train, shut it down.

For rentals, RunPod{:rel=“sponsored nofollow”} is one of the most popular community GPU clouds, with per-second billing and both on-demand and cheaper spot (interruptible) pods. Disclosure: this is an affiliate link — if you sign up and spend your first $10, we each get a randomized $5–$500 in RunPod credits (credits, not cash; they expire after 90 days).

A practical selection checklist

  1. Name your method first. Full fine-tune, LoRA, QLoRA, or from-scratch training — the method decides the multiplier more than the model does.
  2. Do the 4-bytes-per-parameter math for full training, or ~0.5–1 byte/param for QLoRA.
  3. Add 20% headroom for activations, data loading, and the framework’s own overhead. Out-of-memory errors love exactly-full cards.
  4. Match the card to the math, not to the hype. A 24GB rental that fits is better than an 80GB rental that idles.
  5. Use spot pricing for fault-tolerant runs with checkpointing — it routinely cuts costs by 50–70%.

Pick the smallest card your math allows, rent it by the hour, and put the money you saved into more training runs. That is how you pick the right GPU for AI training.

Frequently asked questions

How much VRAM do I need to fine-tune a 7B LLM? With 4-bit quantization and LoRA, a 7B model fine-tunes in about 12–16GB of VRAM, so a 24GB card like the RTX 4090 or A5000 works well. Full-parameter fine-tuning of the same model needs roughly 60–80GB, which calls for an A100 80GB or a rented cloud GPU.

Is 24GB VRAM enough for AI training in 2026? For LoRA/QLoRA fine-tuning of models up to 13B parameters and training small models from scratch, 24GB is enough. For full fine-tuning of 7B+ models or training large models, you will need 48GB or 80GB cards.

Does training use more VRAM than inference? Yes — typically 3–4x more. Training stores gradients and optimizer states on top of the model weights and activations, while inference only needs the weights plus a small activation cache.

Can I rent a high-VRAM GPU instead of buying one? Yes. Cloud GPU platforms rent cards like the A100 80GB or H100 by the hour, often for under $2/hr on community clouds. Renting is usually cheaper than buying for short training runs, and per-second billing means you only pay for the minutes you train.

#ai training#gpu#vram#cloud gpu
Advertisement
Written by the DollarWise team

We turn confusing money topics into plain-English guides for beginners. Every article is written to be actionable on day one — no jargon, no hype, no sales pitch.

Keep reading