DollarWise

How to Pick the Right GPU for AI Workloads: VRAM Guide

October 18, 2026 · 5 min read · By the DollarWise team
How to Pick the Right GPU for AI Workloads: VRAM Guide
On this page

When people shop for AI GPUs, they obsess over model names — 4090, A100, H100. But ask any ML engineer what spec actually decides whether their workload runs, and you’ll hear one word: VRAM. If the model doesn’t fit in GPU memory, it doesn’t run at all. Speed only matters after fit. This guide will help you size VRAM correctly for real workloads — and avoid the two classic mistakes: renting too little (out-of-memory errors) or paying for memory you never touch.

Why VRAM is the bottleneck

A neural network’s weights live in VRAM during training and inference. Rough rule of thumb: a model needs about 1 byte per parameter in 8-bit quantization, 0.5 bytes in 4-bit, plus overhead. So a 7B-parameter model needs roughly 7GB at 8-bit or 3.5GB at 4-bit — before you count the KV cache (which grows with context length and batch size), activations during training, and optimizer states (which can double or triple the footprint for full fine-tuning).

That overhead is where people get burned. A 13B model “fits” in 16GB at 4-bit on paper — until a 32k context window and a modest batch push it over the edge with a CUDA out-of-memory error at 2 AM.

VRAM cheat sheet by workload

Here are realistic minimums with 20–30% headroom baked in:

LLM inference (chat, RAG, agents)

  • 1B–3B models: 8GB is plenty (even 4-bit on a consumer card)
  • 7B–8B models: 12–16GB (4-bit), 16–24GB (8-bit)
  • 13B–14B models: 16GB (4-bit), 24GB+ (8-bit)
  • 32B models: 24GB (4-bit), 40GB+ (8-bit)
  • 70B models: 40GB (4-bit, e.g. 2×24GB or 1×48GB), 80GB (8-bit)

Image generation (Stable Diffusion, Flux)

  • SD 1.5 / SDXL: 8–12GB
  • Flux Dev (12B params): 16–24GB (quantized versions run on 12GB; full quality wants 24GB)

Fine-tuning

  • LoRA/QLoRA on 7B–8B: 16–24GB (QLoRA shines here — full fine-tuning of the same model needs 60GB+)
  • LoRA on 13B: 24GB; on 70B: 40–48GB
  • Full fine-tuning: roughly 4–8× the inference footprint. A 7B full fine-tune wants ~60GB; this is H100/A100 territory.

Picking the GPU: a practical decision tree

  1. List your heaviest workload first. Size for the maximum, not the average. One 70B experiment a month still needs the 70B-capable card.
  2. Choose your quantization. 4-bit (AWQ/GPTQ) roughly halves VRAM needs with small quality loss. If you’re unsure, assume 4-bit for inference — it’s the default in tools like Ollama and vLLM.
  3. Add headroom. Take the weight size and multiply by 1.3. Long contexts (100k+ tokens) and large batches need more.
  4. Match the card. Common cloud tiers: 16GB (RTX 4080-class), 24GB (RTX 4090/3090), 48GB (RTX 6000 Ada / L40S), 80GB (A100/H100). Pick the cheapest tier that clears your headroom number.
  5. Consider multi-GPU. Two 24GB cards via vLLM tensor parallelism often cost less per hour than one 48GB card, with similar effective memory.

Rent before you buy

GPU prices swing wildly, and your needs change per project. Renting cloud GPUs by the hour lets you test whether that 70B model actually fits on 48GB before committing to hardware — and for sporadic workloads it’s dramatically cheaper than owning.

Advertisement

If you go the rental route, If you go the rental route, RunPod is one of the most popular options, with per-second billing and a wide range of VRAM tiers from 16GB consumer cards up to 80GB H100s. Disclosure: that’s our referral link — if you sign up and spend your first $10, we both get a randomized $5–$500 in RunPod credits (not cash), plus we earn 3% of Pod spend and 5% of Serverless spend for your first 6 months; credits expire after 90 days. Start on a small card, verify your VRAM math with nvidia-smi while the model loads, then scale up only if you hit the ceiling. Most people discover they need one tier less than they feared.

The two mistakes to avoid

Undersizing. The out-of-memory error is the most common and most frustrating failure in AI work. When in doubt, round up one tier — an hour of a slightly bigger GPU is cheaper than a day of debugging.

Oversizing. Renting an 80GB H100 to run a 7B chatbot is like hiring a freight truck to deliver a pizza. The 24GB tier handles most real-world LLM inference at a fraction of the cost. Match the card to the math, not the hype.

Frequently asked questions

How much VRAM do I need for AI workloads? Roughly: 8–12 GB for small LLM inference and image generation, 16–24 GB for 7B–13B models in 4-bit quantization or LoRA fine-tuning, 40–80 GB for 70B-class models or full fine-tuning. Always add 20–30% headroom for activations, the KV cache, and the optimizer.

Is VRAM more important than GPU speed for AI? Usually yes. If your model does not fit in VRAM, it does not run — no amount of extra CUDA cores fixes that. Once the model fits, speed (memory bandwidth, tensor cores) determines how fast each token or image generates.

What is the cheapest GPU with 24GB VRAM? The RTX 4090 (24GB) is the classic choice, and on cloud rental platforms like RunPod it is often among the cheapest 24GB options per hour. Older 24GB cards like the RTX 3090 rent even cheaper if speed is not critical.

Can I use multiple GPUs if one doesn’t have enough VRAM? Yes — frameworks like vLLM and Hugging Face Accelerate support tensor and pipeline parallelism across multiple GPUs. Two 24GB cards behave like one 48GB card for inference, though interconnect speed (NVLink vs PCIe) affects performance.

#gpu#vram#ai-training#cloud-gpu
Advertisement
Written by the DollarWise team

We turn confusing money topics into plain-English guides for beginners. Every article is written to be actionable on day one — no jargon, no hype, no sales pitch.

Keep reading