GPU Memory: Why VRAM Matters for LLMs | AI Infrastructure Ep. 3

THE 3 AM DEVELOPER NIGHTMARE

Why Did a 14GB Model Crash a 24GB GPU?

You downloaded a 7B parameter model. The file is only 14GB on disk. Your GPU has 24GB of VRAM. You send your first prompt, and the terminal explodes with:

bash — python serve.py

> python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3-7b INFO: Loading weights (14.0 GB) onto GPU 0: NVIDIA RTX 4090 (24.0 GB VRAM)... INFO: Initializing KV cache for max context length 16,384 tokens... torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.40 GiB (GPU 0; 24.00 GiB total capacity; 23.65 GiB already allocated; 358 MiB free).

🕵️‍♂️ You had 10GB of free headroom! Where did the memory actually go? Let's break down the 3 VRAM buckets.

BUCKET 1: STATIC ALLOCATION

Model Weights: The Fixed Rent

Think of VRAM like a moving truck. The very first item loaded is the massive wardrobe: the model weights. You must pay this static rent before generating a single word.

📐

The Universal Weight Formula:
VRAM (GB) = Parameters × Bytes per Parameter

🔢

In 16-bit (FP16 / BF16), each weight = 2 Bytes:
• 7B Model = 14 GB (Fits 24GB GPU)
• 13B Model = 26 GB (Already overflows 24GB!)
• 70B Model = 140 GB (Needs 2x 80GB H100s!)

⚠️

Static vs. Dynamic: Weights never grow or shrink during inference. But they leave very little breathing room in consumer GPUs!

MODEL WEIGHT SIZE AT FP16 (2 BYTES/PARAM) STATIC BASELINE

Llama-3 (8B)16.0 GB

Mistral (13B)26.0 GB (Exceeds RTX 4090)

Mixtral 8x7B94.0 GB (Exceeds 1x H100)

Llama-3 (70B)140.0 GB (Needs 2x H100s)

Remember: That's just to load the model. You haven't sent a prompt yet!

BUCKET 2: DYNAMIC ALLOCATION

The KV Cache: The Expanding Monster

Every token the model reads or generates requires caching Key and Value attention vectors so it doesn't recalculate history. The longer the chat, the bigger the monster!

🎒

Why We Need KV Cache: Without it, generating token #500 would require recalculating tokens 1 through 499 from scratch. Speed would crater!

📈

The Context Window Explosion:
• 2,048 tokens = ~0.8 GB
• 16,384 tokens = ~6.4 GB
• 32,768 tokens (long document) = ~12.8 GB per user!

👥

The Concurrency Multiplier: If 4 users chat at once with 16k context, the KV Cache alone consumes 25+ Gigabytes!

ATTENTION VECTOR CACHING DYNAMIC SWELL

"The"

"server"

"ran"

"out"

"of"

"VRAM..."

Layer 1: Key & Value Tensors

Layer 32: Key & Value Tensors

KV Size = 2 × Layers × Heads × HeadDim × SeqLen × Batch

The Culprit: This is what turned your 14GB model into a 24GB crash!

LIVE INTERACTIVE BENCHMARK

Interactive VRAM Budget Simulator

Adjust model size, context tokens, and concurrency to watch VRAM fill in real-time. Test how close you get to the dreaded CUDA OOM threshold!

Model Parameters: 7 Billion

7B13B34B70B

Quantization / Precision: FP16 (2 B/param)

Context Window: 8,192 Tokens

2k8k16k32k

Concurrent Users (Batch): 2 Users

1248

Target Hardware:

Total: 18.2 / 24 GB

CAPACITY LIMIT

Model Weights KV Cache CUDA Drivers & Activations

THE ENGINEERING SOLUTION

Quantization: The Vacuum-Sealing Trick

How compressing weight precision from 16-bit to 4-bit shrinks 70B models from 140GB down to 35GB.

FP16 / BF16 (Uncompressed)

🧥

Fluffy Winter Coat

16 bits = 2.0 Bytes / param

70B Model Size: 140 GB

Needs 2x H100s ($60,000+)

➔

FP8 (Quarter Precision)

👔

Folded Suit

8 bits = 1.0 Byte / param

70B Model Size: 70 GB

Fits on 1x H100 (80GB)

➔

INT4 / AWQ / GPTQ

📦

Vacuum Sealed Box

4 bits = 0.5 Byte / param

70B Model Size: 35 GB

Fits on 2x RTX 4090s or 1x A100!

ARCHITECTURAL WARNING

Inference vs. Training: The 4x Trap

A model that easily fits on an RTX 4090 for inference will violently crash the same GPU during fine-tuning or training.

🚀

Inference Footprint: Just Model Weights + KV Cache + Small Activations (~16 GB for 7B).

💥

The Training Multiplier: You must also store:
• Gradients: 4 bytes per parameter.
• Optimizer States (AdamW): 8 to 12 bytes per parameter!
• Backprop Activations: Saved for backward pass.

💡

Rule of Thumb: Training a 7B model in full precision requires at least 60–80 GB VRAM (or Parameter-Efficient Fine-Tuning like LoRA / QLoRA).

7B MODEL MEMORY CONSUMPTION 4x TO 6x GAP

Inference

Weights (14 GB)

KV (2 GB)

~16 GB Total

Training (Full Adam)

Adam States (56 GB)

Gradients (28 GB)

Weights (14 GB)

~98 GB Total!

Architecture Rule: Size your data center clusters for training, not inference!

CHAPTER 03 SUMMARY

Three Golden VRAM Rules

1

Weights are Static Rent

Params × Bytes per weight. 70B in FP16 needs 140GB before anything runs.

2

KV Cache is the Dynamic Monster

Long context windows (32k) and multi-user concurrency cause memory to explode.

3

Quantization Saves Your Budget

INT4 & FP8 shrink memory 50–75%, letting massive models run on affordable GPUs.

UP NEXT: CHAPTER 04

What Actually Happens When You Run an LLM?

Why does processing the initial prompt take 100 milliseconds, but streaming the response words takes 10 seconds? The truth about the Prefill Phase vs. the Decode Phase!

Prefill: Compute-Bound (Matrix-Matrix) Decode: Memory-Bound (Matrix-Vector)