GPUs, Memory & LLM OOM Errors
Explained from First Principles

A precise, interactive guide to understanding why language models need GPUs, where memory goes during training and inference, and exactly what happens when it runs out.

Our Example Model
Qwen 2.5 7B
7.61 billion parameters
Minimum VRAM
15.2 GB
just to load the weights
Full Training Needs
120–170 GB
exceeds any single GPU
CPU Training Time
20–60 days
vs 30 min on an A100
Scroll to start from the beginning
Foundation — Start Here

What Is a Language Model?

Before we can understand why LLMs need GPUs or why they run out of memory, we need a clear picture of what they are and how they actually work.

The Basic Idea

A Large Language Model (LLM) is a deep neural network trained on vast amounts of text to model the statistical structure of language. At its core, the model learns a probability distribution over tokens: given a sequence of prior tokens, it estimates the likelihood of every possible next token and samples from that distribution to generate text.

What makes this powerful — and computationally expensive — is scale. To generalise across all of human language, code, mathematics, and reasoning, the model must compress an enormous amount of structured knowledge into its parameters: the numeric weights that define every linear transformation inside the network. Qwen2.5-7B has 7.61 billion of them. These weights are learned during training by repeatedly adjusting each parameter to reduce prediction error, a process that requires storing not just the weights themselves but gradients and optimiser states for every single one.

Parameter
A single number inside a neural network. A 7B model has 7 billion of them. These numbers collectively store everything the model "knows." During training, the model adjusts all parameters to reduce its prediction errors. During inference, they are fixed.
Token
The basic unit of text a language model processes. A token is roughly ¾ of a word on average. "Hello world" = 2 tokens. A 2,048-token context window ≈ 1,500 words ≈ 3 pages. Most modern LLMs support 32K–128K tokens of context.
Context window
The maximum number of tokens the model can "see" at once — its working memory. Everything outside this window is invisible to the model. Longer context windows are expensive because they dramatically increase memory usage, as we'll see.

What Happens Inside a Transformer

Almost all modern LLMs use the Transformer architecture (introduced by Google in 2017). Understanding its structure explains why GPUs are required and where memory goes.

1
Tokenization & Embedding
Input text is split into tokens. Each token is converted to a vector of numbers (its "embedding") — a list of ~3,584 numbers for Qwen2.5-7B. This happens on CPU and is fast. The embedding layer itself is part of the model's 7.61B parameters.
2
Attention Mechanism — "What should I pay attention to?"
Each token looks at all other tokens in the context and decides how much attention to pay each one. Mathematically, each token computes a Query (Q), Key (K), and Value (V) vector. Q asks "what am I looking for?", K says "what do I represent?", V says "what value do I carry?". The attention scores QKᵀ decide how much each token borrows from every other. This is the most memory-intensive operation in the model.
3
Feed-Forward Network — "Process what I found"
After attention, each token passes through a small neural network (two matrix multiplications with a nonlinearity in between). In Qwen2.5-7B, the hidden layer expands from 3,584 to 18,944 neurons and back — a 5× expansion. This is where ~2/3 of the model's parameters live.
4
Stacking Layers — Depth creates capability
Steps 2 and 3 together form one "transformer layer." Qwen2.5-7B has 28 layers stacked on top of each other. Each layer refines the token representations further. Deep models learn more abstract, powerful representations than shallow ones. Each layer adds to the memory bill.
5
Output Head — "What token comes next?"
The final layer's output goes through a linear projection to produce a score for every token in the vocabulary (152,064 tokens for Qwen2.5-7B). A softmax converts these to probabilities. The model predicts the next token by sampling from this distribution. This projection matrix is 3,584 × 152,064 = 546M parameters — about 7% of the entire model.
Note
The key insight: Every single step above involves multiplying large matrices of numbers together. Step 2 alone at 28 layers does hundreds of large matrix multiplications per token. This is why LLMs need GPUs — not because GPUs are "faster computers," but because GPUs are specifically designed for massive parallel matrix multiplication, which is the dominant operation in neural networks.

Training vs. Fine-tuning vs. Inference — Three Very Different Workloads

These three modes have completely different memory requirements. Understanding the difference is essential before we can understand OOM errors.

Inference
Running the model
You ask the model a question and it generates a response. The model's parameters are frozen — nothing is learned. Memory cost: weights + KV cache + small activation buffer. This is the cheapest mode.
Qwen2.5-7B: ~17–25 GB depending on context length
Fine-tuning
Teaching a new skill
You take an existing pretrained model and train it further on your specific data (e.g., your company's documents). Parameters are updated, so you need to store gradients and optimizer states. Memory cost: weights + activations + gradients + optimizer states.
Qwen2.5-7B (LoRA): ~34–36 GB
Full Training
Building from scratch
Training a model from random weights on trillions of tokens. All 7.61B parameters are updated every step. Memory cost is enormous: weights + gradients + optimizer states (each parameter needs 18 bytes total). The most expensive mode by far.
Qwen2.5-7B: ~156–165 GB — needs multi-GPU

Qwen2.5-7B — Our Concrete Example

Throughout this guide we use Qwen2.5-7B as our case study. It's a real, production-grade model small enough to fine-tune on a single high-end GPU, but large enough to exhibit all the memory challenges at scale. Here is its exact configuration from the official Hugging Face model card:

Qwen2.5-7B — Exact Architecture Configuration
Total parameters
7.61B
6.53B non-embedding
Hidden size
3,584
vector dimension per token
Transformer layers
28
attention + FFN each
Attention heads
28 Q / 4 KV
GQA — 4 KV heads only
Head dimension
128
3584 ÷ 28 = 128
FFN hidden size
18,944
5.3× hidden size
Vocabulary size
152,064
unique tokens
Max context
131,072
tokens (128K)
Warn
Common mistake: Many guides use hidden size 4,096 and 7.07B parameters for Qwen2.5-7B. Both are wrong — those are LLaMA-style numbers. Qwen2.5-7B uses hidden size 3,584 and has 7.61B total parameters. Using the wrong numbers inflates every activation estimate by 13–31%.
Chapter 1

CPU vs GPU — Two Fundamentally Different Machines

The CPU and GPU are both processors, but they are designed for completely different jobs. Understanding this architectural difference explains why you cannot simply use a CPU for LLM training — it's not a software preference, it's a hardware constraint.

What Is a CPU?

A CPU (Central Processing Unit) is the brain of your computer. It is designed for flexibility and low latency — it can switch rapidly between very different tasks: reading a file, making a network request, executing a database query, running your operating system. CPUs do this by having a small number of very powerful, sophisticated cores — each one capable of handling complex branching logic, speculative execution, out-of-order processing, and cache management.

A high-end server CPU like the AMD EPYC 9654 has 96 physical cores, each running at 3.7 GHz. Each core can execute up to 16 floating-point operations per clock cycle using AVX-512 SIMD instructions. Peak throughput: ~5.7 TFLOPS (trillion floating-point operations per second).

What Is a GPU?

A GPU (Graphics Processing Unit) was originally designed to render video game graphics — computing the color of every pixel on screen simultaneously. Pixels are independent: the color of pixel (100, 200) doesn't depend on pixel (300, 400). So instead of a few powerful sequential cores, GPUs use thousands of simple parallel cores that all do the same operation on different data simultaneously.

For AI workloads, this is perfect: multiplying each row of a matrix by each column (which is what a neural network layer does) is exactly the kind of operation where thousands of independent parallel calculations help enormously. An NVIDIA H100 has 16,896 CUDA cores and 528 Tensor Cores (specialized matrix-multiply hardware), plus 80 GB of extremely fast HBM3 memory on the same chip package.

CPU — The Generalist
AMD EPYC 9654 (example)
96
powerful, sophisticated cores — each handles complex logic
Peak FP32 throughput~5.7 TFLOPS
Memory bandwidth~50–100 GB/s (DDR5)
Memory locationMotherboard DIMMs (far from CPU)
Clock speed per core2.4–3.7 GHz
Designed forBranching, I/O, OS work, diversity
LLM GEMM (3584×3584)~80–150 ms per operation
GPU — The Parallel Engine
NVIDIA H100 SXM5
16,896
CUDA cores + 528 Tensor Cores — built for matrix math at scale
BF16 Tensor Core throughput~989 TFLOPS
Memory bandwidth3,350 GB/s (HBM3)
Memory locationOn-package, next to the die
Tensor Core: 16×16 MMA1 clock cycle
Designed forThroughput, matrix math, parallelism
LLM GEMM (3584×3584)~0.3–0.5 ms per operation
Key
The right mental model: A CPU is like having 96 brilliant experts who can each handle any task — writing, math, debugging, managing files — but each works on one thing at a time. A GPU is like having 16,896 factory workers who each do one simple operation — but all 16,896 of them simultaneously, on different pieces of data. LLM training is a factory job, not an expert job.
Core count — visualized at scale
CPU — 96 cores (each square = 1 core)
96 powerful, independent cores. Each handles complex logic.
GPU — 16,896 CUDA cores (each square ≈ 56 cores)
Thousands of simple cores, all running the same math simultaneously.

Why Matrix Multiplication Is the Key

Every single layer in a transformer does the same fundamental operation: matrix multiplication. Given an input vector (representing the current token) and a weight matrix (the learned knowledge), multiply them together. For a single linear layer in Qwen2.5-7B, this means multiplying a [3584]-element vector by a [3584 × 3584]-element matrix — 3584 × 3584 = 12.8 million multiply-add operations, just for one layer of one token.

With 28 layers and ~8 linear projections per layer, a single forward pass through Qwen2.5-7B involves approximately 3.2 billion multiply-add operations per token. On a CPU, even with AVX-512 vectorization, this is sequential. On a GPU, thousands of these multiply-adds happen simultaneously. A transformer layer that takes 80–150 ms on a 64-core EPYC takes 0.3–0.5 ms on an A100 — a 240× difference.

What Is VRAM and Why Can't You Just Use More RAM?

VRAM (Video RAM) is the GPU's dedicated memory. Unlike system RAM which sits on your motherboard connected via a bus, VRAM is physically on the same package as the GPU chip — stacked directly on top of it using a technology called HBM (High Bandwidth Memory). This proximity is what allows HBM3 to deliver 3,350 GB/s of bandwidth.

GPU compute units can only directly process data that is in VRAM. If a model's weights are in system RAM, they must be transferred over the PCIe bus (32–128 GB/s) before the GPU can use them. This creates a bottleneck: at 128 GB/s PCIe, transferring 15.2 GB of weights takes ~120ms — for every forward pass. At HBM3 speeds, the same transfer is essentially instantaneous. This is why you cannot substitute system RAM for VRAM.

Memory bandwidth — the data pipeline that feeds compute units

During inference, the model streams 15.2 GB of weights through the compute units for every token generated. If bandwidth is too low, the cores sit idle waiting for data — this is "memory-bandwidth bound." The GPU's HBM provides 67× more bandwidth than CPU DDR5.

Data
Why bandwidth matters more than FLOPS for inference: Generating one new token requires reading most of the 15.2 GB weights from memory. At 50 tokens/second, that's 760 GB/s of sustained memory reads. A CPU at 100 GB/s delivers ~7 tokens/second even if its compute units were infinitely fast — it's memory-starved, not compute-starved.

What Are Tensor Cores?

Tensor Cores are specialized hardware units inside NVIDIA GPUs (introduced with the Volta V100). While regular CUDA cores do one multiply-add per clock cycle, a Tensor Core performs an entire 16×16 matrix-multiply-accumulate (MMA) operation in one cycle. For the math: 16×16×16 = 4,096 individual multiply-adds, executed simultaneously in a single instruction.

This is why the H100's "BF16 throughput with Tensor Cores" (989 TFLOPS) is so much higher than its "FP32 CUDA core throughput" (~60 TFLOPS). The Tensor Cores are 15× more powerful for matrix math — but they only activate when the data is in BF16 or FP16 format. This is the primary reason precision format choice matters so much for both performance and memory usage.

Precision Formats

FP32, BF16, INT4 — The Memory-Speed Trade-off

Every number stored in the model — every weight, every activation, every gradient — is stored in a specific numeric format. The format controls two things simultaneously: how many bytes it occupies (memory) and how fast the GPU can compute with it (throughput). Smaller formats save memory and can be faster, but they lose precision — and in the wrong place, that destroys training quality.

What is a "bit"?
A bit is the smallest unit of data — a 0 or 1. 8 bits = 1 byte. FP32 uses 32 bits (4 bytes) per number. BF16 uses 16 bits (2 bytes). INT4 uses 4 bits (half a byte). Fewer bits = less memory but also less precision — fewer possible distinct values the number can represent.
What is "precision" and why does it matter for training?
Precision refers to how finely a number can be represented. The weight update during training is often tiny — on the order of 0.000001. If your number format can only represent ~100 distinct values between 0 and 1, that update gets rounded to zero and the model stops learning. FP32's 23-bit mantissa preserves it; BF16's 7-bit mantissa is often sufficient; INT4 would corrupt it.
Warn
T4 does not support BF16 natively. The T4 (Turing architecture, 2018) predates native BF16 Tensor Core support. When you try to use BF16 on a T4, it silently emulates it in software or falls back to FP16 — which requires "loss scaling" tricks to prevent gradient overflow. All newer GPUs (A100, A10G, RTX 3090, H100 — Ampere architecture 2020 and later) support BF16 natively via Tensor Cores. This matters when choosing a GPU for fine-tuning.

Mixed Precision Training — Why We Use Both FP32 and BF16

Modern LLM training uses a technique called Automatic Mixed Precision (AMP): the forward pass and gradient computation happen in BF16 (fast, memory-efficient), but the optimizer update happens in FP32 (slow, but numerically stable).

The reason: the Adam optimizer applies tiny adjustments to weights at every training step. These adjustments (learning_rate × gradient) are often on the order of 10⁻⁶. BF16's 7-bit mantissa only gives ~2–3 significant decimal digits of precision. A weight value of 1.2345678 receiving an update of 0.0000001 would be rounded to 1.2345678 unchanged in BF16 — the update is invisible. FP32's 23-bit mantissa preserves it. So we keep a "master copy" of all weights in FP32 for the optimizer step, then convert back to BF16 for the next forward pass. This "master weight copy" doubles the weight memory cost during training.

Chapter 2

Where Does GPU Memory Actually Go?

A model with 7.61B parameters is not just "15.2 GB." The moment you start training or even run inference with a long context, memory fills up from multiple sources simultaneously. Understanding each source is the key to diagnosing and preventing OOM errors.

The Six Memory Tenants

GPU VRAM during LLM work is occupied by up to six distinct objects. Some are always present; others only appear during training. OOM errors happen when the sum of all active objects exceeds the GPU's capacity.

1
Model Weights — the unavoidable baseline
The learned parameters of the model — all 7.61 billion of them — must be loaded into VRAM before the model can do anything. In BF16 (2 bytes per number): 7.61B × 2 = 15.2 GB. This is the floor for any operation. A T4 with 16 GB barely has room for these weights before anything else happens. Note: some models share weights between the embedding layer and the output head — Qwen2.5-7B does not, so the full 15.2 GB is always required.
2
Activations — the intermediate computation snapshots
Activations are the intermediate results produced at each layer during the forward pass. During training, all activations from all layers must be saved so the backward pass can use them to compute gradients — this is called "storing the computation graph."

Scaling with sequence length: The dominant activation cost inside the attention mechanism is the attention score matrix, shaped [seq × seq]. Without FlashAttention, this grows quadratically — doubling the sequence length quadruples this component. At seq=8192 across 28 layers, this matrix alone consumes ~7.4 GB. FlashAttention converts this from O(seq²) to O(seq) by computing attention in tiles that fit in GPU SRAM and never materialising the full matrix — making long-context training tractable. The remaining activation components (linear layer outputs, FFN intermediates, residuals) scale linearly with seq.

Scaling with batch size: Activation memory scales linearly with batch. Batch=8 needs exactly 8× the activation memory of batch=1. At batch=4, seq=2048, Qwen2.5-7B stores ~16.4 GB of activations — often more than the model weights.
3
Gradients — the learning signal
A gradient tells each parameter: "if you increase by a tiny amount, does the model's prediction improve or get worse, and by how much?" During the backward pass, the model computes one gradient per trainable parameter. For full training with all 7.61B parameters: 7.61B × 4 bytes (FP32) = 30.4 GB. With LoRA, you only compute gradients for the ~40M adapter parameters — reducing gradient memory from 30.4 GB to ~162 MB. Gradients are stored in FP32 (not BF16) because they must be numerically stable — small gradient values would be rounded away in BF16.
4
Optimizer States — the most expensive hidden cost
The Adam optimizer (standard for LLM training) maintains two extra arrays per trainable parameter beyond the gradient itself: the first moment (m) — a running average of recent gradients — and the second moment (v) — a running average of squared gradients. Together these allow Adam to adapt the learning rate per parameter, making training much more stable. The cost: 4 + 4 = 8 extra bytes per trainable parameter in FP32. For 7.61B parameters: 8 × 7.61B = 60.9 GB — just for the optimizer. This is why "the optimizer alone costs more than 4× the model weights" is true. With LoRA (40M trainable params), optimizer states drop to 320 MB.
5
KV Cache — inference's growing hidden cost
During inference, generating one new token requires attending to every previous token. Without caching, this forces recomputation of all prior Key and Value matrices — O(n²) total work for a sequence of n tokens. The KV Cache stores those matrices so each step only computes K and V for the single new token — reducing per-step attention work to O(n).

The trade-off: total KV cache size is O(n) in the number of cached tokens — it grows linearly with context length and batch size. For Qwen2.5-7B at 32K context with 32 concurrent users, the raw KV cache alone is ~60 GB — more than the model weights. This is why production serving at long context requires 80 GB GPUs even when quantisation brings the model weights to 4–6 GB.
6
Framework Overhead — the hidden tax
CUDA (NVIDIA's GPU programming framework) requires its own memory: the CUDA context (~300–500 MB), cuDNN workspace (the library for neural network operations), PyTorch's memory caching allocator, kernel compilation buffers, and serving engine state. This adds 1–8+ GB depending on the workload. It's also the source of memory fragmentation: as the training loop allocates and frees tensors of varying sizes, free memory becomes scattered in non-contiguous pieces. The allocator can report 5 GB "free" but fail to allocate a single 3 GB block because no contiguous chunk that large exists.

Memory Breakdown by Workload

Select a workload to see exactly which memory tenants are active, how large each is, and why. The stacked bar shows relative proportions; the table explains each component.

VRAM consumption — Qwen2.5-7B ~18 GB
ComponentFormulaSizeExplanation

The KV Cache — Deep Dive

The KV Cache deserves special attention because it surprises most engineers the first time they encounter it at scale. The formula below uses Qwen2.5-7B's actual architecture numbers — not approximations:

KV cache bytes = batch_size × seq_len × num_layers × 2 × num_kv_heads × head_dim × bytes_per_element Qwen2.5-7B actual values (BF16): num_layers = 28 num_kv_heads = 4 ← GQA reduces this from 28 to just 4 head_dim = 128 (= hidden_size / num_attention_heads = 3584 / 28) bytes = 2 (BF16) Per token, per request: 28 × 2 × 4 × 128 × 2 = 57,344 bytes = 56 KB per token per concurrent request
What is GQA (Grouped Query Attention)?
In standard Multi-Head Attention (MHA), every attention head has its own Key and Value matrices. With 28 heads, you cache 28 sets of KV. In Grouped Query Attention (GQA), multiple query heads share the same Key and Value. Qwen2.5-7B has 28 query heads but only 4 KV heads — a 7× reduction in KV cache size. Without GQA (full MHA), the 32K context KV cache would be 13 GB at batch=1 instead of 1.88 GB.
Why does batch size multiply the KV cache?
Each concurrent request (user) being processed at the same time has its own independent conversation history — its own keys and values for all its tokens. Serving 32 users simultaneously means 32 independent KV caches. At 32K context, that's 32 × 1.88 GB = 60.1 GB of KV cache alone. This is why production inference at long context requires 80 GB GPUs.
Sequence LengthBatch = 1Batch = 8Batch = 32Note
512 tokens0.03 GB0.23 GB0.94 GBShort chat — trivial
2,048 tokens0.12 GB0.94 GB3.76 GBStandard chat context — manageable
8,192 tokens0.47 GB3.76 GB15.03 GBLong document — batch starts to bite
32,768 tokens1.88 GB15.03 GB60.13 GBLong context serving — fills an 80 GB GPU
131,072 tokens7.52 GB60.13 GB240 GBFull context — requires multi-GPU serving

Why Batch Size and Sequence Length Cause Sudden OOM Crashes

The most common operational surprise: training runs stably for hours, then someone increases batch size from 4 to 8 — and it immediately crashes. This feels random but follows precise mathematical laws.

Batch Size
Batch size multiplies activations linearly
Every sample in a batch needs its own complete set of activations saved for backpropagation — there's no sharing. Batch=8 uses exactly 8× the activation memory of batch=1. Activation memory at batch=4, seq=2048: ~16.4 GB. At batch=8: ~32.9 GB. This is not gradual — it's a hard multiplier on every intermediate tensor.
Fix: Use gradient accumulation — process batch=1 multiple times and sum the gradients before updating. Same optimization effect, fraction of the memory.
Sequence Length
Attention without FlashAttention is O(n²)
The standard attention mechanism computes a score matrix of shape [seq × seq]. At seq=4096 this is 4096×4096 = 16.7M values × 28 layers × 2 bytes = 932 MB. At seq=8192 this quadruples to 3.7 GB. At seq=32K: 59 GB — impossible. This quadratic scaling is why FlashAttention (which avoids materializing this matrix) is not optional for long-context work.
Fix: Enable FlashAttention. It tiles attention into GPU SRAM chunks, eliminating the [seq × seq] matrix entirely.
Logit Spike
The logits spike during loss computation
Computing the cross-entropy loss requires a tensor of shape [batch × seq × vocab_size]. For Qwen2.5-7B with its 152,064-token vocabulary at batch=2, seq=2048: 2 × 2048 × 152,064 × 2 bytes = 1.25 GB of temporary storage — created and destroyed every step. This spike is brief but can be the straw that breaks the camel's back if you're near your memory limit.
Fix: Use chunked cross-entropy (Unsloth implements this) — computes the loss on 2048-token chunks, never building the full tensor.
Fragmentation
Fragmentation makes it worse over time
PyTorch's memory allocator reserves blocks from CUDA and reuses them. After thousands of training steps, free memory becomes fragmented — split into many small non-contiguous pieces. A training job that started with 6 GB headroom may OOM after 8 hours. The error says "3.5 GiB free" but a single 3.5 GB allocation fails because no contiguous block that large exists.
Fix: Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True in your environment to use a more fragmentation-resistant allocator.
OOM
The OOM error message decoded: When you see RuntimeError: CUDA out of memory. Tried to allocate 3.50 GiB (GPU 0; 79.20 GiB total capacity; 74.85 GiB already allocated; 2.94 GiB free; 76.34 GiB reserved in total by PyTorch) — the key numbers are: 74.85 GB actually in use, 76.34 GB reserved by PyTorch (includes fragmented free blocks it holds), only 2.94 GB is genuinely free, and a 3.5 GB contiguous block doesn't exist. The GPU is not actually "full" — it's fragmented.
Chapter 3 — Interactive

OOM Simulator — See It Happen Live

Drag the sliders to change batch size, sequence length, and training mode. Watch the memory bar grow, the GPU gauges fill, and the OOM error appear the moment you cross the limit. This is exactly what happens in real training runs.

Batch size — how many training samples are processed simultaneously in one forward+backward pass. Multiplies activation memory linearly.
Sequence length — number of tokens per sample. Multiplies activations and (without FlashAttention) multiplies attention score matrix quadratically.
Training mode — determines which memory components are active. QLoRA is cheapest (4-bit base); Full Training is most expensive (all optimizer states).
Batch size samples processed at once
4
Sequence length tokens per sample
2,048
Training mode which components are active
Memory breakdown
T4 · 16 GB
GDDR6 · 320 GB/s · No native BF16
/ 16 GB
OOM
TIGHT
FITS
A100 40GB
HBM2e · 1,555 GB/s · BF16 OK
/ 40 GB
OOM
TIGHT
FITS
A100 / H100 80GB
HBM2e/3 · 2,000–3,350 GB/s · BF16 OK
/ 80 GB
OOM
TIGHT
FITS
CUDA Out of Memory
Why does OOM happen even when "free" memory shows up in nvidia-smi?

PyTorch's caching allocator reserves memory blocks from CUDA and holds them in a pool for reuse — even after freeing a tensor, the memory stays "reserved" in the pool. When you need a new allocation, PyTorch first tries to find a free block in its pool. If the required size doesn't exist as a contiguous block (due to fragmentation), it requests more from CUDA. If CUDA has no more to give — even though the pool shows "free" memory — you get OOM. nvidia-smi shows PyTorch's reserved pool as "used," which makes it look like you're out when you're actually fragmented.

Chapter 4

How We Mitigate Memory Constraints

A rich ecosystem of techniques exists to reduce, redistribute, and avoid the memory costs we've seen. None of them make the GPU requirement disappear — they extend how far a given GPU can reach. Every technique trades memory for something: compute time, model quality, hardware complexity, or configuration effort.

Technique Overview

TechniqueWhat it reducesCost / trade-offBest used for
Gradient CheckpointingActivation memory (~8×)~25–33% slower training (recompute)Any long-sequence fine-tuning
LoRAGradients + optimizer statesOnly adapts via low-rank; may not match full FT qualityTask fine-tuning on strong base models
QLoRABase model weight storage (15.2→5 GB)Quantization + dequant overhead; quality check needed7B–34B adaptation on T4/A10G
FlashAttentionO(seq²) attention matrixHardware/kernel compatibilityAny sequence > 2K tokens
UnslothKernel overhead, logit spike, HBM trafficModel/kernel support coverageSingle-GPU LoRA/QLoRA fine-tuning
DeepSpeed ZeRORedundant optimizer/gradient/weight copiesCommunication overhead, setup complexityFull fine-tuning, multi-GPU
vLLM + PagedAttentionKV cache fragmentation wasteServing-engine integrationHigh-throughput production inference
Gradient AccumulationPeak activation memoryMore forward/backward passes per optimizer stepSimulating large batches on small VRAM
FA
FlashAttention
Eliminate the quadratic attention matrix via tiling

The problem: Standard attention computes a full [seq × seq] score matrix and writes it to HBM (slow, global GPU memory). At seq=32K in Qwen2.5-7B, this matrix across all layers would occupy ~241 GB — physically impossible on any single GPU.

FlashAttention's solution (Dao et al. 2022): Instead of computing the full matrix and storing it, tile the computation into blocks small enough to fit in GPU SRAM (the on-chip fast memory, 20–50 MB). Use an "online softmax" algorithm to combine tile results incrementally — mathematically equivalent to the full computation, but HBM reads are reduced from O(seq²) to O(seq²/SRAM_size).

What it saves and doesn't save: Eliminates the [seq × seq] matrix from HBM. Does NOT reduce activation memory from linear layers, the KV cache, or gradients. FlashAttention 3 (2024) further optimizes for H100's Tensor Memory Accelerator (TMA) hardware.

Why CPU can't do this: The algorithm requires GPU SRAM to be physically adjacent to compute units. CPU L1 cache (32–64 KB per core) is too small and structurally incompatible with the warp-level access patterns FlashAttention requires.

17×
less HBM traffic
3–7×
faster attention
GPU only
SRAM required
GC
Gradient Checkpointing
Trade compute for activation memory — recompute on demand

The problem: Backpropagation needs all intermediate activations from the forward pass. Storing all of them for 28 layers costs ~16.4 GB at batch=4, seq=2048 — often more than the weights themselves.

The solution: During the forward pass, only save activations at "checkpoint" boundaries (e.g., every 4 layers). Discard everything in between. During the backward pass, when a gradient is needed for a discarded layer, recompute that layer's forward pass from the nearest saved checkpoint on the fly.

The trade-off: You pay for ~30% of your forward computations twice (once during forward, once recomputed during backward). Training takes ~30% longer. But activation memory drops from ~16.4 GB to ~2 GB — an 8× reduction. For any fine-tuning at sequence length ≥ 2048, this is almost always worth it.

One line to enable: model.gradient_checkpointing_enable() in HuggingFace Transformers.

~8×
less activation memory
~30%
slower training
1 line
to enable
Lo
LoRA — Low-Rank Adaptation
Train 0.5% of parameters instead of all 7.61B

The insight (Hu et al. 2021): When fine-tuning a pretrained model on a new task, the required weight changes (ΔW) have low intrinsic rank — meaning they can be well-approximated by the product of two small matrices. Instead of updating the full W (e.g. [3584 × 3584] = 12.8M parameters), LoRA learns two small matrices A [3584 × r] and B [r × 3584] such that ΔW = A × B.

At rank r=16: A is [3584 × 16] = 57,344 params; B is [16 × 3584] = 57,344 params. Total per layer: 114,688 — just 0.9% of the original 12.8M. Applied across all Qwen2.5-7B linear layers: ~40M trainable parameters (0.53% of 7.61B).

Memory impact: Gradients shrink from 30.4 GB (full 7.61B params) to ~162 MB (40M adapter params). Adam optimizer states shrink from 60.9 GB to ~324 MB. Base model weights still occupy 15.2 GB in VRAM but receive no gradient updates.

Crucial caveat: Activation memory barely changes — the full forward pass still runs through all 28 frozen layers, generating the same intermediate tensors. Activations, not optimizer states, dominate LoRA memory. Gradient checkpointing is usually required alongside LoRA.

0.53%
params trained (r=16)
~61 GB
optimizer saved
A100 40GB
fits comfortably
QL
QLoRA — Quantized LoRA
4-bit frozen base + BF16 trainable adapters

The approach (Dettmers et al. 2023): Takes LoRA further by storing the frozen base model in 4-bit NF4 (NormalFloat4) instead of BF16. NF4 is a specially designed 4-bit data type with 16 quantization levels placed at positions that minimize error for normally distributed weights (which neural network weights typically are).

How it works in practice: The NF4 weights are stored in 4-bit but dequantized to BF16 immediately before each matrix multiplication. Computation always happens in BF16 on Tensor Cores — you're paying for the storage savings, not sacrificing compute precision. The dequantized copy is discarded after use; only the 4-bit version is stored persistently.

Actual size: The raw math gives 7.61B × 0.5 bytes = 3.8 GB. But NF4 requires per-64-element FP16 scale factors, double-quantization metadata, and allocator alignment overhead. Practical loaded size: 4.5–6 GB. Combined with LoRA adapters (~500 MB for rank=16), Unsloth, and gradient checkpointing, QLoRA for Qwen2.5-7B fits on a T4 at batch=2, seq=2048.

4.5–6 GB
base model (NF4)
T4 feasible
with Unsloth
BF16
compute precision
Un
Unsloth
Kernel rewrites in Triton — eliminate avoidable overhead

What it is: Unsloth reimplements the most memory-intensive operations in the training loop using custom Triton kernels (NVIDIA's GPU programming language for high-performance custom operations) instead of relying on generic PyTorch and HuggingFace PEFT implementations that were not designed with peak memory in mind.

Chunked cross-entropy loss: Standard PyTorch materializes the full [batch × seq × 152,064] logits tensor before computing loss — a 1.25 GB spike at batch=2, seq=2048. Unsloth computes the loss on 2048-token chunks, never holding the full tensor. Saves 1–2.5 GB per step with no quality impact.

Fused QKV projection: Standard attention computes Q, K, and V in three separate passes, each reading the full input from HBM. Unsloth fuses all three into one kernel: one HBM read, three outputs. ~1.5–2× speedup for this step alone.

Smart gradient checkpointing: Unsloth profiles each layer type's recompute cost and keeps activations that are expensive to recompute (attention outputs), discarding cheaper ones (MLP outputs). Finds a better memory/speed balance than blanket full checkpointing.

The boundary: Unsloth removes overhead, but cannot eliminate the fundamental memory requirements. Still bounded by base model size, activation memory from sequence length and batch, and GPU VRAM capacity.

2–2.5×
faster training
~60%
less peak VRAM
GPU only
Triton + CUDA
DS
DeepSpeed ZeRO
Partition redundant training state across all GPUs

The problem: Standard data-parallel training (the simplest multi-GPU approach) replicates the full model state on every GPU. With 8× A100 GPUs, the optimizer states alone (60.9 GB) exist on all 8 cards: 487 GB of identical redundant data. Every GPU holds the exact same thing.

ZeRO-1: Partition only optimizer states. Each GPU holds 1/8 of optimizer states (7.6 GB instead of 60.9 GB). All-gather states before the optimizer step. Weights and gradients still replicated.

ZeRO-2: Also partition gradients. Each GPU reduces-scatters gradients during backward, keeping only its 1/8 shard. ~8× combined reduction on gradient+optimizer memory.

ZeRO-3: Also partition model parameters. Each GPU holds 1/N of the weights. Before computing a layer, it all-gathers the weights it needs, computes, then discards. Enables training models larger than any single GPU's VRAM. ~50% more communication overhead than ZeRO-2.

ZeRO-Offload / ZeRO-Infinity: Extends partitioning to CPU RAM (256 GB–2 TB available) or NVMe SSD. Optimizer states live in CPU RAM and are fetched via PCIe during the optimizer step. 40–80% throughput penalty — use only when the model must fit at all costs.

Stage 1
optimizer partitioned
Stage 2
+ gradients
Stage 3
+ parameters

DeepSpeed ZeRO — Interactive Memory Visualizer

Each tab shows how memory distributes across 4 GPUs for a 9B-class model (18 GB weights, 24 GB optimizer, 18 GB gradients). Switch tabs to see exactly what gets partitioned at each ZeRO stage.

Model weights
Optimizer states
Gradients
Activations

4 GPUs — standard data-parallel. Every GPU holds a complete, identical copy of everything.

GPU 0
Model weights 18 GB
Optimizer states 24 GB
Gradients 18 GB
Activations ~4 GB
GPU 1
Model weights 18 GB
Optimizer states 24 GB
Gradients 18 GB
Activations ~4 GB
GPU 2
Model weights 18 GB
Optimizer states 24 GB
Gradients 18 GB
Activations ~4 GB
GPU 3
Model weights 18 GB
Optimizer states 24 GB
Gradients 18 GB
Activations ~4 GB
64 GB
per GPU
0x
memory saved
minimal
communication overhead
All 4 GPUs hold identical data. 75% of the optimizer and gradient memory is pure redundancy. For a 9B model this approach requires 64 GB per GPU — impossible on an A100 40GB before activations are even counted.

ZeRO-1 — Partition optimizer states only. Weights and gradients remain fully replicated on every GPU.

GPU 0
Model weights 18 GB
Optim shard 0 — 6 GB (1/4)
Gradients 18 GB
Activations ~4 GB
GPU 1
Model weights 18 GB
Optim shard 1 — 6 GB (1/4)
Gradients 18 GB
Activations ~4 GB
GPU 2
Model weights 18 GB
Optim shard 2 — 6 GB (1/4)
Gradients 18 GB
Activations ~4 GB
GPU 3
Model weights 18 GB
Optim shard 3 — 6 GB (1/4)
Gradients 18 GB
Activations ~4 GB
46 GB
per GPU
~1.4x
memory saved
low
communication overhead
Adam's two state arrays (momentum m and variance v) are each the full size of the model. Splitting them 4 ways saves 18 GB per GPU. Weights and gradients remain fully replicated — the forward pass requires no additional inter-GPU communication. Rarely used alone: ZeRO-2 saves more for barely any extra overhead.

ZeRO-2 — Partition optimizer states and gradients. Weights still replicated. Recommended default for LoRA fine-tuning.

GPU 0
Model weights 18 GB
Optim shard 0 — 6 GB (1/4)
Grad shard 0 — 4.5 GB (1/4)
Activations ~4 GB
GPU 1
Model weights 18 GB
Optim shard 1 — 6 GB (1/4)
Grad shard 1 — 4.5 GB (1/4)
Activations ~4 GB
GPU 2
Model weights 18 GB
Optim shard 2 — 6 GB (1/4)
Grad shard 2 — 4.5 GB (1/4)
Activations ~4 GB
GPU 3
Model weights 18 GB
Optim shard 3 — 6 GB (1/4)
Grad shard 3 — 4.5 GB (1/4)
Activations ~4 GB
32.5 GB
per GPU
~2x
memory saved
moderate
communication overhead
The practical sweet spot. Weights are still fully replicated so the forward pass requires no extra GPU-to-GPU communication. During the backward pass, GPUs use reduce-scatter to exchange gradient shards — each GPU then keeps only its own 1/N slice. ZeRO-3 + LoRA can trigger "meta tensor" bugs; ZeRO-2 avoids these entirely.

ZeRO-3 — Partition everything including model weights. Every forward pass layer requires an all-gather from all GPUs.

GPU 0
Weight shard 0 — 4.5 GB (1/4)
Optim shard 0 — 6 GB (1/4)
Grad shard 0 — 4.5 GB (1/4)
Activations ~4 GB
GPU 1
Weight shard 1 — 4.5 GB (1/4)
Optim shard 1 — 6 GB (1/4)
Grad shard 1 — 4.5 GB (1/4)
Activations ~4 GB
GPU 2
Weight shard 2 — 4.5 GB (1/4)
Optim shard 2 — 6 GB (1/4)
Grad shard 2 — 4.5 GB (1/4)
Activations ~4 GB
GPU 3
Weight shard 3 — 4.5 GB (1/4)
Optim shard 3 — 6 GB (1/4)
Grad shard 3 — 4.5 GB (1/4)
Activations ~4 GB
19 GB
per GPU
~4x
memory saved
high
communication overhead
Maximum savings. Before computing each transformer layer, GPUs all-gather the weight shard they need, compute the layer, then discard the gathered copy. Every single layer in both forward and backward passes triggers inter-GPU communication — adding 20–40% overhead. Essential for full fine-tuning of 70B+ models where the weights alone exceed any single GPU's VRAM.
Baseline
64 GB
Partitioned: nothing
Comm: all-reduce only
Throughput: fastest
ZeRO-1
46 GB
Partitioned: optimizer
Comm: low overhead
Throughput: near-identical
ZeRO-2 — Recommended
32.5 GB
Partitioned: optim + grad
Comm: moderate (backward)
Throughput: slight drop
ZeRO-3
19 GB
Partitioned: everything
Comm: high (every layer)
Throughput: 20–40% slower
When to use each stage — Qwen2.5-7B / 9B class models, 4 GPUs
ZeRO-1Rarely the right choice alone. ZeRO-2 saves significantly more memory for barely any additional communication overhead. Skip to Stage 2.
ZeRO-2Default recommendation for LoRA SFT and DPO on 9B–70B models. Weights are replicated so the forward pass is fast. Use this unless you hit VRAM limits with Stage 2.
ZeRO-3Use for full fine-tuning of 70B+ models, or when model weights alone exceed single-GPU VRAM at Stage 2 settings.
ZeRO-3 + LoRAAvoid unless absolutely necessary. Known "meta tensor" bugs in several DeepSpeed versions cause silent errors. Use ZeRO-2 + LoRA instead.
vL
vLLM + PagedAttention — Production Inference Efficiency
Eliminate KV cache fragmentation waste during high-concurrency serving
The problem: With 100 concurrent users, each user's conversation has a different length. Pre-allocating the maximum possible KV cache (131K tokens = 7.5 GB) for every user wastes enormous VRAM. But dynamic allocation leads to fragmentation — free space scattered in small non-contiguous pieces that can't serve new requests even when aggregate free memory is sufficient.
PagedAttention: Inspired by OS virtual memory paging, vLLM divides the KV cache into fixed-size "pages" (blocks of 16–32 tokens). Pages are allocated on demand and tracked with a page table. The attention kernel reads from non-contiguous pages via the page table — near-zero fragmentation. Result: 2–4× more concurrent users on the same GPU, 2–24× throughput increase vs naive serving.
Why 80 GB GPUs Remain the Enterprise Minimum — Even After All Optimizations
Long-context production serving: KV cache at 32K context, batch=32 is ~60 GB raw. Add 15.2 GB weights + overhead: exceeds 80 GB. You need 80 GB just for the cache — quantization helps weights but not KV cache.
Production batch throughput: Serving 100+ concurrent users requires large batches. Batch=32 at 8K context hits 35–43 GB before overhead. A 40 GB card can't handle this while also holding weights.
Larger models (13B+): Qwen2.5-14B at BF16 = 28 GB weights. Leaves only 12 GB on a 40 GB card for KV cache and activation overhead — not enough for production throughput.
High-rank LoRA at long sequences: LoRA rank=64, batch=4, seq=8192 produces ~65.8 GB activation memory. The 80 GB card is the minimum for serious long-context fine-tuning, even with gradient checkpointing.

What Fits on Each GPU — Qwen2.5-7B

GPUVRAMBF16 Inference
(run the 7B model in BF16)		4-bit Inference
(run 7B quantised to INT4/NF4)		QLoRA Fine-tune
(adapt 7B, 4-bit base, LoRA adapters)		LoRA Fine-tune
(adapt 7B, BF16 base, LoRA adapters)		Full Fine-tune
(update all 7.61B params)
T416 GBToo tightDev/small batchUnsloth requiredOOMNo
A10G / L424 GBSmall batch onlyComfortableComfortableCareful settingsNo
A100 40GB40 GBComfortableHigh throughputLarge seq+batchComfortableNot enough
A100 80GB80 GBProduction tierMax concurrencyAny configLong contextNeeds ZeRO/FSDP
H100 80GB80 GBOptimalOptimalOptimalOptimalWith ZeRO
2× A100 80GB160 GB70B inference70B inference34B LoRA34B LoRA7B with ZeRO-2
8× H100640 GB70B+ models70B+ models70B fine-tune70B fine-tuneFull 70B training
Inference = running the model to generate text, parameters frozen, no learning. Fine-tune (QLoRA/LoRA) = adapting the model to new data; only adapter weights updated (40M params at r=16), base model frozen. Full fine-tune = updating all 7.61B parameters; requires gradients and optimizer states for every weight. 70B inference means running a 70B parameter model like Llama-3-70B or Qwen2.5-72B — weights alone are ~144 GB in BF16, requiring at least 2× A100 80GB.
Chapter 5

What Happens If You Use a CPU?

This is not a theoretical question. Teams sometimes attempt CPU training when GPU quota is unavailable, or assume a fast CPU server can substitute. The answer: the code runs correctly. The job finishes in 2027.

The Throughput Gap — Concrete Numbers

The fundamental issue is not that CPUs can't do the math — they can. The issue is how long it takes. LLM training is dominated by large matrix multiplications. Let's measure one: a 3584×3584 GEMM (General Matrix Multiply) — the dominant operation in each Qwen2.5-7B transformer layer.

CPU — OpenBLAS on 64-core EPYC
~100 ms
per 3584×3584 GEMM operation
A single transformer layer has ~8 GEMM operations × 28 layers = 224 per forward pass. At 100 ms each: ~22 seconds just for the linear projections. Per step (forward + backward): ~1–20 minutes.
GPU — cuBLAS on A100 (BF16)
~0.4 ms
per 3584×3584 GEMM operation
Same 224 GEMMs per forward pass: ~90 ms total. Full forward + backward step: ~150–250 ms. 10,000 steps: ~30 minutes. The difference is not "faster" — it's a different category of hardware.
Time per training step — Qwen2.5-7B, batch=1, seq=512 (order-of-magnitude estimates)
H100 SXM
~100ms
~100 ms
A100 80GB
~200ms
~200 ms
A10G / L4
~400ms
~400 ms
T4 16GB
~900ms
~900 ms
64-core EPYC
10–20 minutes
~15 min
16-core i9
35–70 minutes per step
~50 min
Slow
The real-world consequence: A QLoRA fine-tuning run on Qwen2.5-7B with 10,000 steps takes ~30 minutes on an A100 80GB. On a 64-core EPYC CPU: 10,000 steps × 15 min/step = ~104 days. On a 16-core workstation CPU: potentially 8 months. Teams attempting CPU fine-tuning invariably give up or exhaust their compute budget long before the model converges. This is not pessimism — it's arithmetic.

The Memory Bandwidth Bottleneck — Why CPUs Are Doubly Crippled

The throughput gap has two independent causes, not one. Even if you somehow solved the TFLOPS problem on a CPU, you would immediately hit the bandwidth wall — and vice versa. These are separate, compounding constraints.

Problem 1 — Compute Throughput

A 64-core CPU can do ~5.7 TFLOPS of FP32 math. An A100 does 312 TFLOPS in BF16 via Tensor Cores — a 55× gap. For a 3,584 × 3,584 matrix multiply: CPU takes 80–150 ms, A100 takes 0.3–0.5 ms. A transformer layer has 8 such GEMMs × 28 layers = 224 GEMMs per forward pass.

Problem 2 — Memory Bandwidth (often worse)

LLM inference is memory-bandwidth bound, not compute-bound. Even if CPUs had enough TFLOPS, they couldn't deliver weights to the compute units fast enough. At 50 tokens/second, inference needs 760 GB/s of sustained reads. CPU DDR5 delivers ~100 GB/s maximum — 7.6× too slow regardless of how many cores you have.

Why adding more CPU cores does not help

For matrix multiplication, parallelism across thousands of simultaneous multiply-adds is what matters — not sequential speed. All CPU cores share the same memory bus. Adding more cores increases compute pressure on the same fixed ~100 GB/s DDR5 bandwidth, making the bandwidth bottleneck worse, not better. A 1,000-core hypothetical CPU would still be memory-starved: 1,000 cores waiting for data from a 100 GB/s bus is slower per-core than 100 cores doing the same.

GPU HBM solves both problems simultaneously: each GPU die has its own dedicated HBM stack (3,350 GB/s on H100), physically adjacent to the Tensor Cores. The bandwidth scales with the GPU, not with a shared motherboard bus. This co-location is the fundamental architectural advantage that cannot be replicated by adding more CPUs.

Why Can't You Just Use More CPU Cores?

Adding more CPU cores does not fix the bandwidth problem — it compounds it. All cores share the same memory bus. More cores means more contention for the same ~100 GB/s DDR5 bandwidth. Even with 1,000 CPU cores, you would be memory-starved: 1,000 cores waiting for data arriving at 100 GB/s is far slower per-core than 16,896 GPU cores fed by 3,350 GB/s HBM. The bandwidth is not shared on the GPU — it scales with the chip architecture.

CPU
GEMM (3584×3584) BF16~80–150 ms
Memory bandwidth~50–100 GB/s (DDR5)
FlashAttention supportNone — SRAM too small
Tensor CoresNone
bitsandbytes (INT4/INT8)Severely degraded
UnslothNot supported (Triton = GPU)
Multi-node comms (NCCL)Falls back to Gloo/MPI, ~100× slower
10K step fine-tune (7B)~20–100 days
Appropriate for LLMsTokenization, preprocessing, orchestration, llama.cpp local inference
GPU (A100 80GB)
GEMM (3584×3584) BF16~0.3–0.5 ms (240× faster)
Memory bandwidth2,000 GB/s (HBM2e)
FlashAttention supportFull — 20 MB SRAM on-chip
Tensor Cores432 (BF16 MMA, 16×16×16 per cycle)
bitsandbytes (INT4/INT8)Full — custom CUDA kernels
UnslothFull — 2–2.5× speedup
Multi-GPU comms (NCCL)NVLink 600 GB/s, all-reduce in ms
10K step fine-tune (7B)~30 minutes (QLoRA)
Appropriate for LLMsTraining, fine-tuning, inference, serving

Why the Entire Software Stack Assumes GPU

FlashAttention is architecturally impossible on CPU
FlashAttention tiles attention into blocks that fit in GPU SRAM (20–50 MB on-chip, ~10 TB/s effective bandwidth). CPU L1 cache is 32–64 KB per core — 500–800× smaller. The algorithm's access patterns require hundreds of threads accessing the same SRAM tile simultaneously, which the CPU cache coherency protocol cannot support efficiently. There is no CPU port of FlashAttention. CPU attention is always the naive O(n²) version.
Fused kernels don't exist on CPU
GPU training fuses attention + softmax + dropout in one Triton/CUDA kernel — one pass through HBM, three operations done. CPU implementations call each function separately, writing intermediate results to RAM between calls. For an attention layer, this triples the memory traffic. PyTorch's CPU attention is a sequence of separate OpenBLAS and NumPy calls, each with full RAM round-trips and Python dispatch overhead.
Multi-node GPU communication vs CPU
Multi-GPU training uses NCCL (NVIDIA Collective Communications Library) with NVLink: 600 GB/s direct GPU-to-GPU bandwidth. Each all-reduce (gradient synchronization) across 8 GPUs takes milliseconds. Multi-CPU distributed training falls back to MPI over Ethernet: 10–100 Gb/s (1–12 GB/s effective). Every gradient sync is 50–600× slower, bottlenecking each step before compute even starts.
When CPU is genuinely the right tool
llama.cpp local inference: Quantized models (Q4_K_M) at 15–25 tokens/sec on M3 Max — fine for personal use and development testing. Tokenization: HuggingFace tokenizers (Rust-based, CPU-only) tokenize millions of samples efficiently. Data preprocessing and dataset curation: Cleaning, filtering, formatting — entirely CPU. Model evaluation (non-time-sensitive): Running MMLU, HumanEval, or perplexity benchmarks on a fine-tuned model offline, where speed is not a constraint.
Summary & Reference

Decision Framework & Complete Glossary

Everything you need to make hardware decisions and speak precisely about GPU memory — without looking anything up.

The One-Paragraph Summary

An LLM does not fail because the code is careless — it fails because the GPU is asked to hold several large objects simultaneously: model weights activations gradients optimizer states KV cache. Batch size and sequence length are the two knobs that most reliably push a stable configuration over the edge. A T4 (16 GB) can handle 4-bit QLoRA fine-tuning with careful settings — it is useful for PoC and development, not production serving. The A100 or H100 at 80 GB is the practical enterprise tier for long-context serving, larger batches, BF16 inference, and full fine-tuning pipelines. GPU infrastructure is not an experiment expense — it is a product dependency with a concrete technical foundation.

Memory Budget — Qwen2.5-7B Quick Reference
ScenarioVRAMMinimum GPU
Inference BF16, 2K ctx, batch=117–19 GBA10G
Inference BF16, 32K ctx, batch=119–24 GBA10G
Inference BF16, 8K ctx, batch=3235–43 GBA100 80GB
4-bit inference, 8K ctx, batch=17–11 GBT4 (comfortable)
QLoRA r=16, batch=2, seq=204811–18 GBT4 + Unsloth
LoRA r=16, batch=4, seq=204834–36 GBA100 40GB
LoRA r=64, batch=4, seq=409670–80 GBA100 80GB
Full FT, AdamW, batch=4, seq=2048~160 GB2× A100 80GB
GPU Selection Guide by Workload
WorkloadMinimumRecommended
Dev / debugging / experimentsT4 16GBRTX 4090 24GB
QLoRA fine-tune 7BT4 + UnslothA10G / L4
LoRA fine-tune 7BA10G 24GBA100 40GB
LoRA fine-tune 13–34BA100 40GBA100 80GB
Full fine-tune 7B2× A100 80GB4× A100 80GB
Production inference 7B (BF16)A10G 24GBA100 40GB
Long-context serving (32K+)A100 80GBH100 80GB
Production inference 70B (4-bit)2× A100 40GB2× A100 80GB
Pre-training from scratch (7B)8× A100 80GB16–64× H100

Formula Reference

Weight Memory
weight_mem = params × bytes / 1e9
FormatBytes/paramQwen2.5-7B
FP32430.4 GB
BF16215.2 GB
INT817.6 GB
NF4 (raw)0.53.8 GB raw / 4.5–6 GB loaded
KV Cache
kv = B × S × L × 2 × Hkv × D × bytes / 1e9 Qwen2.5-7B: L=28, Hkv=4, D=128, bytes=2 Per token, per request = 57,344 bytes = 56 KB
Contextbatch=1batch=8batch=32
2K0.12 GB0.94 GB3.76 GB
8K0.47 GB3.76 GB15 GB
32K1.88 GB15 GB60 GB
Full Training — AdamW AMP (per trainable param)
BF16 weights: 2 bytes FP32 master: 4 bytes ← doubles weight footprint FP32 gradient: 4 bytes Adam m (FP32): 4 bytes Adam v (FP32): 4 bytes ───────────────────────── Total: 18 bytes / trainable param
Qwen2.5-7B: 7.61B × 18 = ~137 GB before activations
Activation Memory & Effective Batch
act ≈ B × S × hidden × L × factor × 2 / 1e9 factor ≈ 10 (attention, FFN×5.3, norms, residuals) Qwen2.5-7B (B=4, S=2048): 4 × 2048 × 3584 × 28 × 10 × 2 / 1e9 ≈ 16.4 GB effective_batch = micro_batch × grad_accum_steps × num_gpus
LoRA at r=16: ~40M trainable params → optimizer states ≈ 485 MB vs 60.9 GB for full training

Complete Glossary

Activation
The intermediate output produced at each layer during the forward pass. All activations must be saved during training so the backward pass can compute gradients via the chain rule. The primary driver of memory spikes when batch size or sequence length increases.
AMP (Automatic Mixed Precision)
Training strategy using BF16 for forward/backward passes (fast, memory-efficient) while keeping FP32 master weights and optimizer states (numerically stable). The modern default for LLM training. Implemented with one line in PyTorch.
Attention (Self-Attention)
The mechanism that lets each token in a sequence "look at" all other tokens and weight their contributions. Computes Query, Key, and Value matrices; scores = Q×Kᵀ; output = softmax(scores)×V. The most memory-intensive operation in a transformer.
Backpropagation
The algorithm for computing gradients. Works backward through the computation graph using the chain rule, calculating how much each parameter contributed to the prediction error. Requires all forward-pass activations to be stored, which is why activations are so expensive during training.
BF16 (Brain Float 16)
A 16-bit floating-point format with the same 8-bit exponent as FP32 but only 7 mantissa bits. Has the same numerical range as FP32 (no overflow/underflow like FP16) at half the memory. The standard for modern LLM training. Not natively supported on T4 (Turing architecture).
CUDA
NVIDIA's parallel computing platform and programming model. All GPU-accelerated deep learning libraries (PyTorch, cuBLAS, cuDNN, FlashAttention, Unsloth, bitsandbytes) are built on CUDA. The reason "GPU support" effectively means "NVIDIA GPU support" in the LLM ecosystem.
FlashAttention
An IO-aware attention algorithm (Dao et al. 2022) that tiles computation into GPU SRAM blocks, avoiding materializing the full [seq × seq] attention matrix in HBM. Reduces HBM traffic 17×, enables sequences of 100K+ tokens. GPU-only — requires on-chip SRAM adjacent to compute units.
FP32 (Full Precision)
32-bit floating-point format. 4 bytes per number, 23-bit mantissa, ~7 significant decimal digits. Used for optimizer states and master weights where small incremental updates must be preserved. Never used for the main forward pass in modern training — too slow and memory-hungry.
GEMM (General Matrix Multiply)
The fundamental mathematical operation in neural networks: C = A×B where A and B are large matrices. Transformer layers are almost entirely sequences of GEMMs. This is the operation GPUs are specifically optimized for via Tensor Cores.
Gradient
The partial derivative of the loss with respect to a trainable parameter — tells the optimizer how much and in which direction to adjust each parameter to reduce prediction error. Stored in FP32 for numerical stability. In full training: one gradient per parameter (30.4 GB for 7.61B params in FP32).
Gradient Accumulation
Processing multiple micro-batches sequentially and summing their gradients before running the optimizer step. Allows simulating a large effective batch size (better training stability) without allocating memory for all samples simultaneously. Decouples batch size from activation memory.
Gradient Checkpointing
A training technique that discards most intermediate activations during the forward pass and recomputes them during backprop. Reduces activation memory ~8× at the cost of ~30% slower training. One line to enable: model.gradient_checkpointing_enable().
GQA (Grouped Query Attention)
A variant of multi-head attention where multiple query heads share the same Key and Value heads. Qwen2.5-7B has 28 query heads but only 4 KV heads — reducing KV cache size 7× compared to full MHA. Used by most modern LLMs to make long-context inference tractable.
HBM (High Bandwidth Memory)
The memory technology used in data center GPUs. Memory dies are stacked vertically and placed on the same package as the GPU die, connected via thousands of parallel lanes. H100 HBM3: 3,350 GB/s vs DDR5's 50–100 GB/s. The physical proximity is what enables the bandwidth advantage.
KV Cache
Stored Key and Value attention matrices for all previous tokens, maintained during autoregressive inference to avoid recomputation. Grows with every token generated. For Qwen2.5-7B BF16: 56 KB per token per concurrent request. At 32K context with 32 users: ~60 GB.
LoRA (Low-Rank Adaptation)
Parameter-efficient fine-tuning method (Hu et al. 2021). Instead of updating full weight matrices, trains small low-rank adapters ΔW = A×B where rank r ≪ hidden_size. For Qwen2.5-7B at r=16: ~40M trainable params (0.53%), reducing optimizer states from 60.9 GB to 485 MB.
NF4 (NormalFloat4)
A 4-bit data type designed for QLoRA. Places 16 quantization levels at optimal positions for normally distributed data (neural network weights). Stored in 4-bit but dequantized to BF16 for every matrix multiply. Practical loaded size for 7.61B params: 4.5–6 GB (includes per-block scale factors and metadata).
NVLink
NVIDIA's proprietary high-speed GPU-to-GPU interconnect. NVLink 4 (H100): 900 GB/s bidirectional per GPU pair. Enables fast gradient synchronization in multi-GPU training. Compared to PCIe (128 GB/s), NVLink is 7× faster — critical for ZeRO-3 which does frequent all-gathers of model parameters.
OOM (Out of Memory)
The CUDA error thrown when the GPU memory allocator cannot satisfy a new allocation request. Can occur even when nvidia-smi shows free memory, due to fragmentation — free memory scattered in non-contiguous pieces too small for the requested allocation. The most common failure mode in LLM training.
Optimizer State
Extra tensors maintained by the optimizer beyond the weights themselves. Adam/AdamW keeps first moment (m, moving average of gradients) and second moment (v, moving average of squared gradients) in FP32 — 8 bytes per trainable parameter. For 7.61B parameters: 60.9 GB of optimizer state alone.
PagedAttention
vLLM's KV cache management technique inspired by OS virtual memory paging. Divides the KV cache into fixed-size pages (16–32 tokens each), allocated on demand and tracked with a page table. Eliminates fragmentation waste, allowing 2–4× more concurrent requests on the same GPU.
Parameter
A single learnable number in the neural network. Qwen2.5-7B has 7.61 billion parameters. These numbers collectively encode everything the model "knows." In BF16: 2 bytes each = 15.2 GB total. In training, each parameter needs up to 18 bytes total (weights + gradients + optimizer states).
PCIe (PCI Express)
The bus connecting the CPU to the GPU on a server motherboard. PCIe 4.0 ×16: ~32 GB/s; PCIe 5.0 ×16: ~64 GB/s. The bottleneck for ZeRO-Offload (CPU RAM offloading) — transferring 15.2 GB model weights over PCIe at 64 GB/s takes ~240ms per forward pass, which is why CPU offload dramatically slows training.
QLoRA
Quantized LoRA (Dettmers et al. 2023). Stores the frozen base model in 4-bit NF4 (reducing base weight VRAM from 15.2 GB to 4.5–6 GB) while training BF16 LoRA adapters on top. The 4-bit weights are dequantized to BF16 for compute. Makes 7B fine-tuning feasible on a 16 GB T4 with Unsloth.
Tensor Core
Specialized hardware unit in NVIDIA GPUs (Volta/V100 and later) that performs a 16×16 matrix-multiply-accumulate in one clock cycle (4,096 individual multiply-adds). Requires BF16/FP16/INT8 inputs. Provides 15× more throughput than regular CUDA cores for matrix math — the primary reason BF16 training is much faster than FP32 on modern GPUs.
Token
The atomic unit of text a language model processes. Roughly ¾ of a word on average. "Hello world" = 2 tokens. A 2,048-token context ≈ 1,500 words ≈ 3 pages of text. Qwen2.5-7B has a vocabulary of 152,064 possible tokens. Sequence length (measured in tokens) is the primary driver of KV cache size.
Transformer
The neural network architecture underlying all modern LLMs (introduced by Google in 2017). Consists of stacked layers, each containing a self-attention block (tokens look at all other tokens) and a feed-forward network (per-token processing). Qwen2.5-7B has 28 transformer layers. All the memory challenges in this guide stem from transformer architecture choices.
Triton
NVIDIA's open-source GPU kernel programming language — a higher-level alternative to raw CUDA for writing custom GPU operations. Used by Unsloth to implement fused attention, chunked cross-entropy, and custom LoRA backward kernels. Triton compiles to PTX (NVIDIA's assembly) and enables expert-level GPU optimizations without writing raw CUDA C++.
VRAM (Video RAM)
The GPU's dedicated memory. Physically located on the GPU package (as HBM) or on the GPU board (GDDR6 for T4). GPU compute units can only directly process data in VRAM. Transfer from system RAM to VRAM goes over PCIe (slow). T4: 16 GB GDDR6. A100: 40 or 80 GB HBM2e. H100: 80 GB HBM3.
ZeRO (Zero Redundancy Optimizer)
Microsoft DeepSpeed's technique for eliminating redundant model state storage in data-parallel training. Stage 1 partitions optimizer states; Stage 2 adds gradients; Stage 3 adds model parameters. Each stage reduces per-GPU memory by 1/N (where N = number of GPUs) for the partitioned components.
Ref
Sources and verification: All numbers in this guide are cross-referenced against: Qwen2.5-7B config.json on Hugging Face (parameter count 7.61B, hidden size 3584, 4 KV heads, 28 layers); NVIDIA H100/A100/T4 official product specifications; QLoRA paper (Dettmers et al. 2023, arXiv:2305.14314); FlashAttention-2 (Dao et al. 2023, arXiv:2307.08691); DeepSpeed ZeRO documentation; vLLM PagedAttention paper (Kwon et al. 2023). Memory estimates are engineering approximations — actual usage depends on framework version, kernel implementation, allocator behavior, and serving configuration.