Two guides in one logical sequence — first build the mental model of what fine-tuning actually does, then learn how QLoRA makes it feasible on a single GPU.
A pre-trained LLM already knows language. Fine-tuning teaches it your specific task — by showing it examples of the behavior you want.
An LLM like Qwen 2.5 Coder has been pre-trained on billions of tokens of text and code. It understands syntax, semantics, and patterns. But it doesn't know your coding style, your domain, or how you want it to respond.
Fine-tuning bridges this gap. You provide training pairs — examples of inputs and desired outputs — and the model adjusts its internal weights to produce similar outputs for similar inputs.
But how exactly does a sequence of JSON messages turn into weight updates inside a neural network? That's what this guide explains, step by step.
We use LoRA (Low-Rank Adaptation) during fine-tuning, which means only a small set of adapter weights are updated while the original billions of parameters stay frozen. Part 2 of this guide explains exactly how that works.
Every model family expects a specific format for structuring conversations. Qwen 2.5 uses ChatML — and your training data must follow it exactly.
The ChatML format wraps every message with special tokens: <|im_start|> marks the beginning and <|im_end|> marks the end. Each message is tagged with a role: system, user, or assistant.
<|im_start|>system You are a helpful coding assistant.<|im_end|> <|im_start|>user Write a Python function to add two numbers.<|im_end|> <|im_start|>assistant def add(a, b): return a + b<|im_end|>
You don't format this manually. When using HuggingFace TRL, the tokenizer's apply_chat_template() handles all formatting automatically. You just provide structured JSON.
Structure your training examples as a list of messages. TRL's SFTTrainer handles the rest.
Each training example is a dictionary with a "messages" key containing a list of role-content pairs:
{
"messages": [
{"role": "system", "content": "You are a helpful coding assistant."},
{"role": "user", "content": "Add two numbers in Python"},
{"role": "assistant", "content": "def add(a, b):
return a + b"}
]
}
Common gotcha: The column must be named "messages" — not "conversations". TRL specifically looks for this key to trigger automatic chat template application.
Training pairs have different lengths. When batched, shorter sequences need pad tokens. Three strategies exist:
| Strategy | How It Works | Best For |
|---|---|---|
| Naive Padding | Pads each batch to longest sequence. Simple but wastes compute on pad tokens. | Long examples |
| Packing | Concatenates examples into one sequence via packing=True in SFTConfig. | Short examples (2–5× speedup) |
| Unsloth | Optimized fused kernels for padding, attention, and loss masking. | Maximum efficiency |
The tokenizer converts human-readable text into a sequence of integer IDs that the model can process.
After apply_chat_template() + tokenizer.encode(), our example becomes a sequence of integers. Watch each token appear:
Each token maps to a unique integer ID in Qwen's vocabulary of 151,936 tokens. For example: <im_s> → 151644, def → 755, add → 912.
An LLM is a next-token predictor. The same sequence is used as both input and target, shifted by one position.
At every position, the model sees all tokens up to and including that position, and must predict the next token. Watch the target row shift into place:
At position 14, the model sees everything up to \n (newline after "assistant") and must predict def. At position 15, it sees up to def and must predict add.
Key insight: The model sees system and user tokens through attention — they provide context. But we only compute loss at assistant positions. This is where loss masking comes in next.
We don't want the model to learn to predict system or user tokens. Only assistant responses matter.
Labels for non-assistant positions are set to -100 — PyTorch's ignore index. Watch which tokens get masked vs trained:
For multi-turn conversations, every assistant turn is trained — every user/system turn is masked. SFTConfig's assistant_only_loss=True does this automatically.
At each unmasked position, the model outputs a probability distribution over the full vocabulary. Loss measures how wrong it is.
For our three trained positions, the model must predict: def (pos 14), add (pos 15), and <im_e> (pos 16). Here's what the untrained model actually predicts:
...<im_s>assistant\n → should output def...assistant\n def → should output add...assistant\n def add → should output <im_e>| Position | Correct Token | Before Training | After Training |
|---|---|---|---|
| 14 | def | P = 0.12 → Loss = 2.12 | P = 0.88 → Loss = 0.13 |
| 15 | add | P = 0.06 → Loss = 2.81 | P = 0.79 → Loss = 0.24 |
| 16 | <im_e> | P = 0.03 → Loss = 3.51 | P = 0.82 → Loss = 0.20 |
Total loss: 2.81 → 0.19. The model has learned to confidently produce the correct assistant response.
The entire journey from a JSON training pair to updated model weights — in eight steps.
Your dataset with "messages" column containing system, user, and assistant turns.
The Qwen tokenizer wraps messages in ChatML format with <|im_start|> and <|im_end|> tokens.
The formatted string converts to a sequence of integers: [151644, 8948, 198, ...]
Input = [tok₀, tok₁, …, tokₙ₋₁] and Target = [tok₁, tok₂, …, tokₙ]
System, user, and pad token labels set to -100. Only assistant positions retain true token IDs.
At every position, the model outputs a probability distribution over all 151,936 vocabulary tokens.
At each unmasked position: loss = −log(P(correct_token)). Total = average of unmasked losses.
Gradients flow to LoRA adapter weights (A, B matrices) only. Base weights stay frozen.
This cycle runs for every batch, every step, every epoch.
Part 1 complete. You now know what fine-tuning is, how data flows through the model, and how the model learns. Part 2 tackles the harder question: how do we do this efficiently on a 32-billion parameter model that doesn't even fit in GPU memory?
Part 1 showed you what fine-tuning does. Part 2 explains QLoRA — the technique that makes fine-tuning a 32-billion parameter model feasible on a single A100, without sacrificing quality.
Storing a 32B model in FP32 takes 128 GB just for weights. Add optimizer state and you need 384 GB — three A100s before you train a single step.
The core problem: GPUs have fixed memory. You cannot fine-tune what you cannot load.
Quantization shrinks the weights. LoRA shrinks what needs updating. Together they make a 32B fine-tune fit on one A100-80GB with room to spare.
Store each weight in 4 bits — not 32. A specially designed codebook makes this near-lossless for normally-distributed weights.
Don't update 32B weights. Add two tiny matrices whose product approximates the needed change. Train only those.
Optimizer state pages to CPU on memory spikes, pages back at the optimizer step. No OOM from gradient checkpointing bursts.
Standard 4-bit quantization wastes levels on values that rarely appear. NF4 uses information theory to concentrate precision exactly where pretrained weights cluster.
Pretrained LLM weights follow a normal (Gaussian) distribution. Most values are tiny — near zero. A handful reach ±1.
Uniform 4-bit quantization spaces its 16 levels evenly across the full range, wasting precision on the sparse tails. NF4 places levels at the quantiles of the bell curve — equally probable regions — so every level is used equally often.
Drag the slider to set a weight value and see which NF4 level it snaps to.
Each block of 64 weights needs one scale factor (absmax). That's 0.5 extra bits per weight. QLoRA quantizes those scales too:
That extra compression saves ~1.5 GB on a 32B model at essentially zero accuracy cost.
Every format uses its bits differently. Click between formats to see exactly how 5.75 is encoded.
| Format | Bits | Exponent | Mantissa | Max value | QLoRA role |
|---|---|---|---|---|---|
| FP32 | 32 | 8 | 23 | 3.4 × 10³⁸ | Optimizer master copy only |
| BF16 COMPUTE | 16 | 8 = FP32 | 7 | 3.4 × 10³⁸ | LoRA adapters + all matmuls |
| FP16 | 16 | 5 | 10 | 65,504 | Avoid in QLoRA RISKY |
| NF4 STORE | 4 | codebook | codebook | [-1, 1] | Frozen base weights |
LoRA's observation: the update a model needs is low-rank. You can approximate it with two thin matrices — and only train those.
Adjust r to see the real numbers for Qwen 2.5 Coder 32B (H=5120, 64 layers, 7 target modules).
For code generation on Qwen 2.5, target all 7 linear projections. Skipping MLP layers loses significant quality.
rsLoRA (use_rslora=True): scales by α/√r instead of α/r. Prevents gradients shrinking at high rank. Use at r ≥ 32. DoRA (use_dora=True): separates magnitude and direction updates, matches full fine-tuning quality at low ranks.
The frozen NF4 base and the trainable BF16 adapter never interact directly — but the adapter learns to compensate for NF4's rounding errors as part of its task adaptation.
Every forward pass through a Linear4bit layer runs two paths in parallel and adds their outputs.
Every byte in GPU memory during QLoRA training belongs to one of three categories. Understanding which is which tells you exactly what you can change, what you cannot, and where OOMs come from.
| Component | Size | What it is | Why it matters |
|---|---|---|---|
| 🔴 Frozen — requires_grad=False · loaded once, never written · persists in HBM all run | |||
| W^NF4 | 16.4 GB | 32.76B params × 4 bits ÷ 8. Each weight is a 4-bit index into the NF4 codebook, packed 2 per byte. | The dominant cost. prepare_model_for_kbit_training calls model.requires_grad_(False) — PyTorch allocates zero gradient buffers for it, saving ~16.4 GB vs full fine-tuning. |
| DQ scales | ~0.5 GB | c₂^FP8: one absmax per 64-weight block. c₁^FP32: one meta-scale per 256 blocks of c₂. | Needed to reconstruct W^BF16 in two dequant steps per forward pass. Without DQ this costs 0.5 bits/param; with DQ it drops to 0.127 bits/param. |
| embed + lm_head | ~3.1 GB | 152,064 × 5,120 × 2 bytes (BF16). Not quantized — bitsandbytes only touches inner Linear layers. | Frozen by default. Add to modules_to_save only when adding new vocabulary tokens — PEFT then saves a full BF16 copy alongside the adapter. |
| 🟢 Trainable — requires_grad=True · updated every step · has gradient buffer + optimizer state | |||
| LoRA A + B | ~270 MB | 134M params × 2 bytes BF16 at r=16, 7 modules, 64 layers. A: Kaiming-uniform init. B: zeros. | B=0 → ΔW=0 at step 0, model starts identical to pretrained. Only these weights move. At inference: merge W_merged = W₀ + (α/r)·B·A for zero latency overhead. |
| Gradients | ~270 MB | PyTorch auto-allocates a gradient buffer per trainable param, same shape and dtype (BF16). | PyTorch computes ∂L/∂B and ∂L/∂A but never ∂L/∂W₀ because requires_grad=False. That skip alone saves ~16.4 GB of gradient memory. |
| AdamW-8bit | ~270 MB | 1st moment m + 2nd moment v per trainable param, INT8-quantized, in CUDA Unified Memory. | With paged_adamw_8bit, pages to CPU RAM under pressure and back at optimizer.step(). Absorbs the transient OOM spikes that gradient checkpointing recomputation causes. |
| 🔵 Transient — materialized and freed within each layer · never accumulates across layers simultaneously | |||
| W^BF16 dequant | ~640 MB peak | NF4 → BF16 decoded in a CUDA kernel, used for one matmul, freed. Only one layer's W^BF16 exists at a time. | Largest single layer is down_proj: 27,648×5,120×2 = 283 MB. PyTorch's caching allocator reuses the memory slot for the next layer — peak cost is one layer, not all 64. |
| Activations | ~1–3 GB | With grad-ckpt: saved only at segment boundaries, recomputed on backward. Without: ~40 GB at seq=2048. | Recomputation costs ~30% slower backward, saves ~8× activation memory. Always use gradient checkpointing with QLoRA — the tradeoff is always worth it. |
| FA2 workspace | ~2–4 GB | Flash Attention 2 tiles Q·Kᵀ; the lse buffer costs batch × heads × seq_len × 4 bytes. | Peak transient hits during backward of the last layer in a grad-ckpt segment — simultaneous recomputed activations + W^BF16 + gradient. This is the spike paged_adamw_8bit is designed to absorb. |
Why quality holds despite 4-bit: NF4 rounding introduces a small, consistent error on each weight. The BF16 LoRA adapter can steer the layer output to compensate — it learns the task residual and the quantization residual simultaneously. The adapter doesn't know the difference; it just sees "the direction that reduces loss", which includes compensation for NF4 error.
Three libraries, seven decisions. Everything else can stay at its default.
| Parameter | Default | Set to | Why |
|---|---|---|---|
| load_in_4bit | False | True | Activates 4-bit path, replaces Linear with Linear4bit |
| bnb_4bit_quant_type | "fp4" | "nf4" | Optimal for normally-distributed weights — always use NF4 |
| bnb_4bit_compute_dtype | float32 | torch.bfloat16 | Default FP32 is silently 2× slower and wrong |
| bnb_4bit_use_double_quant | False | True | Free 1.5 GB — quantizes the scales too |
| bnb_4bit_quant_storage | uint8 | torch.bfloat16 (FSDP only) | Required for FSDP multi-GPU; safe on single GPU too |
LoraConfig defines the adapter architecture — rank, target layers, and scaling. It does not touch the base model. Call get_peft_model(model, lora_cfg) after prepare_model_for_kbit_training to inject the adapter.
| Parameter | Default | For Qwen 2.5 32B |
|---|---|---|
| // Adapter architecture | ||
| r | 8 | 16–32. Rank = dimension of the bottleneck. More rank = more capacity but more memory. At r=16 on Qwen 2.5 32B you get 134M trainable params (0.41%). At r=64 use rsLoRA. |
| lora_alpha | 8 | 2×r (e.g. 32 when r=16). The effective scale on ΔW is α/r = 2. Higher α = bigger adapter signal relative to base. rsLoRA changes this to α/√r internally when use_rslora=True. |
| lora_dropout | 0.0 | 0.05 for most tasks. Regularizes adapter weights during training. Set to 0.0 when using DoRA or PiSSA — they have their own regularization. |
| bias | "none" | "none" — always. "all" or "lora_only" break adapter merging and multi-adapter serving (S-LoRA, vLLM). Only change if you have a specific reason. |
| // Which layers to adapt | ||
| target_modules | None | ["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"] for Qwen 2.5. Or use "all-linear" — PEFT auto-discovers all Linear layers. Do NOT skip gate/up/down_proj for code tasks. |
| layers_to_transform | None (all) | List of layer indices to adapt, e.g. list(range(32,64)) to only adapt the top half. Useful for very constrained memory — top layers matter most for task adaptation. |
| layers_pattern | None | String pattern to match layer name, e.g. "layers". Used with layers_to_transform to scope which sub-module the index applies to. |
| modules_to_save | None | ["embed_tokens","lm_head"] only when adding new vocabulary tokens. These are saved as full BF16 trainable copies alongside the LoRA adapter — not quantized. |
| // Task and init | ||
| task_type | None | "CAUSAL_LM" — always set explicitly. Controls how PEFT wires up the forward pass and chat-template masking. |
| init_lora_weights | True | True (Kaiming-uniform A, zero B) — adapter output is exactly zero at step 0, so fine-tuning starts from the pretrained state. "loftq" for ≤3-bit. "pissa" for faster convergence. False only for debugging. |
| // Advanced scaling | ||
| use_rslora | False | True when r ≥ 32. Standard LoRA scales by α/r — gradient magnitude shrinks as 1/√r when rank grows. rsLoRA scales by α/√r instead, keeping gradient magnitude constant regardless of r. Essential for high-rank experiments. |
| use_dora | False | True at r = 4–8 for best quality. Decomposes W into magnitude × direction and adapts them independently. Full fine-tuning shows negative correlation between magnitude and direction updates; vanilla LoRA can't do this. DoRA closes most of the quality gap at low ranks. Small compute overhead. |
| rank_pattern | {} | Dict mapping module name → rank, e.g. {"q_proj":32,"v_proj":16}. Useful when you want higher rank on attention output but lower on MLP to save memory. |
| alpha_pattern | {} | Same as rank_pattern but for alpha. Use alongside rank_pattern to keep α/r constant per module. |
| fan_in_fan_out | False | True only for Conv1D-style weights (GPT-2). Keep False for all Qwen/Llama/Mistral models — they use standard Linear layers. |
SFTConfig inherits from TrainingArguments but overrides several defaults. TRL sets: lr=2e-5 (vs HF 5e-5), optim=fused (vs plain adamw), gradient_checkpointing=True (vs False), logging_steps=10 (vs 500), report_to="none" (vs "all"), use_cache=False (vs True). Always verify with print(trainer.args).
| Parameter | TRL default | For Qwen 2.5 32B |
|---|---|---|
| // Learning rate & scheduler | ||
| learning_rate | 2e-5 | 1e-4 to 2e-4. LoRA adapters are randomly initialized and need a higher LR than full fine-tuning to converge. The frozen base limits the damage from a large LR — only the small adapter updates. TRL default 2e-5 is too conservative; start at 2e-4 and tune down if loss is unstable. |
| lr_scheduler_type | "linear" | "cosine" — cosine decay keeps LR higher longer then anneals smoothly. Linear decay drops too fast in the middle of training where you still want to learn. |
| warmup_ratio | 0.0 | 0.03 — 3% of total steps. Prevents gradient explosion in the first few batches when the adapter weights are still far from equilibrium. Avoids the need for grad_norm clipping alone. |
| num_train_epochs | 3.0 | 1–2 for large domain datasets, 3–5 for very small ones. More epochs = more memorization, which is fine for SFT (you want the pattern reproduced) but watch for eval loss diverging. |
| // Gradient stability | ||
| max_grad_norm | 1.0 | 0.3 — QLoRA paper recommendation. Adapter gradients can spike early in training; 0.3 clips them more aggressively. If you see sudden loss spikes, lower to 0.1. |
| adam_beta2 | 0.999 | 0.95 — default 0.999 makes AdamW slow to respond to gradient magnitude changes. 0.95 gives faster adaptation, which matters for the small adapter with rapidly changing gradients. This is the single most under-appreciated hyperparameter in LLM fine-tuning. |
| weight_decay | 0.0 | 0.01–0.1 optional. Regularizes adapter weights. Usually not needed since adapters are already small and dropout provides regularization. |
| // Optimizer & memory | ||
| optim | "adamw_torch_fused" | "paged_adamw_8bit" — stores optimizer state (m, v) in CUDA Unified Memory. Under gradient-checkpointing memory spikes, pages m/v to CPU RAM automatically and pages them back at optimizer.step(). Near-zero throughput cost unless you're right at the memory edge. paged_adamw_32bit if you need full-precision optimizer state. |
| gradient_checkpointing | True (TRL) | Keep True. Saves activations only at layer boundaries, recomputes within each layer on backward. Trades ~30% compute for ~8× activation memory reduction. Pass gradient_checkpointing_kwargs={"use_reentrant":False} to avoid the reentrant warning with PEFT. |
| gradient_accumulation_steps | 1 | 16–32. With batch_size=1, effective_batch = grad_accum × n_gpus. 16 steps → effective batch 16 on single GPU. Large effective batches stabilize training when per-device batch must be 1 due to memory. |
| per_device_train_batch_size | 8 | 1 on A100-80GB at seq=2048. 2 on H100-80GB. Scale up with max_length — seq=512 allows batch=4 on A100. |
| // Precision | ||
| bf16 | None | True on A100/H100. Matches bnb_4bit_compute_dtype=bfloat16. Never set fp16=True with 4-bit quantization — causes NaN at step 1 on Ampere+ due to activation overflow. |
| tf32 | None | True — free ~10% throughput on Ampere/Hopper. TF32 rounds FP32 matmul inputs to 10-bit mantissa then accumulates in FP32. Invisible to user code, no precision impact for training. |
| // Dataset (TRL-specific keys) | ||
| max_length | 1024 | 2048–4096 for code. Renamed from max_seq_length in TRL ≥ 0.17 — use max_length in newer versions. Longer context = more GPU memory for activations; use grad-ckpt to compensate. |
| packing | False | True when most examples are shorter than max_length. ConstantLengthDataset packs multiple examples into one sequence, separated by EOS. 2–5× throughput improvement. Do NOT use packing + assistant_only_loss without also setting dataset_kwargs={"skip_prepare_dataset":False} — they can interact incorrectly. |
| assistant_only_loss | False | True for chat-format data. Uses the chat template's generation markers to mask prompt tokens (labels=-100). Loss and gradients only flow from assistant response tokens. Without this, the model trains to reproduce the user's question too, wasting capacity. |
| dataset_text_field | "text" | Column name in your dataset that contains the formatted text string. SFTTrainer reads this field and tokenizes it. Must match what your format function outputs. |
| neftune_noise_alpha | None | 5 or 10 for small domain datasets. Adds random Gaussian noise to embedding vectors during training. Equivalent to input data augmentation — significantly improves out-of-distribution generalization. Costs zero inference overhead (disabled at eval). |
| dataset_num_proc | None | 8–16. Number of CPU workers for dataset tokenization. Parallelizes the map() call. Match to your CPU core count. |
| // Checkpointing & logging | ||
| save_strategy | "steps" | Keep "steps". Use "epoch" only if epochs are very short. |
| save_steps | 500 | 200–500. Save every N steps. Each checkpoint is ~270 MB (adapter only). More frequent = less work lost on preemption. |
| save_total_limit | None | 2–3. Keeps only the N most recent checkpoints. Without this, all checkpoints accumulate on disk. |
| logging_steps | 10 (TRL) | 10 is fine. Log loss, lr, grad_norm, tokens/sec to wandb/tensorboard. Watch for sudden loss spikes (→ lower max_grad_norm) or flat loss (→ raise lr or add MLP targets). |
| report_to | "none" (TRL) | "wandb" for experiment tracking. Set WANDB_PROJECT env var. TRL already logs train/loss, train/learning_rate, train/grad_norm. |
| seed | 42 | Set explicitly for reproducibility. Affects dropout, data shuffling, and model init. |
Sets requires_grad=False on all base params — gradients never flow to W₀.
BF16 norms can be numerically unstable. FP32 norms are stable at negligible memory cost.
Grad-ckpt needs a gradient at layer 1. Frozen embeddings don't have one — this hook creates it.
Pass gradient_checkpointing_kwargs={"use_reentrant": False} to avoid DDP/PEFT warning.
Qwen 2.5 Coder 32B — H=5120, 64 layers, 40 attn heads (GQA 8 KV), I=27648, context 128K via YaRN.
Weights follow a bell curve. NF4 places 16 levels at equal-probability intervals — precision exactly where needed. Double quantization of scales brings total to 4.127 bits/param.
Task adaptation is low-rank. B×A (r=16) captures it with 134M params instead of 32B. B=0 at init — model starts identical to pretrained. Merge at inference for zero overhead.
NF4 rounding error ≈ small residual. BF16 LoRA adapter learns that residual alongside the task — not despite quantization but with it. This is why QLoRA matches 16-bit fine-tuning quality.