The HuggingFace Training Stack

The Stack

How the packages layer together

Think of it as a pyramid. Lower layers are more fundamental. Higher layers are more convenient but more opinionated. They are not alternatives — they are layers.

🦥 Unsloth

Optional speed layer. Replaces attention, cross-entropy, RoPE, LoRA matmul with hand-written Triton kernels. 2× faster, 70% less VRAM, zero accuracy change.

Speed & VRAM

🎯 TRL

Task-specific training objectives: SFTTrainer (instruction tuning), DPOTrainer (preference), GRPOTrainer (RL without critic), PPOTrainer (RLHF), ORPOTrainer, KTOTrainer…

Training Recipes

⚙ Trainer

Generic training loop: gradient accumulation, mixed precision, distributed training, checkpointing, logging, evaluation. All TRL trainers subclass this directly.

Training Loop

🔌 PEFT

Injects low-rank adapter matrices (LoRA, DoRA, IA³…) into frozen base layers. Controls which weights are trainable. Manages adapter saving, loading, and merging.

Adapter Layer

🤗 transformers

The actual model: architecture code, pre-trained weights, tokenizer, quantization (BitsAndBytes), and generation. Every other package sits on top of this.

Model Foundation

PyTorch / CUDA

Tensor ops, autograd engine, GPU kernels. All packages run on top of this runtime.

Runtime

💡

The key mental model: You always need transformers (the model). You usually need PEFT (adapters). You need exactly one trainer — bare Trainer for raw CPT, or a TRL task-specific trainer for everything else. Wrap with Unsloth on single GPU for free speed gains. None of these are alternatives to each other.

Package Reference

Each package, in depth

What it does, what it doesn't do, and the parameters you'll actually use.

pip install transformers

🤗 transformers

Layer 1 — the model itself. Architecture, weights, tokenizer, quantization, generation.

What it contains

AutoModelForCausalLM — loads any LLM (Qwen, Llama, Mistral…)
AutoTokenizer — text ↔ token IDs, chat templates
BitsAndBytesConfig — 4-bit / 8-bit quantization config
TrainingArguments — all training hyperparameters (inherited by ALL trainers)
GenerationConfig — sampling, beam search, temperature for inference

What you do here

Load the model with quantization config
Load and configure the tokenizer
That's it — adapters and training are handled above

Key parameters

quantization_config — pass BitsAndBytesConfig for 4-bit NF4
torch_dtype=torch.bfloat16 — BF16 for non-quantized loads
device_map="auto" — auto-spread across GPUs
attn_implementation="flash_attention_2" — FA2 speedup
trust_remote_code=True — required for custom arch models

BitsAndBytesConfig params

load_in_4bit=True — 4-bit NF4 weight storage
bnb_4bit_quant_type="nf4" — NormalFloat-4 (optimal for weights)
bnb_4bit_use_double_quant=True — compress scale constants too
bnb_4bit_compute_dtype=bfloat16 — dequantization compute dtype

When to use Always. Every other package requires transformers. Your only work here is loading the model + tokenizer with the right quantization config. Never call the model's forward pass directly during fine-tuning — that's the trainer's job.

step_1_load_model.py — transformers

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig import torch # ── 1. Configure 4-bit quantization ────────────────────────────────────── bnb_config = BitsAndBytesConfig( load_in_4bit = True, bnb_4bit_quant_type = "nf4", # NormalFloat-4 — optimal for weights bnb_4bit_use_double_quant = True, # compress scale constants too bnb_4bit_compute_dtype = torch.bfloat16, # dequant compute in BF16 ) # ── 2. Load model ──────────────────────────────────────────────────────── model = AutoModelForCausalLM.from_pretrained( "Qwen/Qwen2.5-Coder-32B-Instruct", quantization_config = bnb_config, device_map = "auto", # spread across all GPUs attn_implementation = "flash_attention_2", torch_dtype = torch.bfloat16, ) # ── 3. Load tokenizer ──────────────────────────────────────────────────── tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-32B-Instruct") tokenizer.padding_side = "right" # right-pad for causal LM training # This is ALL you do with transformers directly. # Adapters → PEFT Training loop → Trainer/TRL Speed → Unsloth

included in transformers

⚙ HuggingFace Trainer

Layer 2 — the generic training loop. All TRL trainers are subclasses of this.

What it handles

Gradient accumulation — accumulates across N mini-batches before optimizer step
Mixed precision — bf16/fp16 automatic casting via AMP
Distributed training — DDP, FSDP, DeepSpeed ZeRO integration
Checkpointing — save_steps, save_total_limit, resume_from_checkpoint
Evaluation & logging — eval loop, WandB/TensorBoard, metric computation
LR scheduling — warmup, cosine/linear decay built-in

Most important TrainingArguments

per_device_train_batch_size — samples per GPU per step (keep 1–4 for large models)
gradient_accumulation_steps — N mini-batches before optimizer.step()
num_train_epochs — full dataset passes
learning_rate — 2e-4 for LoRA, 1e-5 for full FT, 1e-5 for CPT
lr_scheduler_type — "cosine" standard for SFT/RL
warmup_ratio — 0.03–0.05 to prevent early divergence
optim="adamw_8bit" — BnB 8-bit Adam halves optimizer VRAM
gradient_checkpointing — frees activations, ~30% compute overhead
bf16=True — mixed precision on Ampere+ GPUs
max_grad_norm=0.3 — gradient clip (QLoRA paper recommendation)

When to use bare Trainer directly For continued pre-training (CPT) on raw text with no chat template, no RL, no preference — just next-token prediction on a corpus. For everything else (instruction tuning, preference alignment, RL), use a TRL trainer instead — it adds the task-specific loss on top of this same loop.

bare_trainer_cpt.py — Continued Pre-Training

from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling # ── TrainingArguments: shared by ALL trainers (Trainer, SFTTrainer, DPOTrainer…) ─ training_args = TrainingArguments( output_dir = "./cpt-output", # batch & accumulation per_device_train_batch_size = 2, gradient_accumulation_steps = 16, # effective batch = 32 # epochs & learning rate num_train_epochs = 1, # CPT: 1–2 epochs learning_rate = 1e-5, # lower than SFT for CPT lr_scheduler_type = "cosine", warmup_ratio = 0.05, # memory & precision bf16 = True, gradient_checkpointing = True, gradient_checkpointing_kwargs = {"use_reentrant": False}, optim = "adamw_8bit", max_grad_norm = 1.0, # logging & saving logging_steps = 10, save_strategy = "steps", save_steps = 500, save_total_limit = 2, report_to = "wandb", ) # ── DataCollatorForLanguageModeling: handles next-token targets ──────────── # mlm=False → causal LM (GPT-style), not masked LM (BERT-style) collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False) # ── Trainer: plain text corpus, no chat template ─────────────────────────── trainer = Trainer( model = model, args = training_args, train_dataset = raw_text_dataset, # just tokenized text, no messages column data_collator = collator, ) trainer.train()

pip install peft

🔌 PEFT

Layer 3 — Parameter-Efficient Fine-Tuning. Injects tiny trainable LoRA matrices into a frozen base model.

What it does

Freezes all base model weights (the 32B stays frozen)
Injects small A and B adapter matrices into chosen layers
Only A, B are trained — ~0.4% of total parameters
Manages adapter saving (~100 MB), loading, and merging
Supports LoRA, DoRA, IA³, Prefix Tuning, and more

How the math works

Weight update: ΔW = B · A where A ∈ ℝʳˣᵏ, B ∈ ℝᵈˣʳ
Forward pass: h = W₀x + (α/r) · BAx
B initialised to zero → ΔW starts at zero, stable training

LoraConfig key parameters

r (rank) — inner adapter dimension. 8–64. Start at 16.
lora_alpha — scaling factor. Convention: alpha = r → multiplier = 1
target_modules — which layers get adapters: ["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"] or "all-linear"
lora_dropout — 0.0 (required for Unsloth), 0.05–0.1 otherwise
bias="none" — required for Unsloth; recommended generally
use_dora — DoRA: decomposes into magnitude + direction. Slightly better quality at same rank
use_rslora — Rank-Stabilised LoRA: better at high ranks (r ≥ 64)
modules_to_save — full layers saved alongside adapter (e.g. lm_head if expanding vocab)

Direct usage vs. via SFTTrainer You rarely call PEFT directly when using TRL. Pass peft_config=LoraConfig(...) to SFTTrainer and it calls prepare_model_for_kbit_training() and get_peft_model() internally. Only write PEFT code directly for bare Trainer (CPT with LoRA) or custom training loops.

peft_direct.py — manual PEFT usage

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training, PeftModel # ── 1. Prepare: enable grad checkpointing + cast LayerNorm to FP32 ───────── model = prepare_model_for_kbit_training( model, use_gradient_checkpointing = True, gradient_checkpointing_kwargs = {"use_reentrant": False}, ) # ^ also calls model.enable_input_require_grads() — critical for frozen base + LoRA # ── 2. Define LoRA config ────────────────────────────────────────────────── lora_config = LoraConfig( r = 16, lora_alpha = 16, # alpha/r = 1.0 → neutral scaling target_modules = [ "q_proj", "k_proj", "v_proj", "o_proj", # attention "gate_proj", "up_proj", "down_proj", # MLP ], lora_dropout = 0.0, # 0.0 required for Unsloth bias = "none", # "none" required for Unsloth task_type = "CAUSAL_LM", use_dora = False, # True → DoRA (better, small overhead) use_rslora = False, # True → stable at r >= 64 ) # ── 3. Inject adapters → freezes base, adds trainable A and B matrices ──── model = get_peft_model(model, lora_config) model.print_trainable_parameters() # trainable params: 134,217,728 || all params: 32,894,390,272 || trainable%: 0.4080 # ── After training: three save strategies ───────────────────────────────── # Option A — adapter only (~100 MB, fast, swappable) model.save_pretrained("./adapter_only") # Option B — merge adapter into base weights (zero inference overhead) merged = model.merge_and_unload() merged.save_pretrained("./merged_model") # Option C — load adapter on top of base at inference time base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-32B-Instruct") model = PeftModel.from_pretrained(base, "./adapter_only")

pip install trl

🎯 TRL — Transformer Reinforcement Learning

Layer 4 — task-specific training recipes. Pick exactly one trainer based on what you're training for.

What TRL adds on top of Trainer

SFTTrainer — chat template application, loss masking, packing, PEFT auto-wrap
DPOTrainer — preference alignment from (chosen, rejected) pairs
GRPOTrainer — RL from reward functions without a critic model
PPOTrainer — classic RLHF with trained reward model
ORPOTrainer — SFT + preference alignment in a single pass
KTOTrainer — preference from unpaired (prompt, completion, good/bad) labels
RewardTrainer — trains the reward model for PPO

SFTConfig: TRL-specific params (adds to TrainingArguments)

max_length — max tokens per example (was max_seq_length in <0.17)
packing=True — pack multiple short examples into one sequence. Big throughput win.
dataset_text_field="text" — column with pre-formatted text
dataset_num_proc=8 — parallel CPU workers for dataset.map()
assistant_only_loss=True — mask user/system tokens with -100
peft_config — pass LoraConfig here → auto-wraps model in PEFT

The important distinction SFT is the training objective (supervised learning on instruction pairs). LoRA is the parameter update strategy (which weights receive gradients). They work together: you do SFT with LoRA/QLoRA, not instead of it. TRL's SFTTrainer handles the SFT objective; PEFT handles the LoRA injection.

sft_trainer.py — SFTTrainer (most common fine-tuning)

from trl import SFTTrainer, SFTConfig from peft import LoraConfig # ── SFTConfig = TrainingArguments + SFT-specific params ──────────────────── sft_config = SFTConfig( output_dir = "./sft-output", # inherited from TrainingArguments per_device_train_batch_size = 2, gradient_accumulation_steps = 8, # effective batch = 16 num_train_epochs = 3, learning_rate = 2e-4, # higher than full FT (adapters start at zero) lr_scheduler_type = "cosine", warmup_ratio = 0.03, bf16 = True, gradient_checkpointing = True, gradient_checkpointing_kwargs = {"use_reentrant": False}, optim = "adamw_8bit", max_grad_norm = 0.3, report_to = "wandb", # SFT-specific (added by TRL) max_length = 4096, # max tokens per example packing = True, # pack short examples → throughput win dataset_num_proc = 8, # parallel dataset tokenization assistant_only_loss = True, # mask user/system tokens → loss only on assistant ) # ── Pass peft_config: SFTTrainer calls get_peft_model() internally ───────── lora_config = LoraConfig( r = 16, lora_alpha = 16, target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], lora_dropout = 0.0, bias = "none", task_type = "CAUSAL_LM", ) # ── Dataset: "messages" column → SFTTrainer auto-applies Qwen's ChatML template # Each row: {"messages": [{"role": "system",...}, {"role": "user",...}, {"role": "assistant",...}]} trainer = SFTTrainer( model = model, tokenizer = tokenizer, train_dataset = dataset, # "messages" column triggers auto chat-template args = sft_config, peft_config = lora_config, # SFTTrainer wraps model in PEFT for you ) trainer.train()

grpo_trainer.py — GRPO (RL training, no critic model needed)

from trl import GRPOTrainer, GRPOConfig # ── Define reward functions — these drive the RL signal ──────────────────── def format_reward(prompts, completions, **kwargs) -> list[float]: """Reward 1.0 if response has <think>...</think> structure.""" rewards = [] for completion in completions: has_think = "<think>" in completion and "</think>" in completion rewards.append(1.0 if has_think else 0.0) return rewards def correctness_reward(prompts, completions, answer, **kwargs) -> list[float]: """Reward 2.0 if extracted answer matches ground truth.""" rewards = [] for completion, gt in zip(completions, answer): extracted = extract_answer(completion) # your parsing logic rewards.append(2.0 if extracted == gt else 0.0) return rewards # ── GRPOConfig: TrainingArguments + GRPO-specific params ─────────────────── grpo_config = GRPOConfig( output_dir = "./grpo-output", per_device_train_batch_size = 1, gradient_accumulation_steps = 8, num_train_epochs = 2, learning_rate = 1e-5, # lower than SFT — RL is sensitive bf16 = True, # GRPO-specific num_generations = 8, # N samples per prompt for group baseline max_completion_length = 1024, # longer for reasoning chains beta = 0.04, # KL penalty — prevents reward hacking epsilon = 0.2, # PPO-style policy ratio clipping use_vllm = False, # True = 3-5× faster generation step ) trainer = GRPOTrainer( model = model, tokenizer = tokenizer, train_dataset = dataset, reward_funcs = [format_reward, correctness_reward], # mix multiple signals args = grpo_config, ) trainer.train()

pip install unsloth

🦥 Unsloth

Layer 5 (optional) — replaces the slow hot paths with hand-written Triton kernels. Same results, 2× faster, 70% less VRAM.

What Unsloth replaces internally

AutoModelForCausalLM → FastLanguageModel.from_pretrained()
BitsAndBytesConfig → baked into load_in_4bit=True
prepare_model_for_kbit_training() → done internally
get_peft_model() → FastLanguageModel.get_peft_model() with Triton kernels
gradient_checkpointing → use_gradient_checkpointing="unsloth"
SFTTrainer / GRPOTrainer → same API underneath, just faster

Performance vs. vanilla stack

Training throughput: ~2× faster tokens/sec
Activation VRAM: ~70% less vs no checkpointing
vs HF checkpointing: ~30% less VRAM, no speed penalty
Inference: 2× via for_inference() mode
Download size: 4× smaller (pre-quantized NF4 repos)

Hard constraints (Triton kernel requirements)

lora_dropout must be 0 — Triton kernel path requirement
bias must be "none" — Triton kernel path requirement
Single GPU — multi-GPU requires Unsloth Pro or vanilla PEFT+TRL
Don't set gradient_checkpointing=True in args — conflicts with Unsloth's hooks

When to use Unsloth vs vanilla stack Use Unsloth on single A100/H100 — it's a pure win with no code complexity cost. Skip Unsloth for multi-GPU FSDP / DeepSpeed runs — those require vanilla transformers + PEFT + TRL with accelerate. Unsloth is optional; remove it and your code still runs correctly (just slower).

unsloth_sft.py — full Unsloth SFT workflow

from unsloth import FastLanguageModel from trl import SFTTrainer, SFTConfig # ── 1. Load (replaces AutoModelForCausalLM + BitsAndBytesConfig) ─────────── model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Qwen2.5-Coder-32B-Instruct-bnb-4bit", max_seq_length = 4096, dtype = None, # auto: BF16 on Ampere+, FP16 otherwise load_in_4bit = True, # NF4 + double_quant baked in ) # ── 2. Attach LoRA (replaces prepare_model_for_kbit_training + get_peft_model) model = FastLanguageModel.get_peft_model( model, r = 16, target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], lora_alpha = 16, lora_dropout = 0, # must be 0 for Triton kernels bias = "none", # must be "none" for Triton kernels use_gradient_checkpointing = "unsloth", # "unsloth" > True > False random_state = 42, ) # ── 3. Train — same SFTTrainer API, Unsloth kernel underneath ────────────── trainer = SFTTrainer( model = model, tokenizer = tokenizer, train_dataset = dataset, args = SFTConfig( per_device_train_batch_size = 2, gradient_accumulation_steps = 8, num_train_epochs = 3, learning_rate = 2e-4, bf16 = True, # do NOT set gradient_checkpointing=True here — Unsloth handles it above optim = "adamw_8bit", max_length = 4096, packing = True, assistant_only_loss = True, output_dir = "./unsloth-output", ), ) trainer.train() # ── 4. Export: adapter / merged / GGUF ──────────────────────────────────── FastLanguageModel.for_inference(model) # 2× faster inference model.save_pretrained("lora_adapter") # adapter only model.save_pretrained_gguf("model_q4", tokenizer, # GGUF for Ollama quantization_method = "q4_k_m")

End-to-End Training Pipeline

Where each package lives
in the pipeline

From raw data to saved adapter — every step mapped to its owning package.

🤗

Load model & tokenizer transformers

AutoModelForCausalLM.from_pretrained() with BitsAndBytesConfig for 4-bit NF4 loading. AutoTokenizer.from_pretrained() to load the tokenizer. This is your only interaction with transformers directly during setup.

🔌

Inject LoRA adapters PEFT

prepare_model_for_kbit_training() + get_peft_model(model, LoraConfig). Freezes the 32B base weights, injects trainable A and B matrices into the 7 attention + MLP linear layers. ~134M trainable parameters out of 32.9B total. If using SFTTrainer, skip this step — pass peft_config=LoraConfig(...) and it handles PEFT internally.

📝

Prepare dataset TRL / SFTTrainer

Dataset with "messages" column — each row contains a list of {role, content} dicts. SFTTrainer automatically applies tokenizer.apply_chat_template() to format as Qwen's ChatML. Sets -100 on all non-assistant token positions so loss is computed only on assistant responses.

⚙

Configure training loop Trainer / SFTConfig

SFTConfig holds all hyperparameters. TrainingArguments params (lr, batch size, accumulation steps, scheduler, checkpointing) are inherited from Trainer. SFT-specific params (max_length, packing, assistant_only_loss) are added by TRL on top.

🦥

Speed optimisation (optional) Unsloth

Replace AutoModelForCausalLM with FastLanguageModel.from_pretrained() and get_peft_model with FastLanguageModel.get_peft_model(use_gradient_checkpointing="unsloth"). The SFTTrainer call stays identical — Unsloth plugs in at the kernel level, not the API level.

▶

Train SFTTrainer.train()

For each batch: (1) forward pass through model, (2) cross-entropy loss on assistant token positions only, (3) backward pass → gradients flow only into LoRA A and B matrices, (4) optimizer step on adapters, (5) zero_grad. Repeated across gradient_accumulation_steps before each optimizer step. Base model stays frozen and bit-identical throughout.

💾

Save & export PEFT

model.save_pretrained("adapter") saves only the ~100 MB adapter weights. model.merge_and_unload() folds B·A into W₀ for zero inference overhead. Unsloth additionally supports save_pretrained_gguf() for Ollama-compatible GGUF files.

TRL Trainer Variations

Pick exactly one trainer
based on your objective

All TRL trainers extend HF Trainer. They differ only in their training objective — what loss function they use and what data they expect.

Stage 1

SFTTrainer + SFTConfig

Supervised Fine-Tuning

Trains on (prompt → response) pairs. Uses cross-entropy loss on assistant tokens only. Auto-applies chat templates, packs sequences, wraps PEFT. The starting point for almost every fine-tuning project. Dataset: {"messages": [...]}.

Stage 2 — Preference

DPOTrainer + DPOConfig

Direct Preference Optimization

Aligns model to human preferences from (prompt, chosen, rejected) triples. No reward model needed. Maximizes log-probability ratio of chosen vs rejected against a reference model (usually the SFT checkpoint). Run after SFT — DPO on a base model rarely works. Key param: beta=0.1.

Stage 2 — RL (no critic)

GRPOTrainer + GRPOConfig

Group Relative Policy Optimization

DeepSeek-R1's RL algorithm. Estimates baseline from N sampled responses per prompt — no separate value/critic model needed. You provide reward functions (format check, code execution, math correctness). Key params: num_generations=8, beta=0.04, max_completion_length=1024.

Stage 2+3 in one pass

ORPOTrainer + ORPOConfig

Odds-Ratio Preference Optimization

Combines SFT + preference alignment in a single training pass. No reference model required. Trains chosen responses with SFT loss + penalises rejected responses via odds-ratio. Most VRAM-efficient alignment method — one model, no frozen reference copy. Dataset: {"prompt", "chosen", "rejected"}.

Stage 2 — Classic RLHF

PPOTrainer + PPOConfig

Proximal Policy Optimization

Classic RLHF (InstructGPT / ChatGPT). Requires: (1) trained reward model, (2) value/critic model, (3) reference policy. Most complex, most VRAM-hungry. Key params: kl_penalty, init_kl_coef=0.2, target_kl=6.0. For most use cases, prefer GRPO or DPO.

Stage 2 — Unpaired

KTOTrainer + KTOConfig

Kahneman-Tversky Optimization

Works with unpaired feedback — just label each (prompt, response) as good or bad. Based on prospect theory from behavioural economics. No need to pair chosen/rejected. Useful when you can't guarantee you have pairs. Dataset: {"prompt", "completion", "label": true/false}.

Use Case Matrix

Right package for
every training scenario

Every combination of objective × technique mapped to its optimal configuration.

Training Scenario	transformers	PEFT / LoRA	Primary Trainer	Unsloth	Key Config Note
Continued Pre-Training (CPT)	✓	full weights	Trainer	—	DataCollatorForLM, no chat template, lr=1e-5, 1–2 epochs
CPT with LoRA	✓	✓	Trainer	optional	Same as CPT + LoraConfig. Less catastrophic forgetting risk
Instruction Fine-Tuning (SFT)	✓	✓ LoRA/QLoRA	SFTTrainer	recommended	messages column, assistant_only_loss=True, packing=True
Full Fine-Tuning (SFT)	✓	no adapters	SFTTrainer	no (multi-GPU)	FSDP / DeepSpeed ZeRO-3 required for 32B+, bf16=True
Preference Alignment (DPO)	✓	✓	DPOTrainer	optional	Apply after SFT checkpoint. beta=0.1, loss_type="sigmoid"
SFT + Alignment in one pass (ORPO)	✓	✓	ORPOTrainer	optional	No ref model → most VRAM-efficient alignment. lambda=0.1
RL from reward functions (GRPO)	✓	✓	GRPOTrainer	yes (Unsloth GRPO)	reward_funcs list, num_generations=8, lr=1e-5, after SFT
Classic RLHF (PPO)	✓	✓	PPOTrainer	—	Needs separate trained reward model + critic. Prefer GRPO.
Unpaired feedback (KTO)	✓	✓	KTOTrainer	optional	No pairs needed. Just {prompt, completion, label: good/bad}
Train reward model (for PPO)	✓	✓	RewardTrainer	optional	AutoModelForSequenceClassification, outputs scalar rewards

Decision Guide

Pick your stack
in four questions

Answer each question sequentially. By question 4 you have your exact configuration.

What is your training objective?

→Instruction following / code generation / domain adaptation → SFTTrainer (TRL)

→Pre-train on raw text corpus, no instruction format → bare Trainer (HF)

→Align to human preferences from (chosen, rejected) pairs → DPOTrainer or ORPOTrainer

→Optimise with a custom reward signal (code correctness, format) → GRPOTrainer

→Classic RLHF with a trained reward model → PPOTrainer (rarely needed)

Update all weights or just adapters?

→Adapters (LoRA/QLoRA) → add PEFT LoraConfig, pass as peft_config= to SFTTrainer. Saves ~99.5% of optimizer VRAM.

→All weights (full fine-tune) → no PEFT. Requires FSDP or DeepSpeed ZeRO-3. 4× model VRAM for optimizer state on 32B.

Single GPU or multi-GPU?

→Single A100/H100 → Unsloth: replace AutoModelForCausalLM with FastLanguageModel. Set lora_dropout=0, bias="none", use_gradient_checkpointing="unsloth". Free 2× speedup.

→Multi-GPU FSDP / DeepSpeed → vanilla transformers + PEFT + TRL with accelerate. Add gradient_checkpointing=True, optim="adamw_8bit".

Where in the training pipeline are you?

→Starting from scratch → CPT on domain text (Trainer) → SFT on instruction pairs (SFTTrainer)

→Have SFT checkpoint → DPO / ORPO for preference, then GRPO for RL-based task polish

→Going to production → merge_and_unload() for zero inference overhead, or save_pretrained_gguf() for Ollama

🎯

For Qwen 2.5 Coder 32B spec-to-code: transformers (BnB 4-bit NF4 load) → PEFT (LoraConfig r=16, all 7 linear modules) → TRL SFTTrainer (SFTConfig with max_length=4096, packing=True, assistant_only_loss=True, messages column for auto-ChatML) → Unsloth for single-GPU speed, or vanilla accelerate + FSDP for multi-A100. After SFT benchmark passes, layer GRPOTrainer with a code execution reward function for the RL alignment stage.