Not a summary — a from-scratch explanation. We'll build up the concepts one at a time, then show exactly how each one played out when training slm-125m, a 125.8M-parameter model built from nothing.
A language model is a function that predicts the next token. That's the entire mechanical premise — everything else is detail.
Before training, slm-125m has 125,847,552 parameters, and every one of them is a small random number. It has never seen a sentence. It doesn't know English exists, let alone what a legal opinion or an SEC filing looks like.
Pretraining is the process of fixing that — by showing the model an enormous amount of real text and, at every single position in that text, asking it to guess what comes next. It's wrong, almost every time, at first. The gap between its guess and the truth becomes a signal that nudges its parameters. Repeat that billions of times and something that started as noise becomes a model that writes fluent, stylistically-correct (if occasionally fabricated) legal and financial prose.
This is different from fine-tuning, which you may have seen elsewhere — fine-tuning starts from an already-pretrained model and adjusts it with a smaller, more targeted dataset (often using LoRA adapters to update only a fraction of the weights). slm-125m has no starting point. Every one of its 125.8M parameters is trained, from scratch, on this project's own corpus. There's no LoRA here — there's nothing to adapt yet.
Neural networks don't read letters. Before any of this works, text has to become a sequence of integers.
A tokenizer splits text into chunks — not always whole words, sometimes word-pieces or even single characters — and maps each chunk to an integer ID. slm-125m uses a vocabulary of exactly 16,384 possible chunks, trained specifically on this project's own legal/financial/web corpus (more on how that training works in Part 2).
Note the "Ġ" — that's how byte-level tokenizers commonly represent "a space precedes this chunk." Ten tokens for a nine-word sentence is typical; the real tokenizer in this project averages close to one token per word for its trained domains, which we'll measure precisely in Part 2.
Here's the entire trick behind next-token prediction: the same sequence is used as both the input and the answer key, offset by one position.
At every position, the model sees all tokens up to and including that position, and must predict the next one. There's no separate "question" and "answer" dataset — one sentence, read once, generates a training signal at every single token.
At position 6 ("...shall bear the"), the model has seen "<bos> The plaintiff shall bear the" and must predict "burden". At position 7, it's seen one token more and must predict ".". This is exactly what happens 1,024 times per training window, for every one of the 1,991,282 windows in the training set.
If you've read about fine-tuning a chat model, you've likely seen "loss masking" — hiding the user's turn so the model only learns to predict the assistant's reply. Pretraining doesn't need that.
In instruction fine-tuning, a training example has structure: a system prompt, a user turn, an assistant turn. You only want the model to learn to generate the assistant part, so every other token's loss gets set to -100 — PyTorch's "ignore this" signal.
slm-125m's training data has no such structure. It's raw continuous text — court opinions, SEC filings, web pages — with no roles, no turns, nothing to mask. Every single token contributes to the loss:
This is one reason continued pretraining (what this project does) needs far more tokens than instruction fine-tuning typically does — but also why it's conceptually simpler. There's no dataset-formatting decision to make about "what counts as the target." Everything is the target.
At every trained position, the model outputs a probability across all 16,384 possible next tokens. Loss measures how much probability it put on the token that was actually correct.
The formula is simple: loss = −log(P(correct token)). High confidence in the right answer → loss near zero. Confidently wrong → loss shoots up. Here's what the pattern looks like, illustratively, at three positions in our example sentence, before any training has happened:
<bos> The plaintiff → should output shall...shall bear the → should output burden...bear the burden → should output .The specific probabilities above are illustrative — reconstructing the exact softmax output at a single position from a live run isn't practical for a static page. But the overall pattern is exactly what happened, measured for real, averaged across the entire training and validation sets:
| Checkpoint | Avg. loss (nats) | Perplexity | What it means |
|---|---|---|---|
| Step 0 (random init) | 9.8706 | ≈19,412 | = ln(16,384) almost exactly — confirms correct initialization, measured before training began |
| Step 3,889 (final) | 2.326 (val) | ≈10.2 | real, measured on held-out data the model never trained on |
That drop — 9.87 to 2.33 — is the entire story of training, compressed into two numbers. Everything from here on is about how that happened.
Every one of the 3,889 training steps in this project did exactly this, on a fresh 524,288-token batch each time.
524,288 tokens, sampled from the 2.04B-token training corpus, already packed into 1,024-token windows.
Input = tokens[0..N-1], target = tokens[1..N] — the same data, offset.
At every position, the model outputs a probability distribution over all 16,384 vocabulary tokens.
Average of −log(P(correct)) across every position — no masking, everything counts.
Gradients flow to every one of the 125.8M parameters — full pretraining, not adapter-based.
Every parameter nudges slightly in the direction that reduces loss, scaled by the current learning rate.
Each cycle ≈2.6 seconds on the GPU used here. The whole run: 2.85 hours.
Part 1 complete. You now know what pretraining mechanically does. Part 2 gets specific: exactly what data slm-125m trained on, exactly how its tokenizer was built, exactly how big it is and why, and exactly what happened when this ran for real.
Part 1 covered what pretraining does, in general. Part 2 is this project, specifically — the real dataset, the real tokenizer, the real architecture, and the real numbers from the real training run.
The plan was 70% US case law, 20% SEC filings, 10% web text. Measuring the real token yield first ruled that out before a dollar was spent.
Case law and SEC filings together only contain about 2 billion clean tokens, total. There is no version of this project where legal text is 70% of a 10-billion-token corpus — the source material doesn't physically contain that much.
fineweb-edu, by contrast, is nearly bottomless — 11.67B tokens in the sampled slice alone. So the actual strategy: take all of both legal sources, cap web text at just enough to round out the mix.
Realized mix after cleaning: case law ~35%, SEC ~42%, web ~23% — about 78% legal text, nowhere near 70/20/10, but the closest legal-first mix this data actually supports.
Line-level noise filters, boilerplate stripping, repetition and language gates, plus an OCR-garbage gate for scanned case law.
Exact-hash duplicates removed, MinHash near-duplicates caught on case law, and a 13-word overlap check strips anything resembling the eval benchmarks.
Encoded with the custom 16,384-vocab BPE tokenizer (next chapter), packed into 1,024-token windows, 99/1 train/val split.
The tokenizer wasn't downloaded from a larger model — it's trained from scratch, on this project's own cleaned text, so its 16,384 chunks reflect legal and financial language specifically.
BPE starts with every input broken into raw bytes — the smallest possible unit, guaranteeing anything can be represented. Then, repeatedly:
Common English/legal/financial sequences — "tion", "the", whole short words — collapse into single tokens early. Rare sequences stay as smaller pieces or individual bytes. Nothing is ever unrepresentable; worst case, a token falls all the way back to raw bytes.
>>> tok.encode("The plaintiff shall bear the burden of proof by a preponderance of the evidence.") 15 tokens >>> tok.encode("The Company's net revenues increased 12% year over year pursuant to the agreement.") 16 tokens >>> tok.decode(tok.encode(text)) == text True # lossless roundtrip, every time
The starting alphabet is raw bytes, not Unicode characters — so any input, in any language, including malformed or unexpected text, always encodes to something. There's no "unknown character" failure mode.
A bigger vocabulary means fewer tokens per sentence (more efficient) but a bigger embedding table. At 125.8M total parameters, the embedding table is already 16,384 × 768 = 12.58M params — about 10% of the whole model. A 50,000-token vocab (typical for much larger models) would eat a disproportionate share of this model's capacity. 16,384 is sized for this model, not inherited from a bigger one.
Special tokens: <|bos|> <|eos|> <|pad|> <|unk|>, plus three chat-role tokens (<|user|>, <|assistant|>, <|system|>) reserved but unused during this base pretraining run — present in the vocabulary for a possible future fine-tuning stage.
No architectural novelty — a standard LLaMA-style decoder, so every result is attributable to data and training, not a clever model design.
| Parameters | 125,847,552 |
| Layers | 12 |
| Hidden size | 768 |
| Intermediate (MLP) | 3,072 · SwiGLU |
| Attention heads | 12 · head dim 64 |
| KV heads | 12 — plain MHA, not GQA |
| Positional encoding | RoPE, θ=10,000 |
| Normalization | RMSNorm, ε=1e-5 |
| Embeddings | Tied input/output |
| Context length | 1,024 tokens |
Tied embeddings means the same 12.58M-parameter matrix is reused for both "token in" and "token out" — saving ~12.6M parameters versus two separate matrices. Small models benefit from this proportionally more than large ones.
The real model uses hidden=768, layers=12 (the highlighted cell below). Drag either slider to see how parameter count scales.
At 125.8M parameters, full-precision full-parameter training fits comfortably in memory — the tricks that make training a 32B model feasible on one GPU exist to solve a problem this model doesn't have.
| Component | Size | What it is |
|---|---|---|
| Model weights (fp32) | ~503 MB | 125.8M params × 4 bytes. Kept in fp32 as the "master copy"; bf16 is used only transiently during the forward/backward matmuls (autocast). |
| Gradients | ~503 MB | One gradient per parameter, same shape and dtype as the weights — every parameter gets one, since this is full pretraining, not adapter-based. |
| AdamW optimizer state | ~1,006 MB | Two extra numbers per parameter (first and second moment estimates, "m" and "v") — this is why AdamW costs 2× the model size on top of the weights themselves. |
| Activations (transient) | a few GB | Intermediate values from the forward pass, held only until backward pass consumes them. Scales with batch size × sequence length × hidden size × layers. |
If you've read about QLoRA — 4-bit quantized frozen weights, small trainable LoRA adapters, paged optimizers to survive memory spikes — that machinery exists because a 32-billion-parameter model's full fp32 footprint is ~384GB, more than any single consumer or even datacenter GPU holds. slm-125m's entire footprint is about 2GB. This model would likely train fine on a GPU a fraction the size of an H100 — the H100 was chosen for speed, not because anything smaller couldn't fit it.
Compute per epoch ≈ 6 × params × tokens = 6 × 125.8M × 2.04B ≈ 1.54 × 10¹⁸ FLOPs. A single H100 finishes that in a few hours. Eight GPUs would mean paying for cross-GPU gradient-sync overhead with almost nothing to parallelize against at this parameter count — the original default config assumed 8×, and the math above is why it was changed to 1×. Measured result: ~2.63 seconds/step steady-state, ~2.85 hours total.
config.py is the single source of truth — every other file imports from it. Here's every value that mattered, and why it was set that way.
| vocab_size | 16,384 | sized for a 125M model, not inherited from a bigger one |
| hidden_size | 768 | residual stream width |
| intermediate_size | 3,072 | SwiGLU MLP inner dim, 4× hidden |
| num_hidden_layers | 12 | transformer blocks |
| num_attention_heads | 12 | head dim 64 (768÷12) |
| num_key_value_heads | 12 | = heads → plain MHA, not GQA |
| max_position_embeddings | 1,024 | context length |
| rope_theta | 10,000.0 | RoPE base frequency |
| rms_norm_eps | 1e-5 | normalization stability constant |
| tie_word_embeddings | True | saves ~12.6M params |
| min_line_chars | 40 | drop lines shorter than this |
| max_nonalnum_ratio | 0.30 | drop lines that are >30% non-alphanumeric |
| min_doc_chars | 600 | drop documents shorter than this after cleaning |
| repetition_top_k / max_repetition_ratio | 10 / 0.50 | drop if the top 10 most common 4-grams cover >50% of the doc |
| lang_sample_chars | 5,000 | how much text the language-detection gate samples |
| nonword_ratio_max | 0.20 | OCR gate — drop if >20% of words aren't in the dictionary |
| ocr_min_tokens | 50 | minimum words before the OCR gate applies at all |
| seq_len | 1,024 | tokens per training window |
| micro_batch_size | 32 | windows per forward/backward pass |
| global_batch_tokens | 524,288 | = 16 micro-batches accumulated per optimizer step |
| lr / min_lr | 6e-4 / 6e-5 | cosine schedule peak and floor |
| warmup_tokens | 200,000,000 | ≈381 steps of linear warmup before peak LR |
| weight_decay | 0.1 | AdamW regularization |
| beta1 / beta2 | 0.9 / 0.95 | AdamW moment decay rates |
| grad_clip | 1.0 | max gradient norm |
| ckpt_every_steps | 500 | resumable checkpoint cadence |
| eval_every_steps | 1,000 | held-out validation loss check |
| seed | 1337 | data shuffling + weight init |
.remote().spawn() + --detachfrom __future__ import annotationstoken_type_ids, which generate() doesn't acceptreturn_token_type_ids=Falseadd_local_python_source(...)Not illustrative this time — every point below is a real logged value from the actual training run.
Warmup then decay: linear ramp for ~381 steps (jumping straight to peak LR on random weights destabilizes early training), peak 6e-4, cosine decay to 6e-5 by the final step. This exact schedule is what lr_at() in train.py computes at every step.
Every phase, in order — each one a small, cheap, resumable cloud job, not one long fragile script.
Sample each source, extrapolate real token yield — the number that rules out 70/20/10 before any spend.
16 parallel Modal CPU workers, one per data shard. ~718,000 documents streamed, ~97% kept.
MinHash near-dup detection, 13-gram overlap check against eval benchmarks. 2.40B tokens remain.
Fresh 16,384-vocab BPE trained on the whole cleaned corpus.
14 parallel workers encode and pack into 1,024-token windows. 2.04B train / 20.6M val tokens.
modal run --detach modal_app.py::pretrain --epochs 1.0 — 1× H100, ~2.85 hours, ~$12–13.
modal run modal_app.py::deploy — uploads checkpoint, tokenizer, and an auto-generated model card.
modal deploy modal_app.py for a persistent inference endpoint, vercel deploy --prod for the static site that calls it directly from the browser.
# ── Modal ───────────────────────────────────────────────── pip install modal modal token new # ephemeral — stops when your terminal disconnects modal run app.py::main # long-running — survives a dropped connection modal run --detach app.py::train # persistent — gets a stable URL, serves forever modal deploy app.py # ── Vercel ──────────────────────────────────────────────── npm install -g vercel vercel link --yes --project my-site vercel deploy --prod
Why two platforms and not one: Modal has no first-class static-site hosting, and Vercel has no GPU compute. Each does the one thing it's actually built for — the browser-side demo calls the Modal endpoint directly via CORS, so there isn't even a server in between.
One sequence, offset by one position, generates a training signal at every position. No labeling, no separate answer key — the text is the supervision.
No LoRA, no quantization, no frozen base — small enough that full fp32 training fits in ~2GB, comfortably inside any modern GPU.
The data mix, the GPU count, and every hyperparameter above trace back to a measurement, not a guess — that's what makes the $15.54 total cost repeatable.