Before understanding any ML algorithm — from linear regression to large language models — you need to understand what ML is actually trying to compute. This guide builds that understanding from first principles.
Every ML algorithm ultimately optimises one equation: θ* = argmax (1/N) Σᵢ log p_θ(xᵢ). The six ideas below are the steps needed to understand where it comes from and why it is correct.
Every ML algorithm exists to answer one underlying question. Understanding it precisely is the prerequisite for everything that follows.
Consider diagnosing disease from a chest X-ray. A 1000×1000 pixel image is a vector of one million numbers — a single point in ℝ¹⁰⁰⁰⁰⁰⁰. The diagnosis is a label: 0 for diseased, 1 for benign. We want a function f mapping any pixel vector to the correct label. The trouble: f cannot be derived from first principles. Two radiologists examining the same X-ray sometimes disagree. A deterministic function is the wrong model — the right one is a probability distribution P(Y|X).
Choose a parametric form — a neural network with random weights, a linear model. This is your starting hypothesis, almost certainly wrong, but a point in parameter space that can be moved.
Each pair (xᵢ, yᵢ) provides a constraint. Adjust model parameters so the distribution places high probability on correct outputs. Every gradient descent step is one refinement. This is training.
The model should generalise — give good predictions on inputs not seen during training. If it only memorised D, it will fail on new data. This is the generalisation problem that statistical learning theory studies.
Every probability statement rests on a carefully constructed mathematical foundation. Before reasoning about uncertain functions, we need that foundation built from scratch — three objects and three axioms.
The sample space Ω might be "all possible patients." You cannot integrate over patients or compute derivatives with respect to weather states. To do mathematics, you need numbers — and the random variable provides them.
Despite its name, a Random Variable is neither random nor a variable. It is a completely deterministic function X: Ω → ℝ. X(Heads)=1, X(Tails)=0 — always, deterministically. The randomness comes from which ω ∈ Ω occurs in the experiment, not from X itself.
When X maps Ω → ℝ, it transports the entire probability triplet — this is the push-forward. Three things move simultaneously: Ω to ℝ, the σ-algebra F to the Borel σ-algebra B (generated by intervals like (−∞, x]), and the probability measure P to a new measure P_X on ℝ called the distribution of X.
In supervised learning, two random variables — X (input) and Y (label) — are defined on the same Ω. Every patient ω gives both an image X(ω) and a label Y(ω). The joint distribution P_{XY} captures their complete probabilistic relationship.
| Y = 0 (Diseased) | Y = 1 (Benign) | P_X — marginal | |
|---|---|---|---|
| X = Low brightness | 0.30 | 0.05 | 0.35 (sum row) |
| X = Med brightness | 0.10 | 0.20 | 0.30 |
| X = High brightness | 0.05 | 0.30 | 0.35 |
| P_Y — marginal | 0.45 (sum col) | 0.55 | 1.00 |
Two ideas are so central to statistical learning they deserve focused treatment. The IID assumption is what makes learning from data mathematically valid. The density function is the tool for evaluating how well a model fits data.
A dataset D = {x₁,...,xₙ} ~ i.i.d. P_x. These four letters pack in two separate and equally important assumptions.
There are two ways to describe the distribution of a continuous random variable. They carry the same information — one determines the other — but represent fundamentally different things. This distinction matters for everything that follows.
The critical point: for a continuous random variable, the probability of any single exact value is zero. P(X = x) = 0 for any specific x. This is a mathematical consequence of having an uncountable range. Probability only makes sense for intervals in the continuous case.
With the mathematical tools in place, we can state the core problem precisely: given dataset D sampled i.i.d. from unknown P_x, estimate P_x. The word "unknown" is where everything gets hard — and the three-step framework below is how it is resolved.
P_x is an infinite-dimensional object. Restrict to a parametric family indexed by a finite vector θ. Each choice of θ gives one specific distribution.
p_θ(x) = N(x;μ,Σ), θ={μ,Σ} — tractable, unimodal onlyΣₖ πₖ N(x;μₖ,Σₖ) — universal approximator for smooth densitiesMeasure how far p_θ is from the true P_x. Must satisfy d ≥ 0 and d=0 iff P_θ=P_x. The choice of d determines what the model learns and which algorithm results.
Problem: d requires computing something involving P_x — which is unknown. The WLLN resolves this in the next section.
After making d computable, minimise it over θ. For neural networks with millions of parameters: gradient descent, iteratively updating θ in the direction that reduces d. This process is model training.
We start with a question from information theory — how much information does an event carry? — and six connected steps later, we arrive at the training objective used by virtually every ML algorithm ever built.
Define I(A) = −log P(A). This formula is uniquely determined by three requirements: certain events carry zero information; rarer events carry more; information from two independent events adds. The negative log is the only function satisfying all three.
Entropy H(P_x) = E_{P_x}[−log P(x)] = Σᵢ P(xᵢ)·(−log P(xᵢ)) is the average information content — the average surprise when drawing from P_x. A flat distribution has maximum entropy (all outcomes equally surprising). A peaked one has minimum entropy (one outcome dominates).
Cross-entropy H(P, Q) = E_P[−log Q(x)] measures average information when data comes from P but surprise is measured using a different model Q. It always satisfies H(P,Q) ≥ H(P), with equality only when Q = P. Using the wrong model always increases average surprise.
KL divergence is the difference: D_KL(P ‖ Q) = H(P,Q) − H(P) = E_P[log P/Q] = ∫ P(x) log[P(x)/Q(x)] dx. It measures the excess surprise from using model Q when the truth is P. Three key properties: D_KL ≥ 0; D_KL = 0 iff P = Q; and D_KL(P‖Q) ≠ D_KL(Q‖P) — it is asymmetric, not a true metric distance.
We want to minimise D_KL(P_x ‖ P_θ) over θ. Expanding the KL gives D_KL = −H(P_x) − E_{P_x}[log p_θ(x)]. The first term is constant in θ and drops out. What remains is −E_{P_x}[log p_θ(x)] — an expectation under the unknown P_x. We cannot compute this integral directly.
The Weak Law of Large Numbers (WLLN) resolves this. Its role is precise: it does not solve the integral — it provides a substitution that makes the intractable tractable. If x₁,...,xₙ are drawn i.i.d. from P_x, then for any measurable function g with finite expectation:
Notice the essential role of IID: the WLLN requires samples drawn independently and identically from P_x. If samples are not independent, the average does not converge reliably. If not identically distributed, it converges to the wrong target. IID is not a convenience assumption — it is the prerequisite that enables this substitution.