The library

E[X] = Σ k·P(k) for discrete, ∫ x·f(x) dx for continuous. It's the balance point of the distribution — and need not be a value X can actually take (a die's mean is 3.5).

Var(X) = E[(X−μ)²]; the standard deviation is its square root, back in the original units. Variance adds for independent variables, which is why it shows up everywhere.

The support is where the PMF/PDF is positive — the values that can actually occur: supp(f) = closure of { x : f(x) > 0 }. It bounds where estimates and integrals live, and mismatched supports (a model assigning probability 0 to data you observed) quietly break likelihoods and importance sampling.

Probability fixes the parameters and asks about data: P(data | θ), summing to 1 over data. Likelihood fixes the observed data and varies the parameters: L(θ) = P(data | θ) read as a function of θ — it is NOT a probability distribution over θ.

A complete description of a random quantity: every outcome and how much probability (or density) it carries. Everything else — mean, variance, tail risk — is read off from it.

A PMF gives P(X=k) directly and sums to 1. A PDF gives density, where probability is area under the curve; for continuous X, P(X = any exact point) is 0, so only intervals carry probability.

The likelihood L(θ) = ∏ f(xᵢ; θ). Maximum likelihood estimation chooses the θ that maximizes it — usually via the log-likelihood, which turns the product into a sum and the optimization into calculus.

An estimator is a function of the data — itself a random variable with its own distribution (e.g. the sample mean). An estimate is the single value that rule returns for one dataset. Properties like bias and variance belong to the estimator.

Bias = E[estimator] − true value. Unbiased means zero on average across samples — but a slightly biased estimator with much lower variance is often the better bet (the bias–variance trade-off).

An estimator is consistent if it converges in probability to the true parameter as n → ∞. More data should pin it down; an inconsistent estimator never settles on the right answer no matter how much you collect.

A statistic is sufficient for θ if the data carry no extra information about θ once you know it (Fisher–Neyman factorization). The sample sum is sufficient for a Poisson rate — you can throw away the rest.

Among unbiased estimators, the efficient one has the smallest variance — bounded below by the Cramér–Rao limit. Efficiency is why we prefer one valid estimator over another.

A decision procedure: take H₀ as true, compute how surprising the data would be under it, and reject H₀ only if that surprise crosses a pre-set threshold. It controls error rates, not truth.

H₀ is the no-effect / status-quo claim you try to falsify; H₁ is the alternative you'd accept if the evidence is strong enough. You never 'prove' H₀ — you only fail to reject it.

A summary of the data (z, t, χ², F) whose sampling distribution is known when H₀ holds. You locate your observed value in that distribution to judge how extreme it is.

The probability, assuming H₀, of a test statistic at least as extreme as the one observed. It is NOT the probability that H₀ is true, nor the chance your result was luck — just a measure of incompatibility with H₀.

Chosen before you look (often 0.05): the probability of rejecting a true H₀. Reject when p < α. It's a budget for Type-I errors, not a law of nature — pick it for the decision's stakes.

Type I: rejecting a true H₀ (a false positive), rate α. Type II: failing to reject a false H₀ (a missed effect), rate β. Tightening one usually loosens the other for fixed sample size.

Power = 1 − β: the probability of correctly rejecting H₀ when H₁ is true. It rises with sample size, effect size, and lower noise — the thing a good experiment is designed around.

A 95% CI is built by a method that, across repeated samples, contains the true parameter 95% of the time. The frequentist subtlety: it's a property of the procedure, not a 95% probability about this one interval.

Sums and means of many independent, finite-variance variables approach a Normal distribution regardless of the original shape. It's why so much of inference leans on the Normal even when the data aren't.

As n grows, the sample mean converges to E[X]. It's the guarantee that more data, gathered honestly, pulls your estimate toward the truth — the backbone of simulation and bootstrapping.

Resample the dataset with replacement many times, recompute the statistic each time, and use the spread of results as an approximate sampling distribution — confidence intervals with almost no distributional assumptions.

Bootstrap AGGregating: fit a model on each bootstrap resample and average (or vote) their predictions. It tames variance in unstable learners — random forests are bagging plus feature subsampling.

Frequentists treat parameters as fixed unknowns and reason about repeated sampling. Bayesians put a probability distribution on the parameter — a prior updated by data into a posterior. Different questions, both useful; the Beta–Binomial pair is the classic bridge.

A Beta prior on a probability, combined with Binomial data, yields a Beta posterior — just add successes to α and failures to β. That tidy update is why Beta is the natural language for an uncertain rate.

Softmax maps a vector of scores to probabilities pᵢ = e^(zᵢ/T) / Σ e^(zⱼ/T) — exactly the Gibbs/Boltzmann distribution from statistical physics. Temperature T flattens (high) or sharpens (low) the choice.

An attention head softmaxes query·key scores into weights that sum to 1 — a distribution over positions. A transformer layer is a stack of these little distributions mixing values together.

Dropout multiplies each activation by an independent Bernoulli(p) mask at train time — a stochastic ensemble that approximates bagging over exponentially many sub-networks.

Glorot/Xavier and He initialization sample weights from a Normal (or uniform) whose variance is set by the layer's fan-in/out, so signal neither explodes nor vanishes with depth. The Gaussian is doing structural work.

A binary classifier's logit passes through the sigmoid (the logistic CDF) to give P(y=1). Minimizing cross-entropy is exactly maximizing the Bernoulli log-likelihood.

A GP places a joint Gaussian over function values: any finite set of points is multivariate Normal. It's the distribution-first view of regression — predictions arrive with calibrated uncertainty.

An LLM samples the next token from the softmax distribution over the vocabulary; temperature scales the logits, trading determinism (T→0, argmax) for diversity. Top-k / nucleus sampling truncate that distribution's tail.

Persistent homology summarizes data by the connected components and holes that persist across scales — Betti numbers and persistence diagrams. Those summaries are themselves data (counts and lifetimes) you can model statistically: the bridge from shape to inference.

Introduced in “Attention Is All You Need” (2017), the transformer drops recurrence and convolution for stacked self-attention plus feed-forward layers. Because attention is fully parallel across positions, it trains at scale — the architecture under nearly every modern LLM, vision, and multimodal model.

Each token forms a query and compares it to every token's key; the scaled dot products softmax into weights that mix the values — Attention(Q,K,V) = softmax(QKᵀ/√d)·V. Multi-head attention runs several such lookups in parallel subspaces and concatenates them.

Transformers come in three shapes: encoder-only (BERT — bidirectional, for understanding/embeddings), decoder-only (GPT — autoregressive, for generation), and encoder–decoder (T5 — sequence-to-sequence). Alongside them live convolutional (CNN), recurrent (RNN/LSTM), and state-space (SSM/Mamba) families.

An agent senses its environment, chooses actions toward an objective, acts, observes, and repeats (plan → act → observe). LLM agents add tool use, memory, and multi-step autonomy. The rigor is decision-theoretic — acting well under uncertainty, not merely predicting the next token.

A recurrent network carries a hidden state across time steps. Plain RNNs lose long-range context (vanishing gradients); the LSTM adds input/forget/output gates and a cell state to preserve information over long spans. Largely superseded by transformers for long range, still handy when compute or latency is tight.

A CNN slides learned filters over grid-structured data (images, audio), exploiting locality and translation-equivariance with far fewer parameters than a dense net. Convolution is weight-sharing across position; pooling adds a measure of scale invariance.

Generative Pre-trained Transformer: a decoder-only model trained to predict the next token given the previous ones. Scale plus this single objective yields broad capability; generation samples from the softmax over the vocabulary.

BERT reads the whole context at once (bidirectional) and pretrains by predicting masked-out tokens. Encoder-only, it excels at understanding and embedding tasks — classification, retrieval — rather than free-form generation.

The encoder maps input to a latent representation; the decoder generates output from it. Read through information theory it is the whole pipeline — storing, coding, transmitting, de-noising, and extracting. BERT is encoder-only, GPT decoder-only, T5 the full encoder–decoder.

A large language model is a (usually decoder-only) transformer trained on vast text to predict the next token. Scale brings emergent abilities — in-context learning, reasoning traces — and new failure modes: hallucination and calibration drift.

Text is split into tokens — subword pieces from algorithms like BPE or WordPiece — each mapped to an integer id and an embedding. Token count drives context limits and cost, and odd tokenization explains many quirks (arithmetic, spelling).

The context window is the span of tokens a model can attend to at once. Contextualization means a token's representation depends on its neighbors — the same word gets a different vector in a different sentence, unlike static (word2vec) embeddings.

RAG searches a corpus (usually by vector similarity) for passages relevant to a query, then feeds them to the model as grounding. It cuts hallucination and lets a frozen model use fresh or private knowledge without retraining.

Running models locally (Ollama, llama.cpp, on-device) keeps data in your environment — no egress, lower latency, offline capability, privacy by construction. The same compute-to-data principle as keeping client data where it lives.

A GLU computes (xW) ⊗ σ(xV) — one linear path gated elementwise by a nonlinearity of another. The variants GeGLU and SwiGLU (GELU/Swish gates) reliably improve transformer feed-forward blocks.

RoPE rotates pairs of query/key coordinates by an angle proportional to position, so the attention dot product depends only on relative position. It bakes in order without separate position vectors and extrapolates to longer contexts.

ALiBi adds a fixed linear penalty to attention scores proportional to the distance between tokens, favoring nearer ones. With no learned positional embeddings, it extrapolates gracefully to sequences far longer than seen in training.

Softmax attention costs O(n²) in sequence length. Random-feature methods (e.g., Performer's FAVOR+) approximate the softmax kernel with random features and factor it, so attention runs in linear time and memory — trading exactness for scale.

Learn general representations on a large corpus, then reuse them — full fine-tuning, adapters/LoRA, or prompting — for a downstream task with little data. The pretraining → adaptation paradigm is why one base model powers many applications.

Because benchmarks leak and single scores mislead, eval engineering designs held-out and contamination-controlled test sets, task-grounded metrics, rubrics, and LLM-as-judge with calibration — tracked over time. What you measure is what you ship; the discipline is making the measurement trustworthy.