Statistics

Prerequisites: Calculus.

• Why is it interesting?
• Intuitive
• Concrete
• Abstract

Why is it interesting?

For many extremely illuminating visualizations, see http://students.brown.edu/seeing-theory/

• Probability is starting with an animal, and figuring out what footprints it will make
• Statistics is seeing a footprint, and guessing the animal.

Probability is straightforward: you have the bear. Measure the foot size, the leg length, and you can deduce the footprints. "Oh, Mr. Bubbles weighs 400lbs and has 3-foot legs, and will make tracks like this." More academically: "We have a fair coin. After 10 flips, here are the possible outcomes."

Statistics is harder. We measure the footprints and have to guess what animal it could be. A bear? A human? If we get 6 heads and 4 tails, what're the chances of a fair coin?#

A Brief Introduction to Probability & Statistics by Kalid Azad

Important related concepts:

• Bayesian Statistics

Intuitive

Every measurement fluctuates. If we count how many particles hit a detector in one minute and repeat the count, we get a slightly different number each time. Probability theory starts from a known random process and predicts how the results will be spread. Statistics goes the other way: it starts from the observed results and asks what we can say about the process that produced them.

Two questions come up again and again. The first is estimation: what is the best guess for a quantity, such as a particle mass, and how uncertain is that guess? Averaging many measurements helps, and the uncertainty of an average shrinks with the square root of the number of measurements. Four times as much data halves the error bar.

The second is testing: is an observed effect real, or could it be a fluctuation? If a plot of collision events shows a small bump, we ask how often the known background alone would produce a bump at least that large. If that happens only very rarely, we take the bump as evidence for something new. In particle physics a discovery is announced only when the background alone would give such an excess with a probability of less than about one in three and a half million, the "five sigma" standard.

There are two schools of interpretation. Frequentists read probabilities as long-run frequencies in repeated experiments. Bayesians read them as degrees of belief that are updated when new data arrive (see Bayesian statistics). For large data sets the two usually lead to similar numbers, but the statements mean different things.

Concrete

Estimating a mean. Suppose we measure a quantity $N$ times and get $x_1,\ldots,x_N$, each drawn independently from a distribution with mean $\mu$ and standard deviation $\sigma$. The sample mean and sample variance

$$\bar x = \frac{1}{N}\sum_{i=1}^N x_i, \qquad s^2 = \frac{1}{N-1}\sum_{i=1}^N (x_i-\bar x)^2$$

are unbiased estimators: their expectation values are $\mu$ and $\sigma^2$. The variance of $\bar x$ is $\sigma^2/N$, so the standard error of the mean is $\sigma/\sqrt N$, estimated by $s/\sqrt N$. By the central limit theorem $\bar x$ is approximately Gaussian for large $N$, whatever the distribution of the single $x_i$.

Likelihood. If the data are described by a probability density $p(x\mid\theta)$ depending on parameters $\theta$, the likelihood of independent data is

$$L(\theta) = \prod_{i=1}^N p(x_i\mid\theta) ,$$

viewed as a function of $\theta$ at fixed data. The maximum likelihood estimator $\hat\theta$ maximises $L$, or equivalently minimises $-\ln L$. For measurements $y_i$ at points $x_i$ with Gaussian errors $\sigma_i$ around a model $f(x;\theta)$,

$$-2\ln L(\theta) = \sum_i \frac{\left(y_i - f(x_i;\theta)\right)^2}{\sigma_i^2} + \text{const} = \chi^2(\theta) + \text{const},$$

so maximum likelihood reduces to the method of least squares. For a single parameter, the interval in which $\chi^2$ rises by at most $1$ above its minimum is an approximate $68\%$ confidence interval. If the model is correct and linear in $\theta$, the minimal $\chi^2$ follows a $\chi^2$ distribution with $N - k$ degrees of freedom ($k$ the number of fitted parameters), whose mean is $N-k$; a value of $\chi^2_{\min}/(N-k)$ far above one signals a bad fit or underestimated errors.

Counting experiments. The number $n$ of events of a rare process in a fixed time follows a Poisson distribution, $P(n\mid\nu) = \nu^n e^{-\nu}/n!$, with mean and variance both equal to $\nu$. This is the origin of the familiar $\sqrt n$ error on a count.

Significance. To test a background-only hypothesis $H_0$ we choose a test statistic and compute the $p$-value: the probability, assuming $H_0$, of a result at least as extreme as the observed one. It is common to quote it as a Gaussian significance $Z$ through $p = \int_Z^\infty \frac{1}{\sqrt{2\pi}}e^{-t^2/2}dt$. The particle physics discovery threshold $Z = 5$ corresponds to a one-sided $p\approx 2.9\times10^{-7}$. A $p$-value is not the probability that $H_0$ is true; that would require a prior and Bayes' theorem.

Recommended Reading:

• A Brief Introduction to Probability & Statistics by Kalid Azad
• Introduction to the concept of likelihood and its applications by Alexander Etz
• What is Statistical Significance? by Tommaso Dorigo
• Decision Trees, Explained To Kids | Science 2.0
• Stats without Tears by Stan Brown (free online textbook)
• Statistics: (for Bayesian) Statistical Rethinking - A Bayesian Course with Examples in R and Stan - by McElreath
• DISCOVERING THE SIGNIFICANCE OF 5σ Louis Lyons

Abstract

A statistical model is a family $\{P_\theta\}_{\theta\in\Theta}$ of probability measures on a sample space $\mathcal{X}$, with probability itself defined by Kolmogorov's axioms. An estimator is a measurable map $\hat\theta:\mathcal{X}^N\to\Theta$, and a confidence set at level $1-\alpha$ is a data-dependent set $C(x)$ with $P_\theta(\theta\in C(x))\geq 1-\alpha$ for every $\theta$. The probability statement refers to $C$, not to $\theta$.

For a regular one-parameter model with density $p(x\mid\theta)$ the Fisher information is

$$I(\theta) = \mathbb{E}_\theta\left[\left(\frac{\partial}{\partial\theta}\ln p(x\mid\theta)\right)^2\right] = -\,\mathbb{E}_\theta\left[\frac{\partial^2}{\partial\theta^2}\ln p(x\mid\theta)\right].$$

The Cramér-Rao bound states that any unbiased estimator from $N$ independent observations satisfies $\mathrm{Var}_\theta(\hat\theta)\geq 1/(N I(\theta))$. Under regularity conditions the maximum likelihood estimator is consistent and asymptotically normal, $\sqrt N(\hat\theta-\theta)\to\mathcal{N}(0, I(\theta)^{-1})$, so it attains the bound asymptotically. For several parameters $I_{ab}(\theta)$ is a matrix, and it defines a Riemannian metric on the parameter manifold $\Theta$ (the Fisher-Rao metric), the starting point of information geometry.

Hypothesis tests are compared by their size (probability $\alpha$ of rejecting a true $H_0$) and power (probability of rejecting $H_0$ when $H_1$ is true). For two simple hypotheses the Neyman-Pearson lemma states that the test rejecting $H_0$ when $p(x\mid H_1)/p(x\mid H_0)$ exceeds a threshold is the most powerful test of its size. For composite hypotheses one uses the profile likelihood ratio

$$\lambda(\theta_0) = \frac{\sup_{\nu} L(\theta_0,\nu)}{\sup_{\theta,\nu}L(\theta,\nu)} ,$$

where $\nu$ are nuisance parameters. Wilks' theorem states that, under $H_0$ and regularity conditions, $-2\ln\lambda$ is asymptotically $\chi^2$ distributed with as many degrees of freedom as the number of parameters fixed by $H_0$. The conditions fail, for example, when $\theta_0$ lies on the boundary of $\Theta$ (a signal strength restricted to be non-negative, tested at zero), which is a standard situation in searches and changes the asymptotic distribution.

In the Bayesian formulation one specifies a prior $\pi(\theta)$ and computes the posterior $\pi(\theta\mid x) = p(x\mid\theta)\pi(\theta)/\int p(x\mid\theta')\pi(\theta')d\theta'$. Model comparison uses the ratio of marginal likelihoods (Bayes factor). For regular models the posterior concentrates around $\hat\theta$ with covariance $(N I(\hat\theta))^{-1}$ for large $N$ (Bernstein-von Mises theorem), which is why frequentist and Bayesian intervals often agree numerically for large samples.

• Data Analysis in High Energy Physics: A Practical Guide to Statistical Methods by Olaf Behnke et. al.

• Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations by Sander Greenland et. al.

• Probability Theory as Extended Logic Edwin T. Jaynes - Washington University website

• Probability Theory: The Logic of Science by Edwin T. Jaynes

Leads to: Bayesian Statistics, Entropy, Expectation Values, Monte-Carlo Simulations, Random Walks, Statistical Distribution Functions.