Bayesian Statistics

Prerequisites: Statistics. Full path (2 pages)

• Intuitive
• Concrete
• Abstract

Intuitive

Recommended Books:

• The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy by Sharon Bertsch McGray

Concrete

Bayesian statistics treats a probability as a degree of belief about a proposition, including propositions about the values of parameters. The basic tool is Bayes' theorem. For a parameter $\theta$ and observed data $D$,

$$P(\theta \mid D) = \frac{P(D \mid \theta)\, P(\theta)}{P(D)}, \qquad P(D) = \int P(D \mid \theta)\, P(\theta)\, d\theta .$$

• $P(\theta)$ is the prior: what we believe about $\theta$ before seeing the data.
• $P(D \mid \theta)$ is the likelihood: the probability of the observed data if the parameter had the value $\theta$. As a function of $\theta$ it is in general not normalized, so it is not a probability density for $\theta$.
• $P(\theta \mid D)$ is the posterior: the updated belief after seeing the data.
• $P(D)$ is the evidence (or marginal likelihood). It normalizes the posterior.

The theorem itself is a one-line consequence of the product rule, $P(\theta, D) = P(\theta \mid D) P(D) = P(D \mid \theta) P(\theta)$.

Worked example: a coin of unknown bias. Let $\theta$ be the probability of heads. We take a Beta prior,

$$P(\theta) = \frac{\theta^{\alpha-1}(1-\theta)^{\beta-1}}{B(\alpha,\beta)}, \qquad 0 \le \theta \le 1 ,$$

and observe $k$ heads in $n$ flips. The likelihood is binomial, $P(D \mid \theta) = \binom{n}{k}\theta^k (1-\theta)^{n-k}$. Multiplying prior and likelihood gives a posterior proportional to $\theta^{\alpha+k-1}(1-\theta)^{\beta+n-k-1}$, which is again a Beta distribution with parameters $\alpha + k$ and $\beta + n - k$. Its mean is

$$\langle \theta \rangle = \frac{\alpha + k}{\alpha + \beta + n} .$$

For a uniform prior ($\alpha = \beta = 1$) this is $(k+1)/(n+2)$, Laplace's rule of succession. For large $n$ the prior parameters become irrelevant and the mean approaches the observed frequency $k/n$. A prior whose posterior stays in the same family is called conjugate.

Using the posterior. The result of a Bayesian analysis is the whole posterior distribution. From it we quote summaries: the posterior mean or mode, and a credible interval, an interval that contains $\theta$ with posterior probability of, say, 68%. Parameters we do not care about (nuisance parameters $\nu$, for example a detector calibration) are integrated out:

$$P(\theta \mid D) = \int P(\theta, \nu \mid D)\, d\nu .$$

Comparing models. For two models $M_1$ and $M_2$, Bayes' theorem applied to the models gives

$$\frac{P(M_1 \mid D)}{P(M_2 \mid D)} = \frac{P(D \mid M_1)}{P(D \mid M_2)} \cdot \frac{P(M_1)}{P(M_2)} .$$

The ratio of evidences $P(D \mid M_1)/P(D \mid M_2)$ is the Bayes factor. Because the evidence averages the likelihood over the prior, a model with many adjustable parameters spreads its prior over a large space and is penalized unless the data need those parameters.

In practice the integrals are rarely doable in closed form, and posteriors are sampled numerically, typically with Markov chain Monte Carlo methods.

• An Intuitive (and Short) Explanation of Bayes’ Theorem by Kalid Azad
• Understanding Bayes Theorem With Ratiosby Kalid Azad
• Bayesian data analysis for newcomers by John K. Kruschke and Torrin M. Liddell; (Available also here)
• Bayes' Theorem Examples: A Visual Introduction For Beginners by Dan Morris (book)

• Think Bayes - Bayesian Statistics Made Simple

• Probability Theory as Extended Logic

• Probability Theory: The Logic of Science by Edwin T. Jaynes

Abstract

In measure-theoretic form, a Bayesian model consists of a parameter space $(\Theta, \mathcal{A})$ with a prior probability measure $\Pi$, a sample space $(\mathcal{X}, \mathcal{B})$, and a Markov kernel $\theta \mapsto P_\theta$ assigning to each parameter a probability measure on $\mathcal{X}$. These define a joint probability measure on $\Theta \times \mathcal{X}$,

$$Q(A \times B) = \int_A P_\theta(B)\, \Pi(d\theta) .$$

The posterior is the conditional distribution of $\theta$ given $x$: a kernel $x \mapsto \Pi(\cdot \mid x)$ such that

$$Q(A \times B) = \int_B \Pi(A \mid x)\, Q_{\mathcal{X}}(dx)$$

for all measurable $A$ and $B$, where $Q_{\mathcal{X}}$ is the marginal of $Q$ on $\mathcal{X}$. Such a regular conditional distribution exists when $\Theta$ is a standard Borel space, and it is unique up to $Q_{\mathcal{X}}$-null sets. If the model is dominated, $P_\theta(dx) = p(x \mid \theta)\, \mu(dx)$ for a $\sigma$-finite measure $\mu$, the posterior has a density with respect to the prior and we recover Bayes' formula:

$$\frac{d\Pi(\cdot \mid x)}{d\Pi}(\theta) = \frac{p(x \mid \theta)}{\int_\Theta p(x \mid \theta')\, \Pi(d\theta')} .$$

Conjugacy in exponential families. If $p(x \mid \theta) = h(x) \exp\big(\eta(\theta) \cdot T(x) - A(\theta)\big)$, then priors of the form $\pi(\theta) \propto \exp\big(\eta(\theta) \cdot \chi - \nu A(\theta)\big)$ are closed under updating: after $n$ independent observations the hyperparameters become $\chi \to \chi + \sum_{i=1}^n T(x_i)$ and $\nu \to \nu + n$. The Beta-binomial example in the Concrete layer is a special case.

Reparametrization and Jeffreys' prior. A prior density is a density, so it transforms with a Jacobian under $\theta \to \phi(\theta)$. A "flat" prior in one parametrization is not flat in another. The Jeffreys prior $\pi(\theta) \propto \sqrt{\det I(\theta)}$, built from the Fisher information $I_{ij}(\theta) = \mathbb{E}_\theta[\partial_i \log p \, \partial_j \log p]$, is invariant: $I$ transforms as a metric on $\Theta$, and $\sqrt{\det I}\, d\theta$ is its Riemannian volume form.

Large-sample behaviour. For a finite-dimensional model with independent identically distributed data from $P_{\theta_0}$, under regularity conditions that include smoothness of the model, $\theta_0$ in the interior of $\Theta$, non-singular Fisher information and a prior with continuous positive density at $\theta_0$, the Bernstein-von Mises theorem states that the total variation distance between the posterior and the normal distribution $N\big(\hat\theta_n, I(\theta_0)^{-1}/n\big)$ goes to zero in $P_{\theta_0}$-probability, with $\hat\theta_n$ the maximum likelihood estimator. In this regime Bayesian credible intervals and frequentist confidence intervals agree asymptotically. In infinite-dimensional models this can fail.

Foundations. Cox's theorem gives conditions (whose precise form has been debated) under which any rule for assigning real-valued plausibilities to propositions, consistent with Boolean logic and a few regularity requirements, is equivalent to probability theory. This is the basis of the view of probability as extended logic in Jaynes' book linked in the Concrete layer.

Path to this page

1. Calculus
2. Statistics