Lagrange Multiplier

Prerequisites: Calculus.

• Intuitive
• Concrete
• Abstract

Intuitive

• More people should know about Lagrange multipliers by Brian Skinner

Sometimes you want to maximize something (profit, health, utility, floor space) but subject to a constraint. A Lagrange multiplier is an extra variable introduced for each constraint. At the maximum it tells you how much the maximum value would change if the constraint were loosened a little.

Concrete

• No-Nonsense Classical Mechanics by Jakob Schwichtenberg, appendix "Lagrange Multipliers"

Abstract

Let $M$ be a smooth manifold, $f: M \to \mathbb{R}$ smooth, and $g = (g^1, \dots, g^k): M \to \mathbb{R}^k$ smooth with $0$ a regular value. Then $N = g^{-1}(0)$ is an embedded submanifold of codimension $k$ with tangent spaces $T_pN = \ker dg_p$. A point $p \in N$ is a critical point of $f|_N$ exactly when $df_p$ vanishes on $\ker dg_p$. A linear functional vanishes on the common kernel of $dg^1_p, \dots, dg^k_p$ if and only if it is a linear combination of them, so the condition is

$$df_p = \sum_{a = 1}^k \lambda_a\, dg^a_p$$

for unique numbers $\lambda_a$ (unique because the $dg^a_p$ are linearly independent). Equivalently, $(p, \lambda)$ is an unconstrained critical point of the Lagrange function $\mathcal{L}(p, \lambda) = f(p) - \lambda_a g^a(p)$ on $M \times \mathbb{R}^k$. Geometrically, $df_p$ lies in the conormal space of $N$ at $p$. Whether a critical point is a maximum or minimum is decided by the Hessian of $\mathcal{L}$ in $p$ at fixed $\lambda$, restricted to $T_pN$.

Multipliers as sensitivities. Replace the constraint by $g = c$ and let $f^*(c)$ be the value of $f$ at a nondegenerate constrained extremum that depends smoothly on $c$. Then

$$\frac{\partial f^*}{\partial c_a} = \lambda_a.$$

This is how multipliers acquire physical meaning. Maximizing the entropy $S/k_B = -\sum_i p_i \ln p_i$ subject to $\sum_i p_i = 1$ and $\sum_i p_i E_i = U$ gives $p_i = e^{-\beta E_i}/Z$, where $\beta$ is the multiplier of the energy constraint. The sensitivity formula gives $\beta = \partial (S/k_B)/\partial U = 1/k_B T$, so temperature enters as a Lagrange multiplier. In mechanics with holonomic constraints $g^a(q) = 0$, the equations of motion are $m\ddot q = -\nabla V + \lambda_a \nabla g^a$, and $\lambda_a \nabla g^a$ is the constraint force.

Duality. For the problem of minimizing $f$ subject to $g = 0$, the dual function $d(\lambda) = \inf_p \mathcal{L}(p, \lambda)$ is a lower bound on the constrained minimum for every $\lambda$ (weak duality). For convex problems that satisfy a constraint qualification such as Slater's condition, the largest of these bounds equals the constrained minimum (strong duality). Inequality constraints lead to the Karush-Kuhn-Tucker conditions, with sign-constrained multipliers that vanish for inactive constraints. In thermodynamics the switch from the constrained problem (fixed $U$) to the unconstrained one (fixed $\beta$) is a Legendre transformation: with the distribution above, $\ln Z = S/k_B - \beta U$, so that $-k_B T \ln Z = U - TS$ is the free energy.