Which optimization method do I use?
Critical points, boundary checks, or Lagrange multipliers — how to pick.
Read the domain FIRST, not the objective function. An unrestricted domain, a closed-and-bounded region, and a domain pinned to a curve or surface each call for a different method — mixing them up (e.g. running the Second Derivative Test on a boundary point) is the most common mistake this cluster exists to prevent.
Unrestricted domain — classify every critical point
Start here always — find every point where the gradient is zero or undefined.
Let f have continuous second-order partial derivatives near a critical point (a, b) (so f_x(a,b) = f_y(a,b) = 0), and define the discriminant D(a,b) = f_{xx}(a,b)\,f_{yy}(a,b) - \big[f_{xy}(a,b)\big]^2.
- If D(a,b) > 0 and f_{xx}(a,b) > 0, then f has a local minimum at (a,b).
- If D(a,b) > 0 and f_{xx}(a,b) < 0, then f has a local maximum at (a,b).
- If D(a,b) < 0, then (a,b) is a saddle point, neither a local max nor a local min.
- If D(a,b) = 0, the test is inconclusive.
Example
Show proof
Near (a,b), Taylor's theorem in two variables writes f as its constant value plus a quadratic form built from the second partials (the linear terms vanish because (a,b) is critical): f(a+h, b+k) \approx f(a,b) + \tfrac12\left(f_{xx}h^2 + 2f_{xy}hk + f_{yy}k^2\right). Completing the square in h (treating f_{xx} \neq 0) turns the bracket into f_{xx}\left(h + \frac{f_{xy}}{f_{xx}}k\right)^2 + \frac{f_{xx}f_{yy} - f_{xy}^2}{f_{xx}}k^2 = f_{xx}\left(h + \frac{f_{xy}}{f_{xx}}k\right)^2 + \frac{D}{f_{xx}}k^2. When D>0, both squared terms carry the same sign as f_{xx} for every direction (h,k), so the quadratic form never changes sign near (a,b) — a min if that sign is positive, a max if negative. When D<0, the two terms have opposite signs: moving purely in h (with k=0) gives one sign, moving purely in k-dominated directions gives the other, so f rises in some directions and falls in others — the saddle signature. When D=0 the quadratic approximation degenerates along some direction, and the cubic-and-higher terms Taylor's theorem dropped are what actually decide the case, which is exactly why the test goes silent there.
Section
The Second Derivative Test
Closed, bounded region — check the interior AND the boundary
If f is continuous on a closed, bounded region D in \mathbb{R}^2 (or \mathbb{R}^3), then f attains both an absolute maximum value and an absolute minimum value somewhere in D. That is, there exist points \mathbf{c}, \mathbf{d} in D with f(\mathbf{c}) \geq f(\mathbf{x}) and f(\mathbf{d}) \leq f(\mathbf{x}) for every \mathbf{x} in D.
Guarantees an absolute max/min exists at all, before you go looking for one.
Show proof
The single-variable Extreme Value Theorem needs a closed, bounded interval; this is the exact same statement one dimension up, and both hypotheses are load-bearing in the same way. "Bounded" keeps D from letting f's values run off to infinity along some escaping direction. "Closed" — D contains its own boundary curve, not just its interior — keeps a candidate maximum from sitting at a boundary point that's excluded from the region, the way f(x,y) = x on the open disk x^2+y^2<1 has no maximum: any point falls short of the open edge, and a point arbitrarily close to the edge is always beaten by one closer still. A full proof again goes through compactness (a closed, bounded region in \mathbb{R}^n is exactly a compact one) and a convergent-subsequence argument identical in spirit to the one-variable case; what matters here is the payoff — on a closed, bounded region, the candidates test (interior critical points plus the entire boundary) is guaranteed to contain the winner.
Domain pinned to a curve or surface — Lagrange multipliers
Suppose f and g have continuous partial derivatives, and f has a local extremum at a point (a,b) on the constraint curve g(x,y) = k, with \nabla g(a,b) \neq \mathbf{0}. Then there is a scalar \lambda (the Lagrange multiplier) with \nabla f(a,b) = \lambda\, \nabla g(a,b). Every constrained extremum is therefore found among the solutions of the system \nabla f = \lambda \nabla g together with g(x,y) = k: three equations (f_x = \lambda g_x, f_y = \lambda g_y, g = k) in three unknowns x, y, \lambda.
Example
Show proof
Parametrize the constraint curve near (a,b) by arc length, \mathbf{r}(t) = (x(t), y(t)), with \mathbf{r}(0) = (a,b) — this is possible precisely because \nabla g(a,b) \neq \mathbf{0}, which keeps the level curve g=k smooth with a well-defined tangent there. Restrict f to the curve: h(t) = f(x(t), y(t)). If f has a local extremum at (a,b) subject to the constraint, then h has an ordinary single-variable local extremum at t=0, so h'(0) = 0. The chain rule expands that as h'(0) = f_x(a,b)\,x'(0) + f_y(a,b)\,y'(0) = \nabla f(a,b)\cdot \mathbf{r}'(0) = 0, meaning \nabla f(a,b) is perpendicular to the curve's tangent vector \mathbf{r}'(0) at that point. But \nabla g(a,b) is also perpendicular to the same tangent vector — that's what it means for g=k to be a level curve of g. Two vectors both perpendicular to the same tangent line in the plane must be parallel to each other, so \nabla f(a,b) = \lambda \nabla g(a,b) for some scalar \lambda.
Suppose f, g, h have continuous partial derivatives, and f has a local extremum at a point \mathbf{p} on the curve where the two constraint surfaces g(x,y,z) = k_1 and h(x,y,z) = k_2 intersect, with \nabla g(\mathbf{p}) and \nabla h(\mathbf{p}) nonzero and not parallel to each other. Then there are scalars \lambda, \mu with \nabla f(\mathbf{p}) = \lambda\, \nabla g(\mathbf{p}) + \mu\, \nabla h(\mathbf{p}). Every constrained extremum is found among the solutions of the system f_x = \lambda g_x + \mu h_x, f_y = \lambda g_y + \mu h_y, f_z = \lambda g_z + \mu h_z, together with g = k_1 and h = k_2: five equations in the five unknowns x, y, z, \lambda, \mu.
Two constraints intersecting in a curve — one extra multiplier, one extra equation.
Show proof
The intersection of the two surfaces g=k_1 and h=k_2 is (near \mathbf{p}, given the non-parallel gradient hypothesis) a smooth space curve, tangent line \mathbf{r}'(0) at \mathbf{p}. The same restrict-and-differentiate argument as the single-constraint case applies: if f has a constrained extremum at \mathbf{p}, then \nabla f(\mathbf{p}) is perpendicular to \mathbf{r}'(0). So are \nabla g(\mathbf{p}) and \nabla h(\mathbf{p}), since the curve lies in both level surfaces at once. The set of vectors perpendicular to a single line in \mathbb{R}^3 is an entire plane, not just a single direction — so unlike the one-constraint case, being perpendicular to \mathbf{r}'(0) doesn't force \nabla f to be parallel to any one gradient. It does force \nabla f(\mathbf{p}) to lie in the plane that \nabla g(\mathbf{p}) and \nabla h(\mathbf{p}) themselves span (the normal plane to the curve, since both are perpendicular to \mathbf{r}'(0) and non-parallel to each other) — and every vector in that plane is some combination \lambda \nabla g + \mu \nabla h.
Section