Lagrange Multipliers

The best value along a path

The bowl $f\left(x,y\right)=x^2+y^2$f(x, y) = x² + y² has its lowest point at the origin. Now ask for the smallest value of f only among the points on the line x + 2y = 5. The origin is not on that line, so the answer is somewhere else: the point of the line closest to the origin.

The line is a constraint, written g(x, y) = c with g = x + 2y and c = 5. On a contour map, the contours of f are circles, and the constraint is a line drawn across them. Walking along the line, you cross the circles, and f goes down and then up again.

The contours of a bowl, cut at heights 4, 8, 12 and 16, and a straight constraint g = k. At the marked point on the line, the gold $\nabla f$∇f points straight out from the center and the plain $\nabla g$∇g stands square to the line: they point different ways, and the small arrow shows the direction along the line in which f falls.

Crossing gradients mean f can still improve

At a point on the path, let t be a unit vector along the path. Moving along the path, f changes at the rate $\nabla f\cdot t$∇f · t. If that is not 0, f falls one way along the path and rises the other, so the point is neither the smallest nor the largest value of f on the path.

So at a constrained maximum or minimum, $\nabla f\cdot t=0$∇f · t = 0: $\nabla f$∇f is at right angles to the path. The path is a level curve of g, and $\nabla g$∇g is always at right angles to the level curves of g. Two vectors at right angles to the same direction are parallel, so at the best point $\nabla f=\lambda \nabla g$∇f = λ∇g for some number $\lambda$λ, the Lagrange multiplier.

The same map with the point moved to where the smallest contour, the ring at height 4, just touches the line. There $\nabla f$∇f and $\nabla g$∇g lie along one ray, $\nabla f$∇f the longer: $\nabla f=\lambda \nabla g$∇f = λ∇g.

Three equations, three unknowns

Written in components, $\nabla f=\lambda \nabla g$∇f = λ∇g is two equations: $\frac{\partial f}{\partial x}=\lambda \frac{\partial g}{\partial x}$∂f/∂x = λ ∂g/∂x and $\frac{\partial f}{\partial y}=\lambda \frac{\partial g}{\partial y}$∂f/∂y = λ ∂g/∂y. With the constraint g(x, y) = c that makes three equations for x, y and $\lambda$λ. Solve the first two for x and y in terms of $\lambda$λ, then substitute into the constraint.

For $x^2+y^2$x² + y² on x + 2y = 5: $\nabla f=\left(2x,2y\right)$∇f = (2x, 2y) and $\nabla g=\left(1,2\right)$∇g = (1, 2), so $2x=\lambda$2x = λ and $2y=2\lambda$2y = 2λ, giving $x=\frac{\lambda }{2}$x = λ/2 and $y=\lambda$y = λ. The constraint becomes $\frac{\lambda }{2}+2\lambda =5$λ/2 + 2λ = 5, so $2.5\lambda =5$2.5λ = 5 and $\lambda =2$λ = 2. The point is (1, 2), which is on the line, since 1 + 4 = 5, and $f\left(1,2\right)=1^2+2^2=5$f(1, 2) = 1² + 2² = 5.

Other points on the line give more: (5, 0) gives 25, and (3, 1) and (−1, 3) both give 10. Far along the line f grows without limit, so 5 is the minimum.

The line x + 2y = 5, the contour $x^2+y^2=5$x² + y² = 5 touching it at (1, 2), and the dashed contour $x^2+y^2=10$x² + y² = 10 crossing it at (3, 1) and (−1, 3). Each $\nabla f$∇f is drawn at a quarter of its length. At (1, 2), $\nabla f=\left(2,4\right)=2\times \left(1,2\right)$∇f = (2, 4) = 2 × (1, 2) stands square to the line; at (3, 1), $\nabla f=\left(6,2\right)$∇f = (6, 2) does not, and f still falls along the line toward (1, 2).

More than one candidate

The equations can give several points, and each must be checked. Find the largest and smallest values of f = xy on the circle $x^2+y^2=8$x² + y² = 8. Here $\nabla f=\left(y,x\right)$∇f = (y, x) and $\nabla g=\left(2x,2y\right)$∇g = (2x, 2y), so $y=2\lambda x$y = 2λx and $x=2\lambda y$x = 2λy. Then $x=2\lambda \left(2\lambda x\right)=4{\lambda }^{2}x$x = 2λ(2λx) = 4λ²x. If x = 0 then y = 0 too, which is not on the circle, so $4{\lambda }^2=1$4λ² = 1 and $\lambda =\frac{1}{2}$λ = 1/2 or $\lambda =-\frac{1}{2}$λ = −1/2.

With $\lambda =\frac{1}{2}$λ = 1/2, y = x, and $2x^2=8$2x² = 8 gives (2, 2) and (−2, −2), where f = 4. With $\lambda =-\frac{1}{2}$λ = −1/2, y = −x, giving (2, −2) and (−2, 2), where f = −4. All four lie on the circle, since 4 + 4 = 8. Comparing the values, the largest is 4 and the smallest is −4.

What $\lambda$λ measures

The multiplier is the rate at which the best value changes as the constant in the constraint changes. To maximize xy on x + y = 10, the equations $y=\lambda$y = λ and $x=\lambda$x = λ give x = y = 5, a maximum of 25, with $\lambda =5$λ = 5. On x + y = 11 the maximum is 5.5 × 5.5 = 30.25, which is 5.25 more: close to $\lambda$λ for a change of 1.

In the bowl example, the smallest value of $x^2+y^2$x² + y² on x + 2y = c is $\frac{c^2}{5}$c²/5. At c = 5 it is 5, and at c = 5.1 it is 5.202, a rise of about 2 for each unit of c, which is $\lambda =2$λ = 2.

The ellipse $\frac{x^2}{4}+y^2=1$x²/4 + y² = 1 is the constraint, and the line x + 2y = k is a contour of f = x + 2y. At k = 1 the line cuts the ellipse at two points, where the plain $\nabla f=\left(1,2\right)$∇f = (1, 2) and the gold $\nabla g=\left(\frac{x}{2},2y\right)$∇g = (x/2, 2y) point different ways, so a larger k can still be reached. Slide k up and the cuts close in, until at $k=2\sqrt{2}\approx 2.83$k = 2√2 ≈ 2.83 the line just touches at $\left(\sqrt{2},\frac{1}{\sqrt{2}}\right)$(√2, 1/√2), where $\nabla f=\sqrt{2}\nabla g$∇f = √2 ∇g.

The usual mistakes

Solving $\nabla f=\lambda \nabla g$∇f = λ∇g without the constraint. The two component equations alone leave $\lambda$λ undetermined; the constraint fixes it.

Dividing by a variable that might be 0. In xy on $x^2+y^2=8$x² + y² = 8, dividing $x=4{\lambda }^{2}x$x = 4λ²x by x is safe only after checking that x = 0 gives no point on the circle.

Stopping at the first candidate. The method gives every point where the gradients are parallel; the largest and smallest values come from comparing f at all of them.

Taking the gradients to be perpendicular at the best point. They are parallel; it is $\nabla f$∇f and the path that are at right angles.

A beam sawn from a log

In the application below, a beam with a rectangular cross-section is cut from a round log, so its width and depth are tied by the circle of the log. Setting the gradient of its strength equal to $\lambda$λ times the gradient of the constraint gives the strongest beam, which is deeper than it is wide.