Lagrange Multipliers: How to Optimise When You Are Not Free to Move Anywhere

You want to maximise something — say, the area of a rectangular garden — and you have a constraint — say, you only have $20$ metres of fencing. You can choose any dimensions you like, so long as the total perimeter does not exceed the fencing. The free choice has been restricted to a curve in the space of possibilities (the set of dimensions whose perimeter is exactly $20$), and you have to find the best point on that curve.

This is a constrained optimisation problem, and it cannot be solved by the elementary calculus technique of setting the derivative to zero. The condition “derivative equals zero” applies to unconstrained extrema — points where you can move in any direction and the function still does not increase. On a constraint curve, you cannot move in any direction; you can only move along the curve. The standard derivative test is not even defined in the usual way, because the function is being asked to do something its definition does not entail.

Lagrange’s elegant fix, published in 1788 in his Mécanique analytique, is now a workhorse of mathematics, physics, economics, engineering, and machine learning. It introduces a single auxiliary number called a Lagrange multiplier and reduces every constrained-optimisation problem to a system of equations that can be solved with ordinary algebra. This article is about that method, the deceptively simple geometric insight behind it, and the surprising depth of meaning the multiplier itself carries.

The geometric insight

Picture the function $f(x, y)$ you want to optimise as a landscape — every $(x, y)$ has an altitude $f(x, y)$. The level curves $\{f = c\}$ trace out contour lines of constant altitude, exactly like a topographic map. The constraint $g(x, y) = 0$ is a curve drawn on the same map. You want the highest point on the constraint curve.

Walk along the constraint. As you move, your altitude changes; you cross one contour line after another. At an optimum — say, a maximum — something specific must be true: just before the maximum you were climbing across contours of increasing altitude, and just after you are crossing contours of decreasing altitude. At the maximum itself the constraint must be tangent to a contour line. If the constraint crossed the contour transversally, you could move in one direction to enter a higher contour, so the point would not be a maximum after all. Tangency is the necessary condition.

Tangency to a contour line of $f$ means the gradient of $f$ — which by definition is perpendicular to its own contour lines — is perpendicular to the constraint at the optimum. The gradient of $g$ is also perpendicular to the constraint (by the same logic applied to the constraint level set). So both gradients are perpendicular to the constraint at the optimum, which means they are parallel to each other:

$\nabla f(x, y) = \lambda \nabla g(x, y).$

The constant $\lambda$ — the Lagrange multiplier — encodes how the two gradients are scaled relative to one another. The optimisation problem has been reduced to solving this vector equation together with the constraint equation $g(x, y) = 0$.

A worked example

Take the classic problem: maximise $f(x, y) = xy$ subject to $x + y = 10$.

The gradients are $\nabla f = (y, x)$ and $\nabla g = (1, 1)$. Lagrange’s condition gives the two scalar equations

$y = \lambda, \qquad x = \lambda.$

So $x = y$. Combined with $x + y = 10$, the optimum is $x = y = 5$, with $\lambda = 5$ and maximum value $f = 25$.

The picture below shows what is happening. The contour lines $xy = 10, 25, 40$ are hyperbolas. The constraint line $x + y = 10$ runs diagonally across the plot. At the optimum $(5, 5)$, the constraint is tangent to the contour $xy = 25$; on either side of this point the constraint dips into contours of smaller value. You can see by inspection that no point on the constraint reaches the higher contour $xy = 40$.

At the optimum the green contour line $xy = 25$ touches the red constraint line tangentially, and the two gradient arrows — green for $\nabla f$, red for $\nabla g$ — point in the same direction. This is the entire content of Lagrange’s method in one picture.

The Lagrangian function

The condition $\nabla f = \lambda \nabla g$ together with $g = 0$ can be written even more compactly. Define the Lagrangian

$\mathcal{L}(x, y, \lambda) = f(x, y) - \lambda\,g(x, y).$

Setting all the partial derivatives of $\mathcal{L}$ to zero gives exactly the Lagrange conditions: $\partial \mathcal{L}/\partial x = f_x - \lambda g_x = 0$, $\partial \mathcal{L}/\partial y = f_y - \lambda g_y = 0$, and $\partial \mathcal{L}/\partial \lambda = -g = 0$. The original constrained problem has been converted into an unconstrained problem in one more variable: find the critical points of $\mathcal{L}$ in $(x, y, \lambda)$. This trick — encoding a constraint as a new degree of freedom — is one of the most fertile ideas in optimisation theory.

What the multiplier means

The multiplier $\lambda$ looks at first like an algebraic device, but it carries a striking economic and physical interpretation called the shadow price. If $g(x, y) = c$ is the constraint (with $c$ the resource budget, say), then the maximum value $f^* = f^*(c)$ depends on $c$, and one can prove that

$\frac{df^*}{dc} = \lambda.$

So $\lambda$ is the sensitivity of the optimal value to relaxation of the constraint. In economics, if $g$ counts the dollars spent and the constraint forces the total to equal the budget $c$, then $\lambda$ is the value of one extra dollar of budget — exactly how much additional benefit (in units of $f$) you could derive if you had one more dollar to spend. Economists call this number the marginal utility of relaxing the constraint or the shadow price, and decisions about whether to acquire more of a resource depend directly on its size.

In physics, when constraints arise from rigid links or contact forces, the multiplier represents (up to a sign) the magnitude of the constraint force. A pendulum on a string moves as if it were a free particle plus a perpendicular force from the string; the tension in the string is the Lagrange multiplier associated with the length constraint. The technique is the founding move of Lagrangian mechanics, the form of classical mechanics that Lagrange himself developed and that became the foundation of modern theoretical physics.

Many constraints, and inequalities

Real problems often have more than one constraint, and the technique generalises gracefully. For $k$ equality constraints $g_1 = g_2 = \cdots = g_k = 0$, the optimality condition becomes

$\nabla f = \lambda_1 \nabla g_1 + \lambda_2 \nabla g_2 + \cdots + \lambda_k \nabla g_k,$

with one multiplier per constraint. Geometrically, the gradient of $f$ must lie in the subspace spanned by the constraint gradients — the only directions in which the constraints would not permit motion.

For inequality constraints $g(x, y) \leq 0$, the situation is subtler. At the optimum, the constraint is either active ($g = 0$) or inactive ($g < 0$). If inactive, the constraint plays no role and the regular unconstrained condition $\nabla f = 0$ applies. If active, the Lagrange condition $\nabla f = \lambda \nabla g$ applies with the extra requirement that the multiplier is non-negative — moving inside the allowed region must not decrease $f$. The full set of conditions is called the Karush–Kuhn–Tucker (KKT) conditions, formulated in the 1940s, and they are the foundation of modern constrained optimisation. KKT underlies support vector machines, optimal control, economic equilibrium theory, and a great deal of machine learning.

The breadth of the technique

Lagrange multipliers and their KKT extension are everywhere. In economics, consumer utility maximisation subject to a budget constraint is exactly Lagrange. In operations research, the simplex algorithm for linear programming uses Lagrange duality at its core. In engineering, optimal design under stress or weight constraints uses Lagrange techniques routinely. In machine learning, regularised regression and constrained neural-network training use Lagrange-multiplier ideas, often under the name “dual problem.” In physics, all of classical and quantum mechanics with constraints is built on this foundation.

The deep reason for the wide applicability is the principle of duality that Lagrange multipliers encode. Every constrained optimisation problem comes with a partner — its dual problem — in which the roles of variables and constraints are swapped. The two problems share the same optimal value (under reasonable conditions called strong duality), and solving either solves the other. Lagrange multipliers are the bridge between the two. This duality is one of the most elegant ideas in applied mathematics and has been refined and generalised throughout the twentieth century into a whole branch of the subject called convex optimisation.

What started as Lagrange’s geometrical observation — that contour lines and constraint curves must be tangent at an optimum — has grown into the central technique by which the modern mathematical world solves the question: what is the best decision you can make, given the limits on what you can do?

Frequently asked

Who invented Lagrange multipliers?

Joseph-Louis Lagrange introduced the method in his 1788 treatise Mécanique analytique, in the course of formulating classical mechanics as a constrained optimization problem. The technique was not entirely new — Euler had used similar ideas earlier — but Lagrange's systematic presentation made it the standard tool for handling constraints in physics and analysis. The same technique now bears his name in mathematics, economics, and machine learning, even though Lagrange himself was thinking about pendulums and planets.

Why does the gradient have to be parallel to the constraint's gradient at the optimum?

Imagine standing at a point on the constraint curve. The constraint allows you to move in only one direction — along the curve. If the gradient of f at that point has any component in this allowed direction, moving slightly along the curve would increase or decrease f, so the point cannot be an optimum. The only way no allowed direction changes f is if the gradient of f has zero component along the curve — equivalently, the gradient of f is perpendicular to the curve. The constraint gradient ∇g is also perpendicular to the curve by construction, so the two gradients must be parallel.

What does the multiplier λ actually mean?

The multiplier λ has a beautiful economic interpretation called the shadow price. It measures how much the optimal value of f would change if you relaxed the constraint by one unit. In an economics problem where g is a budget constraint, λ is the value of one extra dollar of budget — exactly the dollar's worth of additional utility you could buy. In physics, the multiplier associated with a constraint force is, up to a sign, the magnitude of that force. The multiplier is not a meaningless technical parameter; it carries real information about the problem.

What if there are multiple constraints?

Each one gets its own multiplier. For constraints g_1 = g_2 = ⋯ = g_k = 0, the optimality condition becomes ∇f = λ_1 ∇g_1 + λ_2 ∇g_2 + ⋯ + λ_k ∇g_k, with k multipliers to solve for alongside the original variables. The Karush–Kuhn–Tucker (KKT) conditions extend this further to handle inequality constraints (g ≤ 0), in which case the multiplier must be non-negative and is zero unless the constraint is active. KKT is the foundation of modern constrained optimization, used everywhere from portfolio theory to neural-network training.

Continue reading

Blog

The Mean Value Theorem: At Some Point, the Slope Must Match the Average

Drive at an average speed of 100 km/h between two cities, and at some instant your speedometer must have read exactly 100. That obvious fact is the deep engine behind much of calculus.

Read →Blog

The Fundamental Theorem of Calculus: Why Areas and Slopes Are Two Sides of the Same Idea

Adding up infinitely many thin strips and finding the steepness of a curve look like utterly different problems. One short theorem reveals that they are exact inverses of each other.

Read →Blog

Taylor Series: How a Function Becomes an Infinite Polynomial

Pick a smooth function, write down its value, slope, curvature, and every higher derivative at a single point — and you have rebuilt the whole function from that one spot, as an infinite polynomial.

Read →