The gradient

Partial derivatives give the slope of a surface only along the two coordinate directions, yet a walker on a hillside can set off on any bearing, and the slope depends on which one.

The previous lesson ended on this gap, and on a hint: the chain rule, which differentiates $f (x (t), y (t))$ as $f_{x} x^{'} (t) + f_{y} y^{'} (t)$ , gives the rate of change along any path, and walking in a straight line at unit speed is a path. This lesson defines the slope in every direction, shows that two numbers contain all of them, and draws out the geometry. It assumes partial derivatives, the chain rule along a path, and the dot product with its formula $𝐚 \cdot 𝐛 = | 𝐚 | | 𝐛 | \cos θ$ .

The slope in any direction

A direction in the plane is a unit vector $𝐮 = ⟨ a, b ⟩$ with $a^{2} + b^{2} = 1$ . Starting at $(x_{0}, y_{0})$ and walking a distance $h$ in that direction reaches $(x_{0} + h a, y_{0} + h b)$ , and the change in height divided by the distance walked is a difference quotient like any other. The directional derivative of $f$ at $(x_{0}, y_{0})$ in the direction $𝐮$ is its limit:

D_{𝐮} f (x_{0}, y_{0}) = lim_{h \to 0} \frac{f (x_{0} + h a, y_{0} + h b) - f (x_{0}, y_{0})}{h}

The insistence on a unit vector is what makes this a slope. With $| 𝐮 | = 1$ the parameter $h$ is the distance travelled, so the quotient is rise over run. A direction given as $⟨ 3, 4 ⟩$ must first be divided by its length $5$ , or the answer comes out five times too large.

The partial derivatives are two special cases. Taking $𝐮 = 𝐢 = ⟨ 1, 0 ⟩$ gives the quotient $(f (x_{0} + h, y_{0}) - f (x_{0}, y_{0})) / h$ , whose limit is $f_{x}$ , and $𝐮 = 𝐣$ gives $f_{y}$ .

The definition can be applied as it stands. For the paraboloid $f (x, y) = x^{2} + y^{2}$ at $(3, 4)$ , in the direction $𝐮 = ⟨ 0.6, 0.8 ⟩$ pointing away from the origin, the new point is $(3 + 0.6 h, 4 + 0.8 h)$ and

f (3 + 0.6 h, 4 + 0.8 h) = 9 + 3.6 h + 0.36 h^{2} + 16 + 6.4 h + 0.64 h^{2} = 25 + 10 h + h^{2}

The quotient is $10 + h$ , so $D_{𝐮} f (3, 4) = 10$ . That is the value the first lesson of this course found numerically by stepping $0.001$ outward and getting $10.001$ . But every new direction would need the expansion again.

Two numbers contain every slope

The chain rule removes the labour. Fix the point and the direction, and walk the line through the point at unit speed by setting

g (t) = f (x_{0} + t a, y_{0} + t b)

This is a function of the single variable $t$ , and the difference quotient in the definition is exactly $(g (h) - g (0)) / h$ , so $D_{𝐮} f (x_{0}, y_{0}) = g^{'} (0)$ . Now $g$ is $f$ evaluated along the path $x (t) = x_{0} + t a$ , $y (t) = y_{0} + t b$ , whose derivatives are the constants $x^{'} (t) = a$ and $y^{'} (t) = b$ . If $f$ is differentiable, the chain rule along a path gives $g^{'} (t) = f_{x} a + f_{y} b$ , and at $t = 0$

D_{𝐮} f (x_{0}, y_{0}) = f_{x} (x_{0}, y_{0}) a + f_{y} (x_{0}, y_{0}) b

The right side is a dot product. The vector of partial derivatives is called the gradient of $f$ and written with the symbol $\nabla$ , read "del" or "nabla":

\nabla f = ⟨ f_{x}, f_{y} ⟩, D_{𝐮} f = \nabla f \cdot 𝐮

Two numbers computed once at a point now give the slope in every direction through it. For three variables nothing changes but the length: $\nabla f = ⟨ f_{x}, f_{y}, f_{z} ⟩$ , the line is $(x_{0} + t a, y_{0} + t b, z_{0} + t c)$ , and the chain rule produces three terms instead of two. On the paraboloid, $\nabla f = ⟨ 2 x, 2 y ⟩ = ⟨ 6, 8 ⟩$ at $(3, 4)$ , and $⟨ 6, 8 ⟩ \cdot ⟨ 0.6, 0.8 ⟩ = 3.6 + 6.4 = 10$ , without expanding anything.

The hypothesis matters. Take $f (x, y) = x^{2} y / (x^{2} + y^{2})$ with $f (0, 0) = 0$ . Along any line through the origin $f (h a, h b) = h a^{2} b$ , so every directional derivative exists and equals $a^{2} b$ . Both partials are $0$ (put $b = 0$ , then $a = 0$ ), so the formula would predict $0$ in every direction, yet along $⟨ 0.6, 0.8 ⟩$ the true slope is $0.36 \times 0.8 = 0.288$ . This function has a slope in every direction yet no tangent plane, and the formula holds for differentiable functions only, which includes every function whose partials are continuous.

Example. Find the slope of $f (x, y) = x^{3} - 2 x y + y^{2}$ at $(1, 2)$ in the direction of $⟨ 3, 4 ⟩$ .

The partials are $f_{x} = 3 x^{2} - 2 y$ and $f_{y} = - 2 x + 2 y$ , so at $(1, 2)$ they are $3 - 4 = - 1$ and $- 2 + 4 = 2$ , and $\nabla f (1, 2) = ⟨ - 1, 2 ⟩$ . The direction has length $5$ , so $𝐮 = ⟨ 0.6, 0.8 ⟩$ . Then $D_{𝐮} f = - 1 \times 0.6 + 2 \times 0.8 = - 0.6 + 1.6 = 1$ . A difference quotient with a step of $10^{- 6}$ gives $1.0000008$ , agreeing.

Now you. Find the rate of change of $f (x, y, z) = x y z$ at $(1, 2, 3)$ in the direction of $⟨ 2, - 1, 2 ⟩$ .

Answer

The gradient is $⟨ y z, x z, x y ⟩ = ⟨ 6, 3, 2 ⟩$ at the point. The direction has length $3$ , so $𝐮 = \frac{1}{3} ⟨ 2, - 1, 2 ⟩$ and $D_{𝐮} f = \frac{1}{3} (12 - 3 + 4) = \frac{13}{3} = 4.333$ .

The direction of steepest increase

The dot product formula turns the directional derivative into geometry. If $θ$ is the angle between $\nabla f$ and the unit vector $𝐮$ , then

D_{𝐮} f = | \nabla f | | 𝐮 | \cos θ = | \nabla f | \cos θ

since $| 𝐮 | = 1$ . Only $\cos θ$ depends on the choice of direction, and it lies between $- 1$ and $1$ . So when $\nabla f \neq 𝟎$ , the slope is greatest at $θ = 0$ , walking along the gradient itself, where it equals $| \nabla f |$ . It is least at $θ = π$ , walking straight against the gradient, where it equals $- | \nabla f |$ . It is zero at $θ = π / 2$ , in the two directions perpendicular to the gradient. The gradient points in the direction of steepest increase, and its length is that steepest rate.

The first lesson of this course found four slopes of the paraboloid at $(3, 4)$ by brute force: $6$ due east, $8$ due north, $10$ directly away from the origin, and $0$ along the level circle. All four are $⟨ 6, 8 ⟩ \cdot 𝐮$ . The third is the largest possible, because $⟨ 0.6, 0.8 ⟩$ is the gradient's own direction and $| ⟨ 6, 8 ⟩ | = 10$ , and the fourth is zero because $⟨ - 0.8, 0.6 ⟩$ is perpendicular to it. The infinitely many other directions that lesson left open lie between $- 10$ and $10$ according to the cosine.

The same picture explains an identity from the previous lesson, where $f = x^{2} y$ at $(3, 4)$ had $f_{x}^{2} + f_{y}^{2} = 657$ and also $f_{r}^{2} + (f_{θ} / r)^{2} = {21.6}^{2} + {13.8}^{2} = 657$ . The radial unit vector there is $𝐞_{r} = ⟨ 0.6, 0.8 ⟩$ and the unit vector round the circle is $𝐞_{θ} = ⟨ - 0.8, 0.6 ⟩$ , and with $\nabla f = ⟨ 24, 9 ⟩$ the two dot products are $21.6 = f_{r}$ and $- 13.8 = f_{θ} / r$ . So $f_{x}$ , $f_{y}$ and $f_{r}$ , $f_{θ} / r$ are the components of one vector, the gradient, measured along two different pairs of perpendicular unit directions. Both sums of squares are $| \nabla f |^{2}$ , and a length does not depend on which perpendicular axes it is measured along.

Example. A hill has height $h (x, y) = 500 - 0.0005 x^{2} - 0.001 y^{2}$ metres, with $x$ metres east and $y$ metres north of the summit. A walker stands at $(300, 200)$ . Which way is steepest uphill, how steep is it, and how does it compare with heading straight for the summit?

The height there is $500 - 45 - 40 = 415$ metres. The gradient is $⟨ - 0.001 x, - 0.002 y ⟩ = ⟨ - 0.3, - 0.4 ⟩$ , of length $0.5$ , so the steepest climb heads in the direction $⟨ - 0.6, - 0.8 ⟩$ at a gradient of $0.5$ , which is $\arctan 0.5 = 26.6$ degrees. The summit lies in the direction $⟨ - 300, - 200 ⟩$ , of length $360.56$ , so $𝐮 = ⟨ - 0.8321, - 0.5547 ⟩$ and $D_{𝐮} h = 0.2496 + 0.2219 = 0.4715$ , about $25.2$ degrees. The hill is narrower north to south, so the fall line bends towards the $y$ axis and does not aim at the top.

Now you. On the hill $h (x, y) = 300 - 0.002 x^{2} - 0.0005 y^{2}$ , find the steepest slope at $(50, 400)$ and its angle, and the slope met by walking due east from there.

Answer

The gradient is $⟨ - 0.004 x, - 0.001 y ⟩ = ⟨ - 0.2, - 0.4 ⟩$ , so the steepest slope is $\sqrt{0.2} = 0.447$ , an angle of $24.1$ degrees, heading in the direction $⟨ - 1, - 2 ⟩$ . Due east is $𝐢$ , giving $h_{x} = - 0.2$ : a descent of $0.2$ metres per metre.

A contour map shows the gradient without any formula. With contours every $10$ metres, as on the Ordnance Survey's Landranger maps, the height changes by $10$ metres between neighbouring contours, and walking straight across them is walking at slope $| \nabla h |$ . So the spacing on the ground is $10 / | \nabla h |$ : at the walker's point, $20$ metres, which is $0.4$ mm of paper at 1:50,000. The first lesson measured a path crossing five contour gaps in $4$ mm and called its gradient $0.25$ . That is $| \nabla h |$ only if the path crosses the contours squarely; a path slanting across them is taking a directional derivative, and the ground's true steepest slope there is larger.

Perpendicular to the contours

The slope is zero in the directions perpendicular to $\nabla f$ , and walking along a contour means not climbing. The two facts are the same fact. Let $x (t)$ , $y (t)$ trace the level curve $f (x, y) = c$ through a point, arriving there at $t = 0$ with velocity $𝐯 = ⟨ x^{'} (0), y^{'} (0) ⟩ \neq 𝟎$ . Since $f (x (t), y (t)) = c$ for every $t$ , its derivative is zero, and the chain rule says

f_{x} x^{'} (t) + f_{y} y^{'} (t) = 0, that is, \nabla f \cdot 𝐯 = 0

So the gradient is perpendicular to the tangent of the level curve through the point. For the paraboloid at $(3, 4)$ , the level circle has tangent direction $⟨ - 4, 3 ⟩$ , and $⟨ 6, 8 ⟩ \cdot ⟨ - 4, 3 ⟩ = 0$ . On a map the fall line crosses every contour at right angles, which is why streams, running down the negative gradient, cut the contours squarely.

The gradient therefore serves as a normal to the curve, and the tangent line at $(x_{0}, y_{0})$ is $f_{x} (x - x_{0}) + f_{y} (y - y_{0}) = 0$ . Its slope is $- f_{x} / f_{y}$ , the implicit differentiation formula of the previous lesson, now read as a statement about a perpendicular.

Tangent planes to level surfaces

A function of three variables has level surfaces $F (x, y, z) = c$ rather than curves, and the argument applies to every curve on the surface through a point $P$ : its velocity at $P$ satisfies $\nabla F \cdot 𝐯 = 0$ . So $\nabla F (P)$ is perpendicular to every direction in which the surface can be left along itself, which makes it a normal to the tangent plane, and when it is not $𝟎$ the plane is

F_{x} (P) (x - x_{0}) + F_{y} (P) (y - y_{0}) + F_{z} (P) (z - z_{0}) = 0

A sphere checks it. For $x^{2} + y^{2} + z^{2} = 9$ at $(1, 2, 2)$ , $\nabla F = ⟨ 2 x, 2 y, 2 z ⟩ = ⟨ 2, 4, 4 ⟩$ , parallel to the radius, as geometry demands, and the plane is $x + 2 y + 2 z = 9$ . A graph is a level surface too: $z = f (x, y)$ is $F = f (x, y) - z = 0$ , with $\nabla F = ⟨ f_{x}, f_{y}, - 1 ⟩$ , minus the normal found earlier in this course by a cross product.

Example. Find the tangent plane to the ellipsoid $\frac{x^{2}}{4} + y^{2} + \frac{z^{2}}{9} = 3$ at $(2, 1, 3)$ .

The point lies on it, since $1 + 1 + 1 = 3$ . The gradient is $⟨ \frac{x}{2}, 2 y, \frac{2 z}{9} ⟩ = ⟨ 1, 2, \frac{2}{3} ⟩$ , or $⟨ 3, 6, 2 ⟩$ after multiplying by $3$ . The plane is $3 (x - 2) + 6 (y - 1) + 2 (z - 3) = 0$ , which is $3 x + 6 y + 2 z = 18$ , meeting the $z$ axis at $z = 9$ .

Now you. Find the tangent plane to the surface $x y z = 6$ at $(1, 2, 3)$ .

Answer

The gradient is $⟨ y z, x z, x y ⟩ = ⟨ 6, 3, 2 ⟩$ , so the plane is $6 (x - 1) + 3 (y - 2) + 2 (z - 3) = 0$ , which is $6 x + 3 y + 2 z = 18$ .

Gradient descent

To find the lowest point of a function too complicated to solve for, walk downhill: from the current point take a small step along $- \nabla f$ , the steepest way down, and repeat. With a step size $η$ the rule is $𝐱_{n + 1} = 𝐱_{n} - η \nabla f (𝐱_{n})$ . Augustin-Louis Cauchy proposed it in 1847, and machine learning now uses it to fit models with millions of variables.

Take $f (x, y) = x^{2} + 3 y^{2}$ , whose minimum is $0$ at the origin, starting from $(3, 1)$ with $η = 0.1$ . The gradient is $⟨ 2 x, 6 y ⟩$ , so each step sends $x$ to $x - 0.2 x = 0.8 x$ and $y$ to $y - 0.6 y = 0.4 y$ . The iterates are $(2.4, 0.4)$ , $(1.92, 0.16)$ and $(1.536, 0.064)$ , with values falling from $12$ to $6.24$ , $3.7632$ and $2.3716$ . The first step did not aim at the minimum: $- \nabla f (3, 1) = ⟨ - 6, - 6 ⟩$ while the origin lies along $⟨ - 3, - 1 ⟩$ , the elongated hill again. The steep $y$ direction settles quickly while $x$ shrinks by only a fifth per step.

The step size is a genuine limitation. With $η = 0.4$ the $y$ update becomes $y - 2.4 y = - 1.4 y$ , so $y$ runs $1, - 1.4, 1.96, - 2.744$ and the values go $12, 6.24, 11.54, 22.59$ : each step overshoots the valley floor further than it started. Too small a step crawls, too large a step diverges, and the safe range depends on how sharply the function curves.

Where the gradient vanishes

Suppose a differentiable $f$ has a local maximum at a point, and $\nabla f \neq 𝟎$ there. Walking a short way along $\nabla f$ climbs at rate $| \nabla f | > 0$ and reaches higher values, so the point was not a maximum. Walking against $\nabla f$ rules out a minimum the same way. At a local maximum or minimum, then, $\nabla f = 𝟎$ : every partial derivative is zero and the tangent plane is horizontal. Gradient descent agrees, since its steps stop exactly where the gradient vanishes.

The converse fails. For $f (x, y) = x^{2} - y^{2}$ the gradient $⟨ 2 x, - 2 y ⟩$ vanishes at the origin, yet the surface rises east and west and falls north and south, like a mountain pass. Setting $\nabla f = 𝟎$ finds the candidates, and telling summits, hollows and passes apart is the work of the next lesson.