Symmetric matrices and quadratic forms

Every difficulty in the previous lesson, missing eigenvectors, complex eigenvalues and ill-conditioned changes of basis, disappears for matrices equal to their own transpose.

A matrix is symmetric when $A = A^{T}$ , which for a two by two means the off-diagonal entries are equal: $[(a, b), (b, d)]$ . The definition looks like a coincidence of notation. It is not, and this lesson proves what symmetry guarantees, shows what such matrices compute, and explains why they dominate applied work.

Two proofs

Eigenvectors of distinct eigenvalues are perpendicular. Suppose $A u = λ u$ and $A v = μ v$ with $λ \neq μ$ . Compute $u \cdot (A v)$ in two ways. Directly, it is $u \cdot (μ v) = μ (u \cdot v)$ . But writing the dot product as a matrix product, $u^{T} A v = (A^{T} u)^{T} v = (A u)^{T} v$ , using symmetry to replace $A^{T}$ by $A$ , and that is $λ (u \cdot v)$ . So $λ (u \cdot v) = μ (u \cdot v)$ , and since $λ \neq μ$ the dot product must be zero.

Three lines, and the geometric content is large: the axes of a symmetric transformation meet at right angles.

The eigenvalues are real. For a general real matrix they need not be, as the rotation showed. Suppose $A v = λ v$ with $λ$ and $v$ possibly complex, and let $\overline{v}$ be the entrywise complex conjugate. The quantity ${\overline{v}}^{T} A v$ equals $λ {\overline{v}}^{T} v$ , and ${\overline{v}}^{T} v$ is the sum of $| v_{i} |^{2}$ , a positive real number. Conjugating and transposing the whole expression, and using that $A$ is real and symmetric, returns the same quantity with $λ$ replaced by $\overline{λ}$ . So $λ = \overline{λ}$ , which for a complex number means it is real.

Together with a slightly harder argument that covers repeated eigenvalues, these give the spectral theorem: every real symmetric matrix has a full set of real eigenvalues and an orthonormal basis of eigenvectors, so

A = Q D Q^{T}

with $Q$ orthogonal and $D$ diagonal and real. The inverse of $Q$ is its transpose, so the change of basis costs nothing and is perfectly conditioned. Symmetric matrices are never defective, never need complex numbers, and never produce a badly conditioned $P$ : all three failure modes of the previous lesson are excluded at once.

Example. Diagonalise $A = [(5, 2), (2, 2)]$ and confirm that its eigenvectors are perpendicular.

The trace is $7$ and the determinant is $10 - 4 = 6$ , so $λ^{2} - 7 λ + 6 = 0$ and the eigenvalues are $6$ and $1$ , both real. For $λ = 6$ , the first row of $A - 6 I$ is $(- 1, 2)$ , so $v_{1} = 2 v_{2}$ and the eigenvector is $(2, 1)$ . For $λ = 1$ , the first row of $A - I$ is $(4, 2)$ , so $2 v_{1} = - v_{2}$ and the eigenvector is $(1, - 2)$ . Their dot product is $2 - 2 = 0$ . Normalising, $Q = [(0.8944, 0.4472), (0.4472, - 0.8944)]$ and $D = [(6, 0), (0, 1)]$ .

Now you. Diagonalise $B = [(3, 1), (1, 3)]$ .

Answer

Trace $6$ , determinant $9 - 1 = 8$ , so $λ^{2} - 6 λ + 8 = 0$ and the eigenvalues are $4$ and $2$ . For $λ = 4$ the first row of $B - 4 I$ is $(- 1, 1)$ , giving the eigenvector $(1, 1)$ ; for $λ = 2$ it is $(1, 1)$ , giving $(1, - 1)$ . Their dot product is zero. Normalised, the two are $(0.7071, 0.7071)$ and $(0.7071, - 0.7071)$ .

Quadratic forms

Symmetric matrices exist to compute quadratic expressions. Given a symmetric $A$ , the function

q (x) = x^{T} A x

is a quadratic form: every term is of degree two. For a two by two $A = [(a, b), (b, d)]$ , expanding gives $q (x, y) = a x^{2} + 2 b x y + d y^{2}$ . So $5 x^{2} + 4 x y + 2 y^{2}$ corresponds to $a = 5$ , $2 b = 4$ and $d = 2$ , which is the matrix $[(5, 2), (2, 2)]$ diagonalised above. Note that the cross term is split evenly between the two off-diagonal entries, which is exactly why the matrix can be taken symmetric with no loss.

Now apply the spectral theorem. In the eigenvector basis the matrix is diagonal, so the form becomes a plain sum of squares with the eigenvalues as coefficients: if $u$ and $v$ are the coordinates along the two orthonormal eigenvectors, then

q = λ_{1} u^{2} + λ_{2} v^{2}

with no cross term at all. For the example, $q = 6 u^{2} + v^{2}$ where $u$ is measured along $(2, 1) / \sqrt{5}$ and $v$ along $(1, - 2) / \sqrt{5}$ . The cross term $4 x y$ was an artefact of using coordinates misaligned with the natural axes of the form.

That makes the level curves readable. The set $q = 1$ is $6 u^{2} + v^{2} = 1$ , an ellipse with semi-axis $1 / \sqrt{6} = 0.4082$ in the $u$ direction and $1$ in the $v$ direction: the larger the eigenvalue, the shorter the axis, since a steeper form reaches the value $1$ sooner. Checking one point directly, $0.4082$ along $(2, 1) / \sqrt{5}$ is $(0.3651, 0.1826)$ , and $5 (0.3651)^{2} + 4 (0.3651) (0.1826) + 2 (0.1826)^{2} = 0.6667 + 0.2667 + 0.0667 = 1.0000$ , on the curve as predicted.

Example. Sketch the curve $3 x^{2} + 2 x y + 3 y^{2} = 1$ by finding its axes.

The matrix is $[(3, 1), (1, 3)]$ , diagonalised above with eigenvalues $4$ and $2$ and eigenvectors $(1, 1)$ and $(1, - 1)$ . So the curve is $4 u^{2} + 2 v^{2} = 1$ , an ellipse with semi-axis $1 / 2 = 0.5$ along $(1, 1) / \sqrt{2}$ and $1 / \sqrt{2} = 0.7071$ along $(1, - 1) / \sqrt{2}$ . Check the first: the point $0.5 (1, 1) / \sqrt{2} = (0.3536, 0.3536)$ gives $3 (0.125) + 2 (0.125) + 3 (0.125) = 1.000$ .

Now you. What shape is the curve $x^{2} + 4 x y + y^{2} = 1$ ?

Answer

The matrix is $[(1, 2), (2, 1)]$ , whose eigenvalues are $3$ and $- 1$ with eigenvectors $(1, 1)$ and $(1, - 1)$ . The form is $3 u^{2} - v^{2}$ , and with one positive and one negative coefficient the curve $3 u^{2} - v^{2} = 1$ is a hyperbola, not an ellipse. Signs of the eigenvalues decide the shape.

Definiteness

A quadratic form is positive definite when $q (x) > 0$ for every non-zero $x$ , and since $q = λ_{1} u^{2} + λ_{2} v^{2} + \dots$ in the eigenvector basis, that happens exactly when every eigenvalue is positive. Negative definite means all negative, and indefinite means mixed signs, which is a saddle.

For a two by two there is a shortcut that avoids finding the eigenvalues: the form is positive definite exactly when $a > 0$ and $\det A > 0$ . The determinant is the product of the eigenvalues, so a positive determinant means they share a sign, and $a = q (1, 0)$ being positive settles which sign. For $[(2, 2), (2, 3)]$ : $a = 2 > 0$ and $\det = 6 - 4 = 2 > 0$ , so positive definite, and the eigenvalues confirm it, being $(5 \pm \sqrt{17}) / 2 = 4.562$ and $0.438$ .

This is the multivariable second derivative test, and anyone who has taken calculus has met it in disguise. At a critical point of a function of several variables, the matrix of second partial derivatives, the Hessian, is symmetric because mixed partials are equal. If it is positive definite the point is a minimum, since the surface curves upward along every direction; negative definite gives a maximum; indefinite gives a saddle. The two by two condition $a > 0$ with $\det > 0$ is precisely the rule usually quoted as $f_{x x} > 0$ and $f_{x x} f_{y y} - f_{x y}^{2} > 0$ .

Example. Is the form $q (x, y) = 4 x^{2} - 4 x y + 3 y^{2}$ positive definite?

The matrix is $[(4, - 2), (- 2, 3)]$ . Here $a = 4 > 0$ and $\det = 12 - 4 = 8 > 0$ , so yes. The eigenvalues, from $λ^{2} - 7 λ + 8 = 0$ , are $(7 \pm \sqrt{17}) / 2 = 5.561$ and $1.439$ , both positive, confirming it. The level curves are ellipses.

Now you. Classify $q (x, y) = x^{2} + 6 x y + y^{2}$ .

Answer

The matrix is $[(1, 3), (3, 1)]$ , with $\det = 1 - 9 = - 8 < 0$ . A negative determinant means the eigenvalues have opposite signs, so the form is indefinite: it is positive along $(1, 1)$ , where $q = 8$ , and negative along $(1, - 1)$ , where $q = - 4$ . The level curves are hyperbolas and the origin is a saddle.

Where symmetric matrices come from

They are not rare. Three sources supply most of the symmetric matrices anyone meets.

Any matrix at all generates one: $A^{T} A$ is symmetric, since $(A^{T} A)^{T} = A^{T} (A^{T})^{T} = A^{T} A$ , and it is positive semidefinite, since $x^{T} A^{T} A x = (A x) \cdot (A x) = ‖ A x ‖^{2} \geq 0$ . That is the matrix at the heart of the normal equations four lessons ago, and its being positive definite whenever the columns of $A$ are independent is what made the least squares solution unique.

Physical laws that pair a cause with an effect are usually symmetric, because the underlying energy is a quadratic form: the stiffness matrix of a structure, the inertia tensor of a rigid body, the conductivity of a crystal. Diagonalising the inertia tensor gives the principal axes about which a body spins without wobbling, which is a physical fact recovered by an eigenvector computation.

And covariance matrices are symmetric by construction, since the covariance of $x$ with $y$ equals the covariance of $y$ with $x$ . Diagonalising one is principal component analysis, which finds the directions of greatest variation in data.

Principal components, on four points

Take the data $(2, 1)$ , $(3, 3)$ , $(4, 3)$ and $(5, 5)$ . The mean is $(3.5, 3)$ , so the centred points are $(- 1.5, - 2)$ , $(- 0.5, 0)$ , $(0.5, 0)$ and $(1.5, 2)$ . Dividing the sums of products by $n - 1 = 3$ gives the variances $S_{x x} = 5 / 3 = 1.6667$ and $S_{y y} = 8 / 3 = 2.6667$ and the covariance $S_{x y} = 6 / 3 = 2$ . The covariance matrix $[(1.6667, 2), (2, 2.6667)]$ has trace $4.3333$ and determinant $4.4444 - 4 = 0.4444$ , so the eigenvalues satisfy $λ^{2} - 4.3333 λ + 0.4444 = 0$ , giving $λ_{1} = 4.2282$ and $λ_{2} = 0.1051$ .

The first eigenvalue accounts for $4.2282 / 4.3333 = 97.6$ per cent of the total variance, so the cloud is very nearly one-dimensional. Its direction solves $(1.6667 - 4.2282) v_{1} + 2 v_{2} = 0$ , that is $v_{2} = 1.2808 v_{1}$ , so the principal direction is $(1, 1.2808)$ , or as a unit vector $(0.6154, 0.7882)$ . Projecting the data onto that single direction loses under three per cent of the variation, which is data compression by eigenvector, and it is the same computation performed on matrices with thousands of rows.

Example. The four points above have covariance eigenvalues $4.2282$ and $0.1051$ . What fraction of the variance survives if the data is projected onto the second principal direction instead of the first?

The total variance is the trace, $4.3333$ , and the second eigenvalue is $0.1051$ , so the fraction is $0.1051 / 4.3333 = 0.0243$ , under two and a half per cent. Keeping the wrong direction discards almost everything, which is why the eigenvalues are always sorted before any are thrown away.

Now you. A covariance matrix in three dimensions has eigenvalues $8.0$ , $1.5$ and $0.5$ . How much of the variance is captured by the first two principal directions?

Answer

The total is $8.0 + 1.5 + 0.5 = 10.0$ , and the first two account for $9.5$ , or $95$ per cent. Dropping the third direction turns three-dimensional data into two-dimensional data at a cost of five per cent of the variation, which is the trade principal component analysis exists to make.

Honest limits, and what comes after

The spectral theorem needs the matrix to be genuinely symmetric, and near-symmetry buys nothing in theory, though in practice a small asymmetry perturbs the eigenvalues only slightly, which is the sense in which symmetric problems are well conditioned. The theorem as stated is for real matrices; the complex analogue replaces the transpose by the conjugate transpose and symmetric by Hermitian, and the whole of quantum mechanics is built on that version, where the real eigenvalues are the possible measured values of an observable.

The obvious gap is that most matrices are neither symmetric nor square. The repair is the singular value decomposition: every $m$ by $n$ matrix, with no assumptions whatever, can be written $A = U Σ V^{T}$ with $U$ and $V$ orthogonal and $Σ$ diagonal with non-negative entries. Geometrically it says that every linear map is a rotation or reflection, followed by a scaling along perpendicular axes, followed by another rotation or reflection: no shear survives once the input and output are allowed their own bases. It follows from this lesson, since the axes are the eigenvectors of the symmetric matrices $A^{T} A$ and $A A^{T}$ , and the singular values are the square roots of their shared eigenvalues. It is what a numerical library actually computes when asked for a rank, a condition number, a least squares fit or a low-rank approximation, and it is the natural next subject after this one.

Look back at what a matrix has become. It began in the first lesson as a bookkeeping device for the coefficients of a system nobody wanted to write out twice. It is now a transformation of space, whose columns say where the basis goes, whose determinant says what it does to area, whose rank and null space say what it destroys, and whose eigenvectors are the directions it treats as its own. Reading those five things off a matrix is what it means to see one as a transformation, and it is what this course was for.