Eigenvectors and eigenvalues

A basis in which a matrix becomes diagonal must be made of vectors that the transformation merely stretches, and those vectors can be found by solving one polynomial equation.

The previous lesson showed what a change of basis does, and twice produced a diagonal matrix from a basis chosen without explanation. A diagonal matrix in a basis $b_{1}, b_{2}, \dots$ says exactly that each $b_{i}$ is sent to a multiple of itself, since column $i$ of the new matrix is the image of $b_{i}$ in the new coordinates, and having only the $i$ th entry non-zero means the image is a multiple of $b_{i}$ . So the search is for directions the map does not turn.

The definition

A non-zero vector $v$ is an eigenvector of a square matrix $A$ when

A v = λ v

for some scalar $λ$ , called the corresponding eigenvalue. The transformation may stretch $v$ , shrink it, or reverse it, but it leaves it on the line through the origin that it started on.

The requirement that $v$ be non-zero is not pedantry: $A 0 = λ 0$ holds for every $λ$ and would make the definition empty. Eigenvalues, by contrast, may perfectly well be zero, and $λ = 0$ means $A v = 0$ for a non-zero $v$ , which says the null space is non-trivial and the matrix is singular. So a matrix is invertible exactly when zero is not one of its eigenvalues, which is the determinant criterion in a new dress.

Note also that eigenvectors come in lines, not as individual arrows: if $v$ works then so does $c v$ for any non-zero $c$ , with the same eigenvalue, since $A (c v) = c A v = c λ v = λ (c v)$ . Quoting an eigenvector always means quoting one representative of a whole line.

Some are visible without any computation. For the projection $[(1, 0), (0, 0)]$ , the vector $(1, 0)$ is fixed, so it is an eigenvector with eigenvalue $1$ , and $(0, 1)$ is sent to $0$ , so it is an eigenvector with eigenvalue $0$ . For a reflection, everything along the mirror has eigenvalue $1$ and everything perpendicular to it has eigenvalue $- 1$ . For a scaling by $c$ , every vector is an eigenvector with eigenvalue $c$ .

Finding them

Rewrite the definition as $A v - λ v = 0$ , and insert the identity so that both terms are matrices acting on $v$ :

(A - λ I) v = 0

This says $v$ is in the null space of $A - λ I$ . A non-zero vector is there only if that matrix is singular, and the test for singularity is the determinant. So the eigenvalues are exactly the solutions of the characteristic equation

\det (A - λ I) = 0

and once a $λ$ is known, its eigenvectors are found by solving the singular system $(A - λ I) v = 0$ , whose solutions form a subspace called the eigenspace of $λ$ .

For a two by two matrix the equation takes a memorable form. With $A = [(a, c), (b, d)]$ , the matrix $A - λ I$ is $[(a - λ, c), (b, d - λ)]$ , whose determinant is $(a - λ) (d - λ) - b c = λ^{2} - (a + d) λ + (a d - b c)$ . So

λ^{2} - (trace A) λ + \det A = 0

which is why the trace was worth naming in the previous lesson. Comparing coefficients with $(λ - λ_{1}) (λ - λ_{2})$ gives two facts that hold in every dimension: the eigenvalues sum to the trace and multiply to the determinant. Both are free checks on any computation, and both explain why trace and determinant survive a change of basis, since eigenvalues plainly do.

Example. Find the eigenvalues and eigenvectors of $A = [(4, 1), (- 2, 1)]$ .

The trace is $4 + 1 = 5$ and the determinant is $(4) (1) - (- 2) (1) = 6$ , so the characteristic equation is $λ^{2} - 5 λ + 6 = 0$ , factoring as $(λ - 2) (λ - 3) = 0$ . The eigenvalues are $2$ and $3$ , which sum to $5$ and multiply to $6$ as required.

For $λ = 2$ , solve $(A - 2 I) v = 0$ . That matrix is $[(2, 1), (- 2, - 1)]$ , whose rows are $(2, - 2)$ and $(1, - 1)$ , so both equations say $v_{1} = v_{2}$ and the eigenvector is $(1, 1)$ . Check: $A (1, 1) = 1 (4, 1) + 1 (- 2, 1) = (2, 2) = 2 (1, 1)$ .

For $λ = 3$ , the matrix is $[(1, 1), (- 2, - 2)]$ , with rows $(1, - 2)$ and $(1, - 2)$ , so $v_{1} = 2 v_{2}$ and the eigenvector is $(2, 1)$ . Check: $A (2, 1) = 2 (4, 1) + 1 (- 2, 1) = (6, 3) = 3 (2, 1)$ .

Now you. Find the eigenvalues and eigenvectors of $B = [(1, 2), (2, 1)]$ .

Answer

The trace is $2$ and the determinant is $1 - 4 = - 3$ , so $λ^{2} - 2 λ - 3 = 0$ , that is $(λ - 3) (λ + 1) = 0$ , giving $λ = 3$ and $λ = - 1$ . For $λ = 3$ , the matrix $B - 3 I$ has rows $(- 2, 2)$ and $(2, - 2)$ , so $v_{1} = v_{2}$ and the eigenvector is $(1, 1)$ ; check $B (1, 1) = (3, 3)$ . For $λ = - 1$ , the rows are $(2, 2)$ and $(2, 2)$ , so $v_{1} = - v_{2}$ and the eigenvector is $(1, - 1)$ ; check $B (1, - 1) = 1 (1, 2) - 1 (2, 1) = (- 1, 1) = - 1 (1, - 1)$ . The negative eigenvalue means that direction is reversed as well as kept.

Reading the numbers

An eigenvalue is a stretch factor along its own direction, and its size and sign say what happens there. A value greater than one stretches, between zero and one shrinks, negative reverses, and exactly one leaves the direction fixed pointwise. Zero collapses it.

That reading makes repeated application easy, which is the main practical use of the whole idea. Since $A^{2} v = A (λ v) = λ^{2} v$ and so on, $A^{k} v = λ^{k} v$ : along an eigenvector, applying the matrix a thousand times is one exponentiation. Whichever eigenvalue is largest in absolute value comes to dominate, since its power grows fastest, so the long-run behaviour of a repeated linear process is governed by the largest eigenvalue and its direction. The next lesson makes that quantitative.

Two more consequences are worth having. If $A$ is invertible and $A v = λ v$ , then applying $A^{- 1}$ to both sides gives $A^{- 1} v = (1 / λ) v$ : the inverse has the same eigenvectors with reciprocal eigenvalues. And $(A + c I) v = (λ + c) v$ , so adding a multiple of the identity shifts every eigenvalue by $c$ and moves no eigenvector, a trick used constantly in numerical work.

A triangular matrix gives its eigenvalues away, since $\det (A - λ I)$ for a triangular matrix is the product of its diagonal entries minus $λ$ . The matrix with columns $(2, 0, 0)$ , $(1, 3, 0)$ and $(4, 5, 7)$ has eigenvalues $2$ , $3$ and $7$ , read straight off the diagonal, and that is one reason elimination-based algorithms aim at triangular forms.

Example. The matrix $A = [(2, 1), (1, 2)]$ has eigenvalues $3$ and $1$ with eigenvectors $(1, 1)$ and $(- 1, 1)$ . What is $A^{10} (1, 1)$ , and what does $A$ do to $(2, 1)$ in the long run?

$A^{10} (1, 1) = 3^{10} (1, 1) = 59049 (1, 1)$ . For $(2, 1)$ , write it in the eigenvector basis: $(2, 1) = 1.5 (1, 1) - 0.5 (- 1, 1)$ , since $1.5 + 0.5 = 2$ and $1.5 - 0.5 = 1$ . Then $A^{k} (2, 1) = 1.5 \times 3^{k} (1, 1) - 0.5 \times 1^{k} (- 1, 1)$ . The first term grows without bound and the second stays fixed, so the direction of $A^{k} (2, 1)$ converges to $(1, 1)$ , and after ten steps the two terms are $88573.5 (1, 1)$ against $0.5 (- 1, 1)$ , so the deviation is already invisible.

Now you. For the same $A$ , what are the eigenvalues of $A^{- 1}$ and of $A + 2 I$ ?

Answer

$A^{- 1}$ has eigenvalues $1 / 3$ and $1$ , with the same eigenvectors. $A + 2 I$ has eigenvalues $5$ and $3$ , again with the same eigenvectors. As a check, $A + 2 I = [(4, 1), (1, 4)]$ has trace $8 = 5 + 3$ and determinant $16 - 1 = 15 = 5 \times 3$ .

A three by three, and what its eigenvectors mean

Larger matrices work the same way, with a cubic to solve instead of a quadratic. Take the matrix with columns $(2, - 1, 0)$ , $(- 1, 2, - 1)$ and $(0, - 1, 2)$ , which is symmetric and arises from three equal masses in a line, each joined by identical springs to its neighbours and the outer two to fixed walls.

Expanding $\det (A - λ I)$ along the first row, with $μ = 2 - λ$ for brevity, gives $μ (μ^{2} - 1) - (- 1) (- μ) = μ^{3} - 2 μ$ , which factors as $μ (μ^{2} - 2)$ . So $μ = 0$ or $μ = \pm \sqrt{2}$ , and the eigenvalues are

λ = 2, λ = 2 - \sqrt{2} = 0.5858, λ = 2 + \sqrt{2} = 3.4142

They sum to $6$ , matching the trace, and multiply to $2 (4 - 2) = 4$ , matching the determinant, which expands directly to $2 (4 - 1) - (- 1) (- 2) = 4$ .

Each eigenvector is found by solving $(A - λ I) v = 0$ . For $λ = 2$ the first equation is $- v_{2} = 0$ and the second is $- v_{1} - v_{3} = 0$ , giving $v = (1, 0, - 1)$ : the middle mass stays still while the outer two move oppositely. For $λ = 2 - \sqrt{2}$ the equations give $v_{2} = \sqrt{2} v_{1}$ and $v_{1} = v_{3}$ , so $v = (1, 1.4142, 1)$ : all three move the same way with the middle one furthest. For $λ = 2 + \sqrt{2}$ the sign flips, giving $v = (1, - 1.4142, 1)$ , with the middle mass opposing its neighbours. Checking the last: $A (1, - 1.4142, 1) = (2 + 1.4142, - 1 - 2.8284 - 1, 1.4142 + 2) = (3.4142, - 4.8284, 3.4142)$ , which is $3.4142$ times $(1, - 1.4142, 1)$ .

Those three patterns are the normal modes of the system: the shapes in which it can oscillate without changing shape, each at its own frequency, proportional to the square root of its eigenvalue. Any motion whatever is a combination of the three, which is why eigenvectors are how vibration is analysed, in a violin string, a bridge deck or a molecule. The highest mode here has $\sqrt{3.4142} / \sqrt{0.5858} = 2.414$ times the frequency of the lowest.

When it fails

Two failures are guaranteed to be met, and pretending otherwise makes the next lesson impossible to understand.

A rotation has no real eigenvectors. Take the quarter turn $R = [(0, 1), (- 1, 0)]$ . Its trace is $0$ and its determinant is $1$ , so the characteristic equation is $λ^{2} + 1 = 0$ , which has no real solution. The geometry says the same thing more plainly: a quarter turn moves every direction, so no line survives. For a rotation by $θ$ the equation is $λ^{2} - 2 \cos θ λ + 1 = 0$ , whose discriminant $4 \cos^{2} θ - 4$ is negative unless $θ$ is a multiple of $180 °$ .

Allowing complex numbers rescues the algebra: the roots are $\cos θ \pm i \sin θ$ , of modulus one, which is the algebra reporting that a rotation preserves length while turning by $θ$ . Complex eigenvalues of a real matrix always arrive in conjugate pairs, and a pair with non-zero imaginary part always signals rotation of some kind. This course keeps to real numbers, so a rotation is simply a matrix with no eigenvectors here, and the complex theory is the standard next step.

A shear has too few. Take $S = [(1, 0), (1, 1)]$ , with trace $2$ and determinant $1$ , so the equation is $λ^{2} - 2 λ + 1 = (λ - 1)^{2} = 0$ and $λ = 1$ is a repeated root. Solving $(S - I) v = 0$ : the matrix $S - I$ has columns $(0, 0)$ and $(1, 0)$ , so its rows are $(0, 1)$ and $(0, 0)$ , and the condition is $v_{2} = 0$ . The eigenspace is only the line through $(1, 0)$ , one dimension, though the eigenvalue was repeated twice. Geometrically, the horizontal axis is fixed and every other direction is tilted, so there is nowhere else to look.

A matrix like this, whose eigenvectors do not supply a full basis, is called defective, and no change of basis makes it diagonal. That the failure is possible is exactly what makes the next lesson's theorem worth having.

Example. How many independent eigenvectors does the rotation by $60 °$ have in the real plane, and how many does $[(1, 0), (0, 1)]$ have?

The rotation has none: its characteristic equation is $λ^{2} - λ + 1 = 0$ , with discriminant $1 - 4 = - 3 < 0$ , since $2 \cos 60 ° = 1$ . The identity has an eigenvalue $1$ repeated twice, and every non-zero vector is an eigenvector, so it has two independent ones and is already diagonal. A repeated eigenvalue therefore does not by itself imply trouble.

Now you. The matrix $[(5, 0), (0, 5)]$ and the matrix $[(5, 0), (1, 5)]$ both have $λ = 5$ twice. How many independent eigenvectors does each have?

Answer

The first is $5 I$ , which scales everything, so every vector is an eigenvector and there are two independent ones. For the second, $A - 5 I$ has columns $(0, 0)$ and $(1, 0)$ , so the condition is $v_{2} = 0$ and only the line through $(1, 0)$ qualifies: one independent eigenvector, and the matrix is defective. Two matrices with identical characteristic equations can behave completely differently.

Honest limits

The characteristic polynomial is the right definition and a poor algorithm. Abel proved in 1824 that polynomials of degree five and above have no general solution in radicals, so for a five by five matrix there is no formula for the eigenvalues, and none can exist. Every practical eigenvalue computation is therefore iterative, and the standard method, the QR algorithm from about 1961, repeatedly factorises the matrix into an orthogonal part and a triangular part and reassembles them in the other order, which drives the matrix towards triangular form and reads the eigenvalues off the diagonal. Forming the characteristic polynomial explicitly is also numerically disastrous, since polynomial roots can be wildly sensitive to their coefficients.

Eigenvalues can also be badly conditioned in their own right: for a defective or nearly defective matrix, a perturbation of size $ε$ in the entries can move the eigenvalues by about $\sqrt{ε}$ , so a change in the twelfth digit shifts the answer in the sixth. Symmetric matrices are free of this problem, which is one more reason the final lesson gives them a section to themselves.

When the eigenvectors do form a basis, the change of basis of the previous lesson turns the matrix diagonal, and everything about repeated application becomes arithmetic on a handful of numbers. That is the next lesson.