Math 202A: Lecture 21

Given a morphism A \in \mathrm{Hom}(V,W) we have discussed extensively the singular value decomposition of A.

Theorem 21.1. (Geometric SVD, block form) There exists a positive integer k, distinct positive numbers \sigma_1> \dots > \sigma_k>0, and orthogonal decompositions

V=V_1 \oplus \dots \oplus V_k \oplus \mathrm{Ker}(A)\quad\text{and}\quad W=W_1\oplus \dots \oplus W_k \oplus \mathrm{Im}(A)^\perp

such that the restriction of A to V_i equals \sigma_iU_i, where U_i \in \mathrm{Hom}(V_i,W_i) is an isometric isomorphism.

The spaces V_1,\dots,V_k are the left singular blocks of A, the spaces W_1,\dots,W_k are the right singular blocks of A, and the numbers \sigma_1>\dots>\sigma_k are the singular values of A, and the set \{\sigma_1,\dots,\sigma_k\} is the singular spectrum of A.

Problem 21.1. Give a precise formulation and proof of the following statement: the block SVD of a linear transformation is unique. (Hint: one way to write this down formally is using Fermat’s method of infinite descent. Suppose there do exist morphisms which admit two distinct SVDs. Let r be the minimal rank of such a morphism. Prove that the first singular block and singular value in these two decompositions must be the same. Then, the non-uniqueness comes from the restriction of this morphism to the orthogonal complement of the first singular block, which has lower rank; contradiction).

For each 1 \leq i \leq k, the positive number \dim V_i = \dim W_i is the multiplicity of the singular value \sigma_i, and if this number is equal to one we say that \sigma_i is a simple singular value of A. If all singular values of A are simple, then we say that A has simple singular spectrum; this means precisely that A has r=\mathrm{rank}(A) distinct singular values \sigma_1> \dots > \sigma_r. In this case, the left singular blocks V_1,\dots,V_r are one-dimensional subspaces and we call them the left singular lines of A. Likewise, W_1,\dots,W_r are one-dimensional subspaces of W, the right singular lines of A. It is often useful to employ this perspective even when A does not have simple singular spectrum.

Theorem 21.2. (Geometric SVD, line form) There exist r = \mathrm{rank}(A) positive numbers \sigma_1 \geq \dots \geq \sigma_r \geq 0

V=V_1 \oplus \dots \oplus V_r \oplus \mathrm{Ker}(A)\quad\text{and}\quad W=W_1\oplus \dots \oplus W_r \oplus \mathrm{Im}(A)^\perp

such that the restriction of A to V_i equals \sigma_iU_i, where U_i \in \mathrm{Hom}(V_i,W_i) is an isometric isomorphism.

When k=r, the block and line forms of the SVD are literally the same statement. However, when k <r, they are not, because the line form becomes ambiguous: there is at least one i \in \{1,\dots,k\} such that we are decomposing each of the singular blocks V_i,W_i into a direct sum of orthogonal lines, and these decompositions can be performed arbitrarily and independently. I have generally been pretty sloppy about accounting for this, because it almost never happens.

Theorem 21.3. (Generic simplicity) The set \mathrm{Hom}(V,W)^\circ of morphisms with simple singular spectrum is a dense open set in \mathrm{Hom}(V,W).

Proof: Proving this is definitely my job, and I am still thinking about how to present the simplest and most elegant argument. Here is the basic idea. First, notice that I did not specify a norm on \mathrm{Hom}(V,W) in the statement. This is intentional: the vector space \mathrm{Hom}(V,W) is finite-dimensional, so all norms are equivalent (note to self: go back and prove this in the discussion of norms on Hilbert space), and since density is a topological as opposed to geometric property (closure equals whole space) we are free to choose whichever norm is the most convenient. The best choice is operator norm, since this allows us to construct an infinite descent argument compatible with the iterative construction of the SVD based on repeatedly saturating the operator norm; this is similar in spirit to the one used to verify uniqueness of the block SVD. Namely, since the block SVD is constructed iteratively by looking for vectors which saturate the operator norm inequality, we can reduce to verifying that the set of transformations A \in \mathrm{Hom}(V,W) such that V_1(A) = \{v \in V \colon \|Av\|=\sigma_1(A)\|v\|\} is one-dimensional is an operator norm dense open set in \mathrm{Hom}(V,W), which is actually not too difficult. (Note to student: if you can either fine-tune this argument, or find a better one, please show me your approach). \square

Theorem 21.3 means that every time I am being sloppy about non-uniqueness of the line SVD, I am being sloppy on the complement of a dense open set, and such sets do not have civil rights. In the remainder of the lecture I will continue doing this instead of altering all statements to account for the possibility of repeated singular values. The actual meaning of this is that all theorems from this point forward a priori only hold for A \in \mathrm{Hom}(V,W)^\circ a morphism with simple singular spectrum. Theorem 2.3 is an extra incantation which mystically causes these results to hold on all of \mathrm{Hom}(V,W).

For the remainder of the lecture, fix a rank r morphism A \in \mathrm{Hom}(V,W)^\circ with simple singular spectrum \sigma_1>\dots>\sigma_r>0. Let V_1,\dots,V_r be its uniquely determined left singular lines, and let W_1,\dots,W_r be the corresponding right singular lines.

Let us describe the polar decomposition of A \in \mathrm{Hom}(V,W), which is an almost automatic consequence of the upstairs SVD of A,

A = \sigma_1E_1+\dots+\sigma_rE_r,

where E_i=U_iP_i and P_i is the orthogonal projection of V onto the left singular line V_i.

Proposition 21.4. (Left polar decomposition) We have

A = UP

P \in \mathrm{End}(V) is the operator

P=\sigma_1P_1 + \dots + \sigma_rP_r

and U \in \mathrm{Hom}(V,W) is the partial isometry

U=E_1+\dots+E_r=U_1P_1+\dots+U_rP_r.

Proof: Because the left singular lines V_1,\dots,V_r of A are pairwise orthogonal, the projections P_1,\dots,P_r satisfy

P_iP_j = \delta_{ij}I_V,

where I_V \in \mathrm{End}(V) is the identity operator. We thus have

UP = \sum\limits_{i,j=1}^r U_iP_i\sigma_jP_j=\sum\limits_{i=1}^r\sigma_iU_iP_i = \sum\limits_{i=1}^r \sigma_iE_i = A.

as claimed. \square

Problem 21.2. Prove that A is a partial isometry if and only if all its singular values are equal to 1.

As the name “left polar decomposition” suggests, every A \in \mathrm{Hom}(V,W) also admits a right polar decomposition. This is obtained by observing that each of the rank one partial isometries E_i \in \mathrm{Hom}(V,W) in the upstairs SVD of A can be written as

E_i=Q_iU_iP_i,

where Q_i \in \mathrm{End}(W) is the orthogonal projection of W onto the right singular line W_i.

Proposition 21.3. (Right polar decomposition) We have

A=QU,

where U \in \mathrm{Hom}(V,W) is as in Proposition 21.2 and Q \in \mathrm{End}(W) is the operator

Q=\sigma_1Q_1 + \dots + \sigma_rQ_r.

Proof: Because the right singular lines W_1,\dots,W_r of A are pairwise orthogonal, we have

Q_iU_jP_j = \delta_{ij} U_iP_i.

Thus,

QU = \sum\limits_{I,j=1}^r\sigma_iQ_iU_jP_j=\sum\limits_{i=1}^r \sigma_iU_iP_j = A,

as claimed. \square

Now we come to the geometric definition of the adjoint, which relies on uniqueness of the SVD.

Definition 21.4. The adjoint of A \in \mathrm{Hom}(V,W) is the morphism A^* \in \mathrm{Hom}(W,V) defined by the conditions:

  1. The left singular lines of A^* are the right singular lines of A;
  2. The right singular lines of A^* are the left singular lines of A;
  3. The singular values of A^* coincide with the singular values of A.

As automatic consequences of the definition, we have that

\mathrm{rank}(A^*)=\mathrm{rank}(A) \quad\text{and}\quad \mathrm{Ker}(A^*)=\mathrm{Im}(A)^\perp.

Equivalently, Definition 21.4 says the following: given the “upstairs” SVD of A \in \mathrm{Hom}(V,W),

A=\sigma_1U_1P_1+\dots+\sigma_rU_rP_r,

we define A^* \in \mathrm{Hom}(W,V) to be the morphism whose upstairs SVD is

A^* = \sigma_1U_1^{-1}Q_1+\dots+\sigma_rU_r^{-1}Q_r.

From here, we see that the left polar decomposition of A^* \in \mathrm{Hom}(W,V) is

A^*=U^*Q,

where U^* \in \mathrm{Hom}(W,V) is the morphism

U^*= U_1^{-1}Q_1 + \dots + U_r^{-1}Q_r

and Q \in \mathrm{End}(W) is the morphism

Q=\sigma_1Q_1+\dots+\sigma_rQ_r.

The right singular decomposition of A^* \in \mathrm{Hom}(V,W) is

A^*=PU^*,

where as P =\sigma_1P_1+\dots+\sigma_rP_r \in \mathrm{End}(V) is once again the SV-weighted sum of orthogonal projections onto the left singular lines of A, which are the right singular lines of A^*.

Problem 21.3. Show that the operators A^*A \in \mathrm{End}(V) and AA^* \in \mathrm{End}(W) are given by weighted projection sums

A^*A=\sum\limits_{i=1}^r \sigma_i^2P_i \quad\text{and}\quad AA^*=\sum\limits_{i=1}^r \sigma_i^2Q_i.

The rest of the course (and virtually all of Math 202B) is about the Hilbert space \mathrm{End}(V)=\mathrm{Hom}(V,V) of endomorphisms of an underlying Hilbert space V. The reason for this is that \mathrm{End}(V) is an algebra: in addition to scalar multiplication (Frobenius), it carries a notion of vector multiplication (composition), and this additional structure is the core content of Math 202B. But even ignoring this additional structure for the moment and simply thinking of \mathrm{End}(V) as a special case of \mathrm{Hom}(V,W) in which V and W happen to coincide, there are interesting things to be said wrt SVD.

Definition 21.6. An operator A \in \mathrm{End}(V) is said to be normal if its left and right singular lines coincide.

Perhaps you know that this is not the normal definition of normal – if so, forget that corrupt knowledge and accept in its place the geometric truth that is Definition 21.1.

Problem 21.4. Show that every normal operator A \in \mathrm{End}(V) commutes with its adjoint, A^*A=AA^*.

Published by Jonathan Novak

Mathematician at UC San Diego.

Leave a Reply