Mathematical Immaturity

1.11 Inner products, Euclidean spaces. Norms.

In Volume 1, Chapter 12, the dot product of two vectors $x = (x_1, \dots, x_n)$ and $y = (y_1, \dots, y_n)$ was defined by the formula \begin{align*} x \cdot y &= \sum_{i = 1}^n x_iy_i \end{align*} The generalization of the dot product for a general linear space is called the inner product and is written as $(x, y)$ instead of $x \cdot y.$ Instead of being defined formulaically, it is defined axiomatically. That is, there are a number of properties that the inner product must satisfy. \begin{align*} \end{align*} Definition. $\quad$ A real linear space $V$ is said to have an inner product if for each pair of elements $x$ and $y$ in $V$ there corresponds a unique real number $(x, y)$ satisfying the following axioms for all choices of $x,$ $y,$ and $z$ in $V$ and all real scalars $c.$
$\quad\text{(1)}\quad$ $(x, y) = (y, x)\qquad$ (commutativity, or symmetry).
$\quad\text{(2)}\quad$ $(x, y + z) = (x, y) + (x, z)\qquad$ (distributivity, or linearity).
$\quad\text{(3)}\quad$ $c(x, y) = (cx, y)\qquad$ (associativity, or homogeneity).
$\quad\text{(4)}\quad$ $(x, x) > 0\quad$ if $x \neq O \qquad$ (positivity).

A real linear space with an inner product is called a real Euclidean space. In a complex linear space, an inner product $(x, y)$ is a complex number satisfying the same axioms as those for a real inner product, except that the symmetry axiom is replaced by the relation \begin{align*} (1') \qquad (x, y) &= \overline{(y, x)} \end{align*} where $\overline{(y, x)}$ denotes the complex conjugate of $(x, y).$ In the homogeneity relation, $c$ can be any complex number and we get the accompanying relation \begin{align*} (x, cy) &= \overline{(cy, x)} = \bar{c}\overline{(y, x)} = \bar{c}(x, y) \end{align*} A complex linear space with an inner product is called a complex Euclidean space.

Theorem 1.8.$\quad$ In a Euclidean space $V,$ every inner product satisfies the Cauchy-Schwarz inequality \begin{align*} \left|(x, y)\right|^2 \leq (x, x)(y, y) \end{align*} for all $x$ and $y$ in $V.$ Moreover, the equality sign holds if and only if $x$ and $y$ are dependent.

Proof.$\quad$ If either $x$ or $y$ is zero, then the theorem holds trivially. Suppose instead that $x$ and $y$ are both nonzero, and let $z = ax + by$ be an element of $V$ with $a$ and $b$ being complex scalars. Then, $(z, z) > 0$ if $ax \neq -by.$ By the linearity property, we have $(z, z) = (z, ax) + (z, by)$ which gives us $(z, ax) + (z, by) \gt 0.$ Applying the modified commutativity property and the linearity property to the inner products on the left-hand side, we find that \begin{align*} (z, ax) &= \bar{a}(z, x) \\ &= \bar{a}\overline{(x, z)} \\ &= \bar{a}\overline{(x, ax)} + \bar{a}\overline{(x, by)} \\ &= \bar{a}a(x, x) + \bar{a}b(y, x) \\ \\ (z, by) &= \bar{b}(z, y) \\ &= \bar{b}\overline{(y, z)} \\ &= \bar{b}\overline{(y, ax)} + \bar{b}\overline{(y, by)} \\ &= \bar{b}a(x, y) + \bar{b}b(y, y) \end{align*} As such, the inequality $(z, ax) + (z, by) > 0$ becomes \begin{align*} \bar{a}a(x, x) + 2a\bar{b}(x, y) + \bar{b}b(y, y)\gt 0 \end{align*} If we set $a = (y, y),$ the above inequality becomes \begin{align*} \bar{a}(y, y)(x, x) + 2\bar{b}(y, y)(x, y) + \bar{b}b(y, y)\gt 0 \end{align*} Cancelling the positive term $(y, y),$ we get \begin{align*} (x, x)\overline{(y, y)} + 2\bar{b}(x, y) + 2\bar{b}b > 0 \end{align*} Now, if we set $b = -(x, y),$ we have $\bar{b} = -\overline{(x, y)}$ and the relation becomes \begin{align*} (x, x)\overline{(y, y)} > |(x, y)|^2 \end{align*} And by $(1'),$ we know that $\overline{(y, y)} = (y, y)$ and hence, $(x, x)(y, y) > |(x, y)|^2.$ \begin{align*} \end{align*} If we return to the initial condition where we required $ax \neq -by$ and allow $ax = -by,$ we find that $z = O$ and hence the inner product $(z, z) = 0,$ from which we can see that equality is satisfied if $x$ and $y$ are dependent. Now, let $(x, x)(y, y) = |(x, y)|^2.$ If $(x, y) = 0$ then we can express $(x, y)$ as $0(x, y) = (0x, y)$ or equivalently as $0\overline{(y, x)} = (x, 0y).$ As such, $x$ or $y$ must be the zero vector which makes the set $\{x, y\}$ trivially dependent. As such, suppose $x$ and $y$ are nonzero, we will show that equality implies that there exists some complex $c \neq 0$ such that $x = cy.$ Note that $(x, x)$ is a scalar. If we set $x' = (x, x),$ we can write $(x, x)(y, y) = (x'y, y),$ and if we set $q = \overline{(x, y)},$ we have \begin{align*} |(x, y)|^2 &= \overline{(x, y)}(x, y) \\ &= (qx, y) \end{align*} Plugging these values back into our previous equation, we find that \begin{align*} (x'y, y) &= (qx, y) \end{align*} But because $q \neq 0,$ this implies that there exists a complex $c = x'/q$ such that $x = cy,$ which means that $x$ and $y$ must be dependent. As such, we have shown that equality holds if and only if $x$ and $y$ are dependent. This completes the proof for a complex linear space $V.\quad\blacksquare$

Definition.$\quad$ In a Euclidean space $V,$ the nonnegative number $\|x\|$ defined by the equation $$\|x\| = (x, x)^{1/2}$$is called the norm of the element $x.$ We can then express the Cauchy-Schwarz inequality in terms of norms: $$|(x, y)| \leq \|x\|\,\|y\|.$$

Theorem 1.9.$\quad$ In a Euclidean space, every norm has the following properties for all $x$ and $y$ and all scalars $c:$
$\quad(a)\quad \|x\| = 0\quad$ if $x = O.$
$\quad(b)\quad \|x\| \gt 0\quad$ if $x \neq O.\qquad$ (positivity).
$\quad(c)\quad \|cx\| = |c|\,\|x\|\qquad$ (homogeneity).
$\quad(d)\quad \|x + y\| \leq \|x\|+\|y\|\qquad$ (triangle inequality).
The equality sign holds in $(d)$ if $x = O,$ $y = O,$ or if $y = cx$ for some $c > 0.$

Proof.$\quad$
$(a)\quad$ If $x = O,$ then we can use the associativity of an inner product to rewrite $x = 0x$ and $\|x\| = [0(x, x)]^{1/2} = 0. \quad \blacksquare$ \begin{align*} \end{align*} $(b)\quad$ This follows from the positivity of an inner product. If $x \neq 0,$ then $(x, x) > 0$ and hence $(x, x)^{1/2} > 0. \quad \blacksquare$ \begin{align*} \end{align*} $(c)\quad$ Applying associativity and commutativity, we find that \begin{align*} \|cx\| &= [(cx, cx)]^{1/2} \\ &= [c(x, cx)]^{1/2} \\ &= [c(cx, x)]^{1/2} \\ &= [c^2(x, x)]^{1/2} \\ &= |c|\,\|x\| \quad \blacksquare \end{align*} \begin{align*} \end{align*} $(d)\quad$ From the linearity property of the inner product, we know that \begin{align*} \\ \|x + y\|^2 &= (x + y, x + y) \\ &= (x + y, x) + (x + y, y) \\ &= (x, x) + 2(x, y) + (y, y) \\ &= \|x\|^2 + 2|(x, y)|^2 + \|y\|^2 \end{align*} Then, taking the square of $\|x\| + \|y\|,$ we get \begin{align*} \\ \left[\|x\| + \|y\|\right]^2 &= [(x, x)^{1/2} + (y, y)^{1/2}]^2 \\ &= (x, x) + 2[(x, x)(y, y)]^{1/2} + (y, y) \\ &= \|x\|^2 + 2\|x\|^2\|y\|^2 + \|y\|^2 \end{align*} But as we proved in Theorem 1.8, the inner product satisfies the Cauchy-Schwarz inequality, and when expressed in terms of norms, this means that $|(x, y)| \leq \|x\|\,\|y\|$ or equivalently, \begin{align*} \\ |(x, y)|^2\leq \|x\|^2\|y\|^2 \end{align*} When applied to the above equations for $\|x + y\|$ and $\|x\| + \|y\|,$ we can see that \begin{align*} \|x + y\| \leq \|x\|+\|y\|. \end{align*} If $x = O$ or $y = O,$ the equality sign holds trivially since $x + y = x$ or $y.$ If $y = cx$ for some scalar $c > 0,$ then $x + y = x + cx = (1 + c)x$ and we get $\|x + y\| = |1 + c|\|x\|.$ But since $c > 0,$ $|1 + c| = 1 + c,$ giving us \begin{align*} \|x + y\| &= (1 + c)\|x\| \\ &= \|x\| + c\|x\| \\ &= \|x\| + \|cx\| \\ &= \|x\| + \|y\| \quad \blacksquare \end{align*}

Definition.$\quad$ In a real Euclidean space $V,$ the angle between two nonzero elements $x$ and $y$ is defined to be that number $\theta$ in the interval $0 \leq \theta \leq \pi$ which satisfies the equation \begin{align*} (1.7) \qquad \cos \theta &= \frac{(x, y)}{\|x\|\,\|y\|} \end{align*} Note:$\quad$ The Cauchy-Schwarz inequality shows that the quotient on the right-hand side of $(1.6)$ lies in the interval $[-1, 1]$ so there is exactly one $\theta$ in $[0, \pi]$ whose cosine is equal to this quotient.