Introduction
Finding an orthogonal basis is a fundamental skill in linear algebra with profound implications across mathematics, physics, computer science, and engineering. Still, an orthogonal basis for a vector space (or subspace) is a set of vectors that are mutually perpendicular—meaning their dot product is zero—and span the space. This property simplifies complex calculations dramatically, turning difficult projection problems into simple arithmetic and stabilizing numerical algorithms against rounding errors. Practically speaking, whether you are performing QR decomposition, solving least-squares regression problems, analyzing signals via Fourier transforms, or diagonalizing matrices in quantum mechanics, the ability to construct an orthogonal basis from an arbitrary set of vectors is indispensable. This article provides a complete guide to the theory, the step-by-step Gram-Schmidt process, practical examples, and the numerical considerations required to master this essential technique Practical, not theoretical..
Detailed Explanation
What is an Orthogonal Basis?
To understand how to find an orthogonal basis, we must first define the terms precisely. A basis for a vector space $V$ is a set of linearly independent vectors ${ \mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_n }$ that spans $V$. An orthogonal basis adds a geometric constraint: for any two distinct vectors $\mathbf{v}_i$ and $\mathbf{v}_j$ in the set, their inner product (usually the standard dot product in $\mathbb{R}^n$) is zero: $\mathbf{v}_i \cdot \mathbf{v}_j = 0$ for all $i \neq j$. If, additionally, every vector in the basis has a length (norm) of 1, the basis is called an orthonormal basis.
The power of an orthogonal basis lies in coordinate representation. Practically speaking, in a standard non-orthogonal basis, finding the coordinates of a vector $\mathbf{x}$ requires solving a linear system $A\mathbf{c} = \mathbf{x}$. In an orthogonal basis ${\mathbf{u}_1, \dots, \mathbf{u}_n}$, the coordinates (coefficients) $c_i$ are given explicitly by the projection formula: $c_i = \frac{\mathbf{x} \cdot \mathbf{u}_i}{\mathbf{u}_i \cdot \mathbf{u}_i}$ If the basis is orthonormal, this simplifies further to $c_i = \mathbf{x} \cdot \mathbf{u}_i$. This eliminates the need for matrix inversion, offering massive computational savings and numerical stability.
The official docs gloss over this. That's a mistake.
Why Do We Need to Find One?
Rarely does a problem present itself with a ready-made orthogonal basis. In applied fields, data arrives as a set of linearly independent but non-orthogonal vectors—columns of a design matrix in statistics, a set of basis functions in approximation theory, or state vectors in a dynamical system. We must algorithmically transform this "skewed" basis into an orthogonal one while preserving the span (the subspace covered by the vectors). The standard algorithm for this transformation is the Gram-Schmidt process, though modern numerical linear algebra often prefers Householder reflections or Givens rotations for stability, which we will discuss later The details matter here. That alone is useful..
Step-by-Step Concept Breakdown: The Gram-Schmidt Process
The Gram-Schmidt orthogonalization process is the classical, constructive method for converting a basis ${\mathbf{a}_1, \mathbf{a}_2, \dots, \mathbf{a}_k}$ into an orthogonal basis ${\mathbf{q}_1, \mathbf{q}_2, \dots, \mathbf{q}_k}$ for the same subspace. The logic is recursive: we build the new basis one vector at a time, subtracting the "shadow" (projection) of the current vector onto the space already spanned by the previous orthogonal vectors Simple, but easy to overlook..
The Algorithm
Let the input basis be $\mathbf{a}_1, \mathbf{a}_2, \dots, \mathbf{a}_n$. We compute the orthogonal basis $\mathbf{q}_1, \mathbf{q}_2, \dots, \mathbf{q}_n$ as follows:
Step 1: Initialize the first vector. The first orthogonal vector is simply the first input vector. $\mathbf{q}_1 = \mathbf{a}_1$
Step 2: Iterate for $k = 2$ to $n$. For each subsequent vector $\mathbf{a}_k$, subtract its projection onto the subspace spanned by ${\mathbf{q}1, \dots, \mathbf{q}{k-1}}$. $\mathbf{q}k = \mathbf{a}k - \sum{j=1}^{k-1} \text{proj}{\mathbf{q}_j}(\mathbf{a}k)$ Where the projection of $\mathbf{a}$ onto $\mathbf{q}$ is defined as: $\text{proj}{\mathbf{q}}(\mathbf{a}) = \frac{\mathbf{a} \cdot \mathbf{q}}{\mathbf{q} \cdot \mathbf{q}} \mathbf{q}$
Step 3 (Optional): Normalize to obtain an orthonormal basis. If unit length vectors are required (denoted $\mathbf{u}_k$), divide each orthogonal vector by its norm: $\mathbf{u}_k = \frac{\mathbf{q}_k}{|\mathbf{q}_k|}$
Classical vs. Modified Gram-Schmidt
You've got two ways worth knowing here.
- Classical Gram-Schmidt (CGS): Computes all projections using the original $\mathbf{a}_k$ and sums them at the end. Practically speaking, mathematically equivalent but numerically unstable in floating-point arithmetic because errors accumulate when subtracting large, nearly equal vectors. That said, * Modified Gram-Schmidt (MGS): Updates the vector sequentially. Let $\mathbf{v} = \mathbf{a}k$. For $j = 1$ to $k-1$: compute $r{jk} = \mathbf{q}j^T \mathbf{v}$, then update $\mathbf{v} = \mathbf{v} - r{jk}\mathbf{q}_j$. In real terms, finally, set $\mathbf{q}_k = \mathbf{v}$. In practice, this re-orthogonalizes against the updated vector immediately, significantly reducing error propagation. **MGS is the standard implementation in practice.
Real Examples
Example 1: Orthogonalizing a Basis in $\mathbb{R}^3$
Find an orthogonal basis for the subspace spanned by: $\mathbf{a}_1 = \begin{bmatrix} 1 \ 1 \ 0 \end{bmatrix}, \quad \mathbf{a}_2 = \begin{bmatrix} 1 \ 0 \ 1 \end{bmatrix}, \quad \mathbf{a}_3 = \begin{bmatrix} 0 \ 1 \ 1 \end{bmatrix}$
Step 1: $\mathbf{q}_1 = \mathbf{a}_1 = \begin{bmatrix} 1 \ 1 \ 0 \end{bmatrix}$ Simple, but easy to overlook..
Step 2: Compute $\mathbf{q}_2$. Projection of $\mathbf{a}_2$ onto $\mathbf{q}1$: $\text{proj}{\mathbf{q}_1}(\mathbf{a}_2) = \frac{\mathbf{a}_2 \cdot \mathbf{q}_1}{\mathbf{q}_1 \cdot \mathbf{q}_1} \mathbf{q}_1 = \frac{1(1) + 0(1) + 1(0)}{1^2+1^2+0^2} \begin{bmatrix} 1 \ 1 \ 0 \end{bmatrix} = \frac{1}{2} \begin{bmatrix} 1 \ 1 \ 0 \end{bmatrix} = \begin{bmatrix} 0.5 \ 0.5 \ 0 \end{bmatrix}$ Subtract: $\mathbf{q}_2 = \mathbf{a}2 - \text{proj}{\mathbf{q}_1}(\mathbf{a}_2) = \begin{bmatrix} 1 \ 0 \ 1 \end{bmatrix} - \begin{bmatrix} 0.5 \ 0.5 \ 0 \end{bmatrix} = \begin{bmatrix} 0.5 \ -0.5 \ 1 \end{bmatrix}$ (We can multiply by 2 to clear fractions: $\mathbf{q}_2 = \begin{bmatrix} 1 \ -1 \ 2 \end{bmatrix}$).
Step 2 (continued):
Compute $\mathbf{q}_3$.
First, project $\mathbf{a}_3$ onto $\mathbf{q}1$:
$\text{proj}{\mathbf{q}_1}(\mathbf{a}_3) = \frac{\mathbf{a}_3 \cdot \mathbf{q}_1}{\mathbf{q}_1 \cdot \mathbf{q}_1} \mathbf{q}_1 = \frac{0(1) + 1(1) + 1(0)}{1^2 + 1^2 + 0^2} \begin{bmatrix} 1 \ 1 \ 0 \end{bmatrix} = \frac{1}{2} \begin{bmatrix} 1 \ 1 \ 0 \end{bmatrix} = \begin{bmatrix} 0.5 \ 0.5 \ 0 \end{bmatrix}.$
Next, project $\mathbf{a}_3$ onto $\mathbf{q}2$:
$\text{proj}{\mathbf{q}_2}(\mathbf{a}_3) = \frac{\mathbf{a}_3 \cdot \mathbf{q}_2}{\mathbf{q}_2 \cdot \mathbf{q}_2} \mathbf{q}_2.$
Using $\mathbf{q}_2 = \begin{bmatrix} 1 \ -1 \ 2 \end{bmatrix}$, compute:
$\mathbf{a}_3 \cdot \mathbf{q}_2 = 0(1) + 1(-1) + 1(2) = 1,$
$\mathbf{q}_2 \cdot \mathbf{q}2 = 1^2 + (-1)^2 + 2^2 = 6.$
Thus,
$\text{proj}{\mathbf{q}_2}(\mathbf{a}_3) = \frac{1}{6} \begin{bmatrix} 1 \ -1 \ 2 \end{bmatrix} = \begin{bmatrix} 1/6 \ -1/6 \ 1/3 \end{bmatrix}.$
Subtract both projections:
$\mathbf{q}_3 = \mathbf{a}3 - \text{proj}{\mathbf{q}_1}(\mathbf{a}3) - \text{proj}{\mathbf{q}_2}(\mathbf{a}_3) = \begin{bmatrix} 0 \ 1 \ 1 \end{bmatrix} - \begin{bmatrix} 0.5 \ 0.5 \ 0 \end{bmatrix} - \begin{bmatrix} 1/6 \ -1/6 \ 1/3 \end{bmatrix} = \begin{bmatrix} -2/3 \ 2/3 \ 2/3 \end{bmatrix}.$
To simplify, multiply by 3:
$\mathbf{q}_3 = \begin{bmatrix} -2 \ 2 \ 2 \end{bmatrix}.$
Step 3 (Optional): Normalize the vectors.
- $|\mathbf{q}_1| = \sqrt{1^2 + 1^2 + 0^2} = \sqrt{2} \implies \mathbf{u}_1 = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 \ 1 \ 0 \end{bmatrix}$.
- $|\mathbf{q}_2| = \sqrt{1^2 + (-1)^2 + 2^2} = \sqrt{6} \implies \mathbf{u}_2 = \frac{1}{\sqrt{6}} \begin{bmatrix} 1 \ -1 \ 2 \end{bmatrix}$.
- $|\mathbf{q}_3| = \sqrt{(-2)^2 + 2^2 + 2^2} = \sqrt{12} = 2\sqrt{3} \implies \mathbf{u}_3 = \frac{1}{2\sqrt{3}} \begin{bmatrix} -2 \ 2 \ 2 \end{bmatrix} = \begin{bmatrix} -1/\sqrt{3} \ 1/\sqrt{3} \ 1/\sqrt{3} \end{bmatrix}$.
Conclusion
The Gram-Schmidt process effectively constructs an orthogonal (or orthonormal) basis for a subspace by iteratively subtracting projections of subsequent vectors onto their predecessors. While Classical Gram-Schmidt (CGS) is mathematically equivalent, Modified Gram-Schmidt (MGS) is preferred in practice due to its numerical stability. In the example, the orthogonal basis ${\mathbf{q}_1, \mathbf{q}_2, \mathbf{q}_3}$ spans the same subspace as ${\mathbf{a}_1, \mathbf{a}_2, \mathbf{a}_3}$, and normalization yields an orthonormal basis. This methodology is foundational in applications like QR decomposition, computer graphics, and signal processing, where orthogonal representations simplify computations and enhance stability.
Final Answer
The orthogonal basis is $\left{\begin{bmatrix}1\1\0\end{bmatrix}, \begin{bmatrix}1\-1\2\end{bmatrix}, \begin{bmatrix}-2\2\2\end{bmatrix}\right}$, and the orthonormal basis is $\left{\frac{1}{\sqrt{2}}\begin{bmatrix}1\1\0\end{bmatrix}, \frac{1}{\sqrt{6}}\begin{bmatrix}1\-1\2\end{bmatrix}, \begin{bmatrix}-\frac{1}{\sqrt{3}}\frac{1}{\sqrt{3}}\frac{1}{\sqrt{3}}\end{bmatrix}\right}$.
\boxed{\text{Orthonormal basis: } \left{\frac{1}{\sqrt{2}}\begin{bmatrix}1\1\0\end{bmatrix}, \frac{1}{\sqrt{6}}\begin{bmatrix}1\-1\2\end{bmatrix}, \begin{bmatrix}-\
The Gram-Schmidt process constructs an orthogonal or orthonormal basis for a subspace by iteratively orthogonalizing vectors against prior ones. Now, in the example, starting with $\mathbf{a}_1 = \begin{bmatrix}1\1\0\end{bmatrix}$, $\mathbf{a}_2 = \begin{bmatrix}1\-1\2\end{bmatrix}$, and $\mathbf{a}_3 = \begin{bmatrix}0\1\1\end{bmatrix}$, the orthogonal basis is derived as ${\mathbf{q}_1, \mathbf{q}_2, \mathbf{q}_3}$, where $\mathbf{q}_1 = \mathbf{a}_1$, $\mathbf{q}_2 = \mathbf{a}2 - \text{proj}{\mathbf{q}_1}(\mathbf{a}_2)$, and $\mathbf{q}_3 = \mathbf{a}3 - \text{proj}{\mathbf{q}_1}(\mathbf{a}3) - \text{proj}{\mathbf{q}_2}(\mathbf{a}_3)$. Normalizing these vectors yields the orthonormal basis ${\mathbf{u}_1, \mathbf{u}_2, \mathbf{u}_3}$ Worth keeping that in mind..
For the given vectors:
- $\mathbf{q}_1 = \begin{bmatrix}1\1\0\end{bmatrix}$, normalized to $\mathbf{u}_1 = \frac{1}{\sqrt{2}}\begin{bmatrix}1\1\0\end{bmatrix}$.
- $\mathbf{q}_2 = \begin{bmatrix}1\-1\2\end{bmatrix}$, normalized to $\mathbf{u}_2 = \frac{1}{\sqrt{6}}\begin{bmatrix}1\-1\2\end{bmatrix}$.
- $\mathbf{q}_3 = \begin{bmatrix}-2\2\2\end{bmatrix}$, normalized to $\mathbf{u}_3 = \frac{1}{2\sqrt{3}}\begin{bmatrix}-2\2\2\end{bmatrix} = \begin{bmatrix}-1/\sqrt{3}\1/\sqrt{3}\1/\sqrt{3}\end{bmatrix}$.
The orthonormal basis is: $ \boxed{\left{\frac{1}{\sqrt{2}}\begin{bmatrix}1\1\0\end{bmatrix}, \frac{1}{\sqrt{6}}\begin{bmatrix}1\-1\2\end{bmatrix}, \begin{bmatrix}-\frac{1}{\sqrt{3}}\\frac{1}{\sqrt{3}}\\frac{1}{\sqrt{3}}\end{bmatrix}\right}} $
Conclusion: The Gram-Schmidt process is foundational for generating orthogonal or orthonormal bases, enabling stable computations in applications like QR decomposition, computer graphics, and signal processing. The derived orthonormal basis ensures numerical stability and simplifies operations in these domains Not complicated — just consistent..
