Vector and Matrix Norms - Executable Engineering: Interactive Numerical Methods in Python

import numpy as np
import plotly.graph_objects as go

try:
    import executable_engineering as exe
except ImportError:
    %pip install -q executable_engineering
    import executable_engineering as exe

Vector Norms¶

A norm is a mathematical way to assign a strictly positive “length” or “magnitude” to a vector.

You are already familiar with the standard Euclidean magnitude (the $L^2$ norm), which is the physical length of an arrow in space. But there are actually many different ways to measure length!

The general definition for the $L^p$ norm of a vector $\mathbf{v}$ is:

\|\mathbf{v}\|_p = \left( \sum_{i} |v_i|^p \right)^{1/p}

(1)

Common Vector Norms¶

Here are the three most common vector norms used in numerical methods:

The 1-Norm ( $L^1$ ) - “Manhattan Distance” Simply the sum of the absolute values. It is the distance if you were only allowed to travel on a grid (like city blocks).
$\|\mathbf{v}\|_1 = \sum |v_i|$
(2)
The 2-Norm ( $L^2$ ) - “Euclidean Distance” The standard “as the crow flies” physical length.
$\|\mathbf{v}\|_2 = \sqrt{\sum v_i^2}$
(3)
The Infinity Norm ( $L^\infty$ ) - “Maximum Distance” As $p \rightarrow \infty$ , the largest element completely dominates the sum. It is simply the absolute value of the largest element in the vector.
$\|\mathbf{v}\|_\infty = \max |v_i|$
(4)

Matrix Norms¶

Just as we can measure the “length” of a vector, we can measure the “size” or “stretching power” of a matrix $\mathbf{A}$ .

An induced matrix norm is defined as the maximum possible amount that the matrix can stretch any vector $\mathbf{x}$ :

\|\mathbf{A}\| = \max_{\mathbf{x} \neq 0} \frac{\|\mathbf{A}\mathbf{x}\|}{\|\mathbf{x}\|}

(5)

Because this requires a complex maximization search, we often use easier-to-calculate approximations.

Common Matrix Norms¶

The 2-Norm (Spectral Norm) The exact maximum stretching power (the largest singular value of the matrix). It is computationally expensive to calculate.
$\|\mathbf{A}\|_2 = \sigma_{\max}(\mathbf{A})$
(6)
The Frobenius Norm Calculated exactly like a Euclidean vector norm, by summing the square of every element in the matrix. It is very fast to compute and acts as a safe upper bound on the 2-Norm ( $\mathbf{\|A\|_2 \le \|A\|_F}$ ).
$\|\mathbf{A}\|_F = \sqrt{\sum_{i} \sum_{j} |A_{ij}|^2}$
(7)
The Infinity Norm (Max Row Sum) The maximum absolute sum of any single row. Also very fast to compute and bounds the stretching power.
$\|\mathbf{A}\|_\infty = \max_{i} \sum_{j} |A_{ij}|$
(8)

Visualizing Matrix Norms¶

If we apply a matrix $\mathbf{A}$ to the unit circle, the circle stretches into an ellipse.

The furthest point on that ellipse from the origin is precisely the 2-Norm.
The Frobenius and Infinity norms are fast approximations that cast a “net” around the ellipse, guaranteeing that the true distortion never exceeds their radius!

Let’s plot the distorted ellipse and overlay circles with radii equal to these different norms.

A = [[2, 1], 
     [-1, 3]]

fig = exe.visualize_matrix_norms(A)
fig.show()