Truncation Error

import numpy as np
import math

import plotly.graph_objects as go

Truncation error occurs when we approximate a mathematical function by truncating an infinite series.

Taylor Series¶

The Taylor series represents a function as an infinite sum of its derivatives at a point.

\begin{align} f(x+\Delta x) &= f(x) + f'(x) \Delta x + f''(x) \frac{\Delta x^2}{2!} + f'''(x) \frac{\Delta x^3}{3!} + \cdots \\ &= \sum_{n=0}^{\infty} \frac{f^{(n)}(x)}{n!} \Delta x^n \end{align}

(1)

But computers cannot compute an infinite number of terms, or even a large number efficiently.

Some Taylor series expansions diverge outside of a radius of convergence which obviously limits their application. Other series converge globally (for all $x$ ) but only after infinite terms!

Truncating the Series¶

When we truncate the series, we create an approximation:

f(x+\Delta x) \approx f(x) + f'(x) \Delta x + f''(x) \frac{\Delta x^2}{2!} + E_k

(2)

Truncation Error ( $E_k$ ): The sum of all neglected terms:

E_k = \sum_{n=k}^{\infty} \frac{f^{(n)}(x)}{n!} \Delta x^n

(3)

Big-O Notation¶

Truncation error is often expressed using Big-O notation which describes the approximate size of a set of terms. Assuming that the term is the largest, we ignore all the rest and consider only $\mathcal{O}(\Delta x^k)$ .

\begin{aligned} f(x+\Delta x) &= f(x) + f'(x) \Delta x + f''(x) \frac{\Delta x^2}{2!} + f'''(x) \frac{\Delta x^3}{3!} + \cdots \\ &= f(x_0) + f'(x_0) \Delta x + f''(x_0) \frac{\Delta x^2}{2} + \mathcal{O}(\Delta x^3) \end{aligned}

(4)

Why? It tells us how the error scales when we change the step size $\Delta x$ .

Example: Forward Difference¶

Approximating the first derivative:

\tilde{f}'(x) \approx \frac{f(x+\Delta x) - f(x)}{\Delta x}

(5)

Substitute Taylor series for $f(x+\Delta x)$ :

\begin{aligned} \tilde{f}'(x) &\approx \frac{(f(x) + f'(x) \Delta x + f''(x) \frac{\Delta x^2}{2} + \cdots) - f(x)}{\Delta x} \\ &= f'(x) + f''(x) \frac{\Delta x}{2} + \cdots \\ &= f'(x) + \mathcal{O}(\Delta x) \end{aligned}

(6)

By using the Taylor series, we have therefore shown that the forward difference is a first-order / linear approximation. Reducing the step size shrinks the error by approximately the same factor.

Example: Central Difference¶

A more accurate approximation is called the central difference:

\tilde{f}'(x) \approx \frac{f(x+\Delta x) - f(x-\Delta x)}{2 \Delta x}

(7)

Substitute Taylor series for both terms:

\begin{align} \tilde{f}'(x) &\approx f'(x) + f'''(x) \frac{\Delta x^2}{6} + \cdots \\ &\approx f'(x) + \mathcal{O}(\Delta x^2) \end{align}

(8)

Therefore, the central difference is a second-order accurate approximation. Reducing the step size improves the answer by the square of the factor!

Comparing Error Behaviour: Forward vs Central¶

Let’s compare the error behavior of both methods as we decrease $\Delta x$ . We will calculate the true absolute error ( $E_t$ ) for the derivative of $2\sin(x)$ at $x=0.1$ (to avoid a special case for odd functions at $sin(x=0)$ )

Forward Difference: Error should scale linearly with $\Delta x$ (Slope = 1 on log-log plot).
Central Difference: Error should scale quadratically with $\Delta x$ (Slope = 2 on log-log plot).

Does this behave as you would expect? What is happening at small $\Delta x$ ? (hint: what operation does ‘difference’ imply?)

x = 0.1
true_val = 2 * np.cos(x)
delta_x = np.logspace(0, -10, 11)

def forward_diff(delta_x):
    return (2*np.sin(x + delta_x) - 2*np.sin(x)) / delta_x

def central_diff(delta_x):
    return (2*np.sin(x + delta_x) - 2*np.sin(x - delta_x)) / (2 * delta_x)

err_forward = np.abs(true_val - forward_diff(delta_x))
err_central = np.abs(true_val - central_diff(delta_x))

fig = go.Figure()
fig.add_trace(go.Scatter(x=delta_x, y=err_forward, mode='lines+markers', name='Forward Diff (O(Δx))'))
fig.add_trace(go.Scatter(x=delta_x, y=err_central, mode='lines+markers', name='Central Diff (O(Δx²))'))

fig.update_xaxes(type='log', title='Step Size (Δx)')
fig.update_yaxes(type='log', title='True Absolute Error (E_t)')
fig.update_layout(title='Truncation Error: Forward vs Central Difference at x=1')
fig.show()

Loading...

Common Mathematical Functions¶

Computers are excellent at basic arithmetic, but how do they calculate more complex functions? Often, they use Taylor series expansions.

Function	Taylor Expansion
$\sin(x)$	$x - \frac{x^3}{3!} + \frac{x^5}{5!} - \frac{x^7}{7!} + \cdots$
$\cos(x)$	$1 - \frac{x^2}{2!} + \frac{x^4}{4!} - \frac{x^6}{6!} + \cdots$
$\exp(x)$	$1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \frac{x^4}{4!} + \cdots$
$\ln(1+x)$	$x - \frac{x^2}{2} + \frac{x^3}{3} - \frac{x^4}{4} + \cdots$

In infinite precision, these series are globally convergent. However, in finite precision, they can lead to issues.

Calculating $\sin(x)$ via Taylor Series¶

def taylor_sin(x, n_terms):
    approx = 0
    for n in range(n_terms):
        term = ((-1)**n * x**(2*n + 1)) / math.factorial(2*n + 1)
        approx += term
        print(f"Term {n+1:2}: {term: .2e} | Sum: {approx: .10f}")
    
    print(f"\nActual sin({x}): {math.sin(x):.10f}")

Small $x$ : Rapid Convergence¶

For small $x$ , the terms decrease rapidly.

taylor_sin(1, 10)

Term  1:  1.00e+00 | Sum:  1.0000000000
Term  2: -1.67e-01 | Sum:  0.8333333333
Term  3:  8.33e-03 | Sum:  0.8416666667
Term  4: -1.98e-04 | Sum:  0.8414682540
Term  5:  2.76e-06 | Sum:  0.8414710097
Term  6: -2.51e-08 | Sum:  0.8414709846
Term  7:  1.61e-10 | Sum:  0.8414709848
Term  8: -7.65e-13 | Sum:  0.8414709848
Term  9:  2.81e-15 | Sum:  0.8414709848
Term 10: -8.22e-18 | Sum:  0.8414709848

Actual sin(1): 0.8414709848

Medium $x$ :¶

For medium $x$ , terms grow large before shrinking and converging.

taylor_sin(10, 10)  # The true value is between -1 and 1!

Term  1:  1.00e+01 | Sum:  10.0000000000
Term  2: -1.67e+02 | Sum: -156.6666666667
Term  3:  8.33e+02 | Sum:  676.6666666667
Term  4: -1.98e+03 | Sum: -1307.4603174603
Term  5:  2.76e+03 | Sum:  1448.2716049383
Term  6: -2.51e+03 | Sum: -1056.9392336059
Term  7:  1.61e+03 | Sum:  548.9651500763
Term  8: -7.65e+02 | Sum: -215.7512231057
Term  9:  2.81e+02 | Sum:  65.3945023288
Term 10: -8.22e+01 | Sum: -16.8118501374

Actual sin(10): -0.5440211109

Large $x$ :¶

For large $x$ , terms grow so large roundoff error is going to kick in long before convergence.

taylor_sin(100, 10)  # The true value is between -1 and 1!

Term  1:  1.00e+02 | Sum:  100.0000000000
Term  2: -1.67e+05 | Sum: -166566.6666666667
Term  3:  8.33e+07 | Sum:  83166766.6666666567
Term  4: -1.98e+10 | Sum: -19758103074.6031723022
Term  5:  2.76e+12 | Sum:  2735973819323.9858398438
Term  6: -2.51e+14 | Sum: -247785110035093.1875000000
Term  7:  1.61e+16 | Sum:  15811258726786520.0000000000
Term  8: -7.65e+17 | Sum: -748905114455195136.0000000000
Term  9:  2.81e+19 | Sum:  27365667429000011776.0000000000
Term 10: -8.22e+20 | Sum: -794697857233433001984.0000000000

Actual sin(100): -0.5063656411

Why You Should Use Standard Packages¶

While the remedy for this specific issue is relatively simple (e.g., using range reduction), it highlights the complexities of implementing numerical functions. In practice, the methods used in packages like NumPy are highly sophisticated, often employing a combination of techniques, including different expansion methods and look-up tables, to ensure both performance and stability.

This is why it is almost always better to use a well-tested package rather than implementing these functions yourself.