3.17 Taylor's Theorem (Optional)

Table of Contents

Review of Taylor’s Formula for Functions of a Single Variable

Let’s review the Taylor series for single variable functions. Suppose $y = f (x)$ is differentiable at $x_{0}$ , then it has a linear approximation at $x_{0}$ , and we have:
$f (x) = f (x_{0}) + f^{'} (x_{0}) (x - x_{0}) + (x - x_{0}) ε_{1} (x - x_{0}),$ with $lim_{x \to x_{0}} ε_{1} (x - x_{0}) = 0.$ Therefore, $P_{1, x_{0}} (x) = f (x_{0}) + f^{'} (x_{0}) (x - x_{0})$ is the linear approximation and the error in this approximation is: $R_{1, x_{0}} (x) = (x - x_{0}) ε_{1} (x - x_{0}) .$ The subscripts $1$ and $x_{0}$ in $P_{1, x_{0}}$ and $R_{1, x_{0}}$ show the maximum power of $x$ that appears in the polynomial and the point about which we approximate the function $f$ .

Note that as $x \to x_{0}$ , $R_{1, x_{0}} (x)$ tends to 0 faster than $(x - x_{0})$ because $R_{1, x_{0}} (x) / (x - x_{0}) = ε_{1, x_{0}} (x - x_{0}) \to 0.$ Mathematically we write $R_{1, x_{0}} (x) = o (x - x_{0}) .$

If we want a better approximation of $f$ , instead of a linear function, we may use a quadratic function $P_{2, x_{0}} (x) = a_{0} + a_{1} x + a_{2} x^{2} .$ To determine the coefficients $a_{0}, a_{1}$ and $a_{2}$ , we match the values of the functions and their first and second derivatives at $x = x_{0}$ . This means the graph of $P_{2, x_{0}} (x)$ has the same value, the same slope and the same concavity as the graph of $f$ at $x = x_{0}$ . The quadratic polynomial reads:
$P_{2, x_{0}} (x) = f (x_{0}) + f^{'} (x_{0}) (x - x_{0}) + \frac{1}{2!} f ” (x_{0}) (x - x_{0})^{2},$
and the error is
$R_{2, x_{0}} (x) = (x - x_{0})^{2} ε_{2, x_{0}} (x - x_{0}), where lim_{x \to x_{0}} ε_{2, x_{0}} (x - x_{0}) = 0.$

We still can improve approximations of $f$ at $x = x_{0}$ by using higher order polynomials and matching more derivatives at the selected base point. If we use a polynomial of order $m$ , we can prove (see the following theorem) that the error in the approximation goes to zero faster than $(x - x_{0})^{m}$ as $x \to x_{0}$ . Mathematically we write $R_{m, x_{0}} (x) = o ((x - x_{0})^{m})$ which means $R_{m, x_{0}} (x) / (x - x_{0})^{m} \to 0$ as $x \to x_{0}$ . In general, we have the following theorem:

Theorem 1. Suppose

f

is a function that is

m (\geq 1)

times differentiable at

x_{0}

; that is

f^{'} (x_{0}), f^{''} (x_{0}), \dots, f^{(m)} (x_{0})

all exist. Let

P_{m, x_{0}} (x) = a_{0} + a_{1} (x - x_{0}) + \dots + a_{m} (x - x_{0})^{m},

where

a_{k} = \frac{f^{(k)} (x_{0})}{k!}, 0 \leq k \leq m,

and

ε_{m, x_{0}} (x) = \frac{f (x) - P_{m, x_{0}} (x)}{(x - x_{0})^{m}} .

Then

lim_{x \to x_{0}} ε_{m, x_{0}} (x) = 0

Remark that $f^{(0)} (x_{0}) = f (x_{0})$ , $f^{(1)} (x_{0}) = f^{'} (x_{0})$ , $f^{(2)} (x_{0}) = f^{''} (x_{0})$ , $\dots$ , and $0! = 1$ .

Show the proof

Proof: Notice that
$\begin{aligned} lim_{x \to x_{0}} ε_{m, x_{0}} (x) & = lim_{x \to x_{0}} \frac{f (x) - P_{m, x_{0}} (x)}{(x - x_{0})^{m}} \\ = lim_{x \to x_{0}} \frac{f (x) - f (x_{0}) - \frac{f^{'} (x_{0})}{1!} (x - x_{0}) - \dots \frac{f^{(m)} (x_{0})}{m!} (x - x_{0})^{m}}{(x - x_{0})^{m}} = \frac{0}{0} . \end{aligned}$
If we apply l’Hôpital’s rule $m$ times, we obtain:
$lim_{x \to x_{0}} \frac{f^{(m)} (x) - m! \frac{f^{(m)} (x_{0})}{m!}}{m!} = lim_{x \to x_{0}} \frac{f^{(m)} (x) - f^{(m)} (x_{0})}{m!} = 0.$

Definitions of the “Taylor polynomial” and “remainder”

$P_{m, x_{0}} (x)$ is called the Taylor polynomial of degree $m$ for $f (x)$ at $x_{0}$ . The error $R_{m, x_{0}} (x)$ is also called the remainder term. You should verify that
$\begin{aligned} P_{m, x_{0}} (x_{0}) & = f (x_{0}), \\ P_{m, x_{0}}^{'} (x_{0}) & = f^{'} (x_{0}), \\ P_{m, x_{0}}^{''} (x_{0}) & = f^{''} (x_{0}), \\ ⋮ \\ P_{m, x_{0}}^{(m)} (x_{0}) & = f^{(m)} (x_{0}) . \end{aligned}$
To estimate the error of this approximation, we would like to have an expression for the remainder $R_{m, x_{0}} (x)$ . Various expressions under stronger regularity assumptions on $f$ exist in the literature. We mention one of them which is called the Lagrange form of the remainder term, after the great mathematician Joseph-Louis Lagrange.

Theorem 2. If

f^{(m + 1)}

is continuous on an open interval

I

that contains

x_{0}

and

x \in I

, then there exists a number

ξ

between

x

and

x_{0}

such that

R_{m, x_{0}} (x) = \frac{f^{(m + 1)} (ξ)}{(m + 1)!} (x - x_{0})^{(m + 1)} .

Show the proof

Proof: For clarity we replace $x$ by $b$ and show that there is some $ξ$ between $x_{0}$ and $b$ such that $R_{m, x_{0}} (b) = \frac{f^{(m + 1)} (ξ)}{(m + 1)!} (b - x_{0})^{m + 1} .$ We choose $K$ such that $\begin{matrix} (*) & f (b) = P_{m, x_{0}} (b) + K (b - x_{0})^{m + 1} \end{matrix}$ and define
$\begin{aligned} g (x) & = f (x) - P_{m, x_{0}} (x) - K (x - x_{0})^{m + 1} \\ = f (x) - P_{m, x_{0}} (x) - \frac{f (b) - P_{m, x_{0}} (b)}{(b - x_{0})^{m + 1}} (x - x_{0})^{m + 1} . \end{aligned}$
Notice that
$g (x_{0}) = g^{'} (x_{0}) = g^{''} (x_{0}) = \dots = g^{(m)} (x_{0}) = 0$ because we constructed $P_{m, x_{0}} (x)$ by matching the first $m$ derivatives of $P_{m, x_{0}} (x)$ and $f (x)$ at $x = x_{0}$ , and the first $n$ derivatives of $K (x - x_{0})^{m + 1}$ are all zero. Also note that we chose $K$ such that $g (b) = 0$ .

According to Rolle’s theorem, there is $ξ_{1}$ between $x_{0}$ and $b$ such that $g^{'} (ξ_{1}) = 0$ .

Again because $g^{'} (x_{0}) = g^{'} (ξ) = 0$ , by Rolle’s theorem there is a number $ξ_{2}$ between $x_{0}$ and $ξ_{1}$ such that $g ” (ξ_{2}) = 0$ . We can repeat this argument, finding a $ξ_{m}$ between $x_{0}$ and $ξ_{m - 1}$ such that $g^{'} (ξ_{m}) = 0$ . If we use this argument again, there is a number $ξ_{m + 1}$ between $x_{0}$ and $ξ_{m}$ such that $g^{'} (ξ_{m + 1}) = 0$ . Let’s evaluate $g^{'} (x)$ : $g^{(m + 1)} (x) = f^{(m + 1)} (x) - (m + 1)! K .$ Thus:
$g^{(m + 1)} (ξ_{m + 1}) = f^{(m + 1)} (ξ_{m + 1}) - (m + 1)! K = 0 \Rightarrow K = \frac{f^{(m + 1)} (ξ_{m + 1})}{(m + 1)!} .$
Let’s put $ξ = ξ_{m + 1}$ . If we use the definition of $K$ in (*), we have:
$R_{m, x_{0}} (b) = f (b) - P_{m, x_{0}} (b) = K (b - x_{0})^{m + 1} = \frac{f^{(m + 1)} (ξ)}{(m + 1)!} (b - x_{0})^{m + 1} .$

Rolle’s theorem says if $f (a) = f (b)$ for $b \neq a$ and $f$ is differentiable between $a$ and $b$ and continuous on $[a, b]$ , then there is at least a number $c$ such that $f^{'} (c) = 0$ . This theorem is very intuitive just by looking at the following figure.

Rolle's Theorem

We don’t know anything about $ξ$ except that $ξ$ is between $x_{0}$ and $x$ .

If we place $x = x_{0} + h$ , we have:

$\begin{aligned} f (x_{0} + h) = & f (x_{0}) + \frac{f^{'} (x_{0})}{1!} h + \frac{f^{''} (x_{0})}{2!} h^{2} + \dots + \frac{f^{(m)} (x_{0})}{m!} h^{m} + \frac{f^{(m + 1)} (x_{0} + θ h)}{(m + 1)!} h^{m + 1} \\ = & \sum_{k = 1}^{m} \frac{f^{(k)} (x_{0})}{k!} h^{k} + \frac{f^{(m + 1)} (x_{0} + θ h)}{(m + 1)!} h^{m + 1} \\ for some 0 < & θ < 1. \end{aligned}$

Taylor’s Formula for Functions of Several Variables

Now we wish to extend the polynomial expansion to functions of several variables. We learned that if $f (x, y)$ is differentiable at $(x_{0}, y_{0})$ , we can approximate it with a linear function (or more accurately an affine function), $P_{1, (x_{0}, y_{0})} (x, y) = a_{0} + a_{1} x + a_{2} y$ . Matching the value and first partial derivatives and placing $x = x_{0} + h$ and $y = y_{0} + k$ result in $P_{1, (x_{0}, y_{0})} (x, y) = f (x_{0}, y_{0}) + f_{x} (x_{0}, y_{0}) h + f_{y} (x_{0}, y_{0}) k .$ For a better approximation we consider $P_{2, (x_{0}, y_{0})} = a_{0} + a_{1} x + a_{2} y + b_{1} x^{2} + b_{2} x y + b_{3} y^{2}$ . Matching the zero, first, and second partial derivatives results in
$P_{2, (x_{0}, y_{0})} (x, y) = f (x_{0}, y_{0}) + h f_{x} + k f_{y} + \frac{1}{2!} [h^{2} f_{x x} + 2 h k f_{x y} + k^{2} f_{y y}],$
where the partial derivatives are evaluated at $(x_{0}, y_{0})$ . The above expression can also be written as:
$P_{2, (x_{0}, y_{0})} (x, y) = f (x_{0}, y_{0}) + {[(h \frac{\partial}{\partial x} + k \frac{\partial}{\partial y}) f (x, y)]}_{\binom{x = x_{0}}{y = y_{0}}} + {[{(h \frac{\partial}{\partial x} + k \frac{\partial}{\partial y})}^{2} f (x, y)]}_{\binom{x = x_{0}}{y = y_{0}}},$
where
${(h \frac{\partial}{\partial x} + k \frac{\partial}{\partial y})}^{2} = h^{2} \frac{\partial^{2}}{\partial x^{2}} + 2 h k \frac{\partial^{2}}{\partial x \partial y} + k^{2} \frac{\partial^{2}}{\partial y^{2}} .$

Another form of writing $P_{2, (x_{0}, y_{0})}$ is:
$P_{2, (x_{0}, y_{0})} (x, y) = f (x_{0}, y_{0}) + [\begin{matrix} \frac{\partial f}{\partial x} & \frac{\partial f}{\partial y} \end{matrix}] [\begin{matrix} h \\ k \end{matrix}] + \frac{1}{2!} [\begin{matrix} h & k \end{matrix}] [\begin{matrix} \frac{\partial^{2} f}{\partial x^{2}} & \frac{\partial^{2} f}{\partial x \partial y} \\ \frac{\partial^{2} f}{\partial x \partial y} & \frac{\partial^{2} f}{\partial y^{2}} \end{matrix}] [\begin{matrix} h \\ k \end{matrix}],$

Hessian Matrix

where again the partial derivatives are evaluated at $(x_{0}, y_{0})$ . The $2 \times 2$ matrix in the above expression is called the Hessian matrix and is denoted by $H (x_{0}, y_{0})$ . We will talk about it later in this section.

We can use the results for functions of a single variable to derive formulas for the Taylor polynomial and the remainder of a function $f$ of two or more variables. Assume $f (x, y)$ is continuous and has continuous partial derivatives at $(x_{0}, y_{0})$ . Let $x = x_{0} + h t, y = y_{0} + k t$ where $x_{0}, y_{0}, h$ and $k$ are treated as constants and $t$ is a variable. Then $F (t) = f (x (t), y (t)) = f (x_{0} + h t + y_{0} + k t)$ By Taylor’s formula, we have:
$\begin{aligned} F (t) = & F (0) + F^{'} (0) t + \frac{1}{2!} F^{''} (0) t^{2} + \frac{1}{3!} F^{'''} (0) t + \dots \\ (†) & + & \frac{1}{m!} F^{(m)} (t) t^{m} + \frac{1}{(m + 1)!} F^{(m + 1)} (ξ) t^{m + 1} \end{aligned}$
where $ξ$ is a number between 0 and $t$ . Using the chain rule, as we saw before, we have:
$\begin{aligned} F^{'} (t) = & \frac{\partial f}{\partial x} \frac{d x}{d t} + \frac{\partial f}{\partial y} \frac{d y}{d t} \\ = h \frac{\partial f}{\partial x} + k \frac{\partial f}{\partial y} \\ = (h \frac{\partial}{\partial x} + k \frac{\partial}{\partial y}) f \\ = ((h, k) \cdot \vec{\nabla}) f \end{aligned}$
We can show
$\begin{aligned} F^{''} (t) & = h^{2} \frac{\partial^{2} f}{\partial x^{2}} + 2 h k \frac{\partial^{2} f}{\partial x \partial y} + k^{2} \frac{\partial^{2} f}{\partial y^{2}} \\ = {(h \frac{\partial}{\partial x} + k \frac{\partial}{\partial y})}^{2} f \\ = {((h, k) \cdot \vec{\nabla})}^{2} f, \end{aligned}$
the third derivative is
$\begin{aligned} F^{'''} (t) & = h^{3} \frac{\partial^{3} f}{\partial x^{3}} + 3 h^{2} k \frac{\partial^{3} f}{\partial x^{2} \partial y} + 3 h k^{2} \frac{\partial^{3} f}{\partial x \partial y^{2}} + k^{3} \frac{\partial^{3} f}{\partial y^{3}} \\ = {(h \frac{\partial}{\partial x} + k \frac{\partial}{\partial y})}^{3} f \\ = {((h, k) \cdot \vec{\nabla})}^{3} f \end{aligned}$
and in general:
$\begin{aligned} F^{(m)} (t) & = {(h \frac{\partial}{\partial x} + k \frac{\partial}{\partial y})}^{m} f \\ (**) & = {((h, k) \cdot \vec{\nabla})}^{m} f \end{aligned}$

This may be proved by induction.

Therefore

Induction has two steps: (1) We prove (**) holds true for $n = 1$ , (2) We prove that if (**) is true for any value $m = k$ , then it is also true for $m = k + 1$

$F^{(m)} (0) = {[{(h \frac{\partial}{\partial x} + k \frac{\partial}{\partial y})}^{m} f (x, y)]}_{\binom{x = x_{0}}{y = y_{0}}} = {((h, k) \cdot \vec{\nabla})}^{m} f (x_{0}, y_{0}),$ and
$F^{(m + 1)} (ξ) = {[{(h \frac{\partial}{\partial x} + k \frac{\partial}{\partial y})}^{m + 1} f (x, y)]}_{\binom{x = x_{0} + ξ h}{y = y_{0} + ξ k}} = {((h, k) \cdot \vec{\nabla})}^{m + 1} f (x_{0} + ξ h, y_{0} + ξ k)$
Substituting in ( $†$ ), we have
$\begin{aligned} F (t) = & [f]_{\binom{x = x_{0}}{y = y_{0}}} + t [(h \frac{\partial}{\partial x} + k \frac{\partial}{\partial y}) f]_{\binom{x = x_{0}}{y = y_{0}}} + \dots + \frac{t^{m}}{m!} [{(h \frac{\partial}{\partial x} + k \frac{\partial}{\partial y})}^{m} f]_{\binom{x = x_{0}}{y = y_{0}}} \\ + \frac{t^{m + 1}}{(m + 1)!} [{(h \frac{\partial}{\partial x} + k \frac{\partial}{\partial y})}^{m + 1} f]_{\binom{x = x_{0} + ξ h}{y = y_{0} + ξ k}} \\ = & f (x_{0}, y_{0}) + t ((h, k) \cdot \vec{\nabla}) f (x_{0}, y_{0}) + \dots + \frac{t^{m}}{m!} {((h, k) \cdot \vec{\nabla})}^{m} f (x_{0}, y_{0}) \\ + \frac{t^{m + 1}}{(m + 1)!} {((h, k) \cdot \vec{\nabla})}^{m + 1} f (x_{0} + ξ h, y_{0} + ξ k) \end{aligned}$
for some $ξ$ between 0 and $t$ . Because this is true for all values of $t$ , we can plug $t = 1$ in and find the Taylor’s formula for functions of two variables. Here we proved the following theorem for $n = 2$ , but generalization is easy.

Theorem 3. Let

f : U \to R

where

U \subseteq R^{n}

is an open set. Suppose

f

has continuous partial derivatives up (at least) to order

m + 1

, and consider

x_{0} \in U

and

h \in R^{n}

such that

x_{0} + t h \in U

for

0 \leq t \leq 1

. Then there is a number

0 < θ < 1

such that

\begin{aligned} f (x_{0} + h) = f (x) & + (h \cdot \vec{\nabla}) f (x_{0}) + \frac{1}{2!} {(h \cdot \vec{\nabla})}^{2} f (x_{0}) + \dots \\ + \frac{1}{m!} {(h \cdot \vec{\nabla})}^{m} f (x_{0}) + \frac{1}{(m + 1)!} {(h \cdot \vec{\nabla})}^{m + 1} f (x_{0} + θ h) \end{aligned}

Another form of writing Taylor’s formula is:

for some $0 < θ < 1$ .

If we place $h = x - x_{0}$ in the above formula, the polynomial that we obtain is called the polynomial approximation of $f$ of degree $m$ at $x_{0}$ .

Example 1

Given

f (x, y) = \sin x e^{y - 2 x}

, find a second degree polynomial approximation to

f

near the point

(\frac{π}{2}, π)

and use it to estimate the value

f (0.95 \frac{π}{2}, 1.1 π)

Solution

For the polynomial approximation of degree 2, we need to find the first and second partial derivatives of

f

$\begin{aligned} f (x, y) = \sin x e^{y - 2 x} \Rightarrow f (\frac{π}{2}, π) = 1, \\ f_{x} (x, y) = \cos x e^{y - 2 x} - 2 \sin x e^{y - 2 x} \Rightarrow f_{x} (\frac{π}{2}, π) = - 2, \\ f_{y} (x, y) = \sin x e^{y - 2 x} \Rightarrow f_{y} (\frac{π}{2}, π) = 1, \\ f_{x x} (x, y) = 3 \sin x e^{y - 2 x} - 4 \cos x e^{y - 2 x} \Rightarrow f_{x x} (\frac{π}{2}, π) = 3, \\ f_{x y} (x, y) = \cos x e^{y - 2 x} - 2 \sin x e^{y - 2 x} \Rightarrow f_{x y} (\frac{π}{2}, π) = - 2, \\ f_{y y} (x, y) = \sin x e^{y - 2 x} \Rightarrow f_{y y} (\frac{π}{2}, π) = 1. \end{aligned}$
Thus
$f (\frac{π}{2} + h, π + k) \approx 1 - 2 h + k + \frac{1}{2!} (3 h^{2} - 2 \times 2 h k + k^{2}) .$
If we place $x = \frac{π}{2} + h$ and $y = π + k$ , we obtain the second degree polynomial approximation of $f$ near $(π / 2, π)$
$\begin{aligned} f (x, y) \approx & P_{2, (\frac{π}{2}, π)} (x, y) = 1 - 2 (x - \frac{π}{2}) + (y - π) + \\ \frac{3}{2} {(x - \frac{π}{2})}^{2} - 2 (x - \frac{π}{2}) (y - π) + \frac{1}{2} (y - π)^{2} \end{aligned}$
Therefore:
$\begin{aligned} f (\frac{0.95}{2} π, 1.1 π) \approx & 1 - 2 (- 0.025 π) + 0.1 π + \frac{3}{2} (- 0.025 π)^{2} \\ - 2 (- 0.025 π) (0.1 π) + \frac{1}{2} (0.1 π)^{2} \approx 1.57919 . \end{aligned}$
The exact value of $f (\frac{0.95}{2} π, 1.1 π)$ is 1.59704. The error of this approximation is $\approx 1.12 %$ , while if we used the linear approximation $P_{1, (\frac{π}{2}, π)} = 1 - 2 (x - \frac{π}{2}) + (y - π)$ , the error would be $\approx 7.88 %$ .

The quadratic term $\sum_{i, j = 1}^{n} \frac{\partial^{2} f}{\partial x_{i} \partial x_{j}} (x_{0}) h_{i} h_{j}$ can be written as
$[\begin{matrix} h_{1} & h_{2} & \dots & h_{n} \end{matrix}] [\begin{matrix} \frac{\partial^{2} f}{\partial x_{1}^{2}} & \frac{\partial^{2} f}{\partial x_{1} \partial x_{2}} & \dots & \frac{\partial^{2} f}{\partial x_{1} \partial x_{n}} \\ \frac{\partial^{2} f}{\partial x_{2} \partial x_{1}} & \frac{\partial^{2} f}{\partial x_{2}^{2}} & \dots & \frac{\partial^{2} f}{\partial x_{2} \partial x_{n}} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ \frac{\partial^{2} f}{\partial x_{n} \partial x_{1}} & \frac{\partial^{2} f}{\partial x_{n} \partial x_{2}} & \dots & \frac{\partial^{2} f}{\partial x_{n}^{2}} \end{matrix}] [\begin{matrix} h_{1} \\ h_{2} \\ ⋮ \\ h_{n} \end{matrix}]$
Similar to the case of $n = 2$ , the $n \times n$ matrix of the second order derivatives ${[\frac{\partial^{2} f}{\partial x_{i} \partial x_{j}}]}_{n \times n}$ is called the Hessian matrix and is denoted by $H (x)$ . Therefore, we can write:
$\sum_{i, j = 1}^{n} \frac{\partial^{2} f}{\partial x_{i} \partial x_{j}} (x_{0}) h_{i} h_{j} = {(h \cdot \vec{\nabla})}^{2} f (x_{0}) = h^{T} H (x_{0}) h,$
where $h$ is considered an $n \times 1$ column matrix.¹ Using this notation, the linear expansion of $f$ can be written as:
$f (x_{0} + h) = f (x_{0}) + h \cdot \nabla f (x_{0}) + h^{T} H (x_{0} + θ h) h .$

¹ Some books consider $h$ as a $1 \times n$ row matrix. When there is any ambiguity, it is better to write it as $⟨ h | H (x_{0}) | h ⟩$

Review of Taylor’s Formula for Functions of a Single Variable

Show the proof

Hide the proof

Definitions of the “Taylor polynomial” and “remainder”

Show the proof

Hide the proof

Taylor’s Formula for Functions of Several Variables

Hessian Matrix