Let $f(x)=\sqrt{x+1}$ and $g(x)=x^{2}$. We can define a new function $h$ as
\[
h(x)=f(g(x))=f(x^{2})=\sqrt{x^{2}+1}.
\] That is, to obtain $h(x)$, we substitute $g(x)$ for $x$ in the expression $f(x)$.

To generalize this process suppose $f$ and $g$ are any two given functions. We start with a number $x$ in the domain of $g$ and apply $g$ to it to get $g(x)$, then we apply $f$ to $g(x)$ and thereby obtain the number $f(g(x))$. Obviously this process presupposes that it makes sense to calculate $f$ at the point $g(x)$. In other words, the new function is defined only if $g(x)$ is in the domain of $f$ (otherwise we cannot use $g(x)$ as the input for $f$).

Consecutive application of functions is known as composition of functions. The new function that takes $x$ and assigns to it the value $f(g(x))$ is often denoted by $f\circ g$. The symbol $f\circ g$ is read “$f$ circle $g$.” Figure 1(a) shows the composition $f\circ g$ as a machine diagram and Figure 1(b) illustrates it as an arrow diagram.

(a)
(b)
 Fig 1(a) Machine digram for $ f\circ g$. A composite (or composition) function $ f\circ g$ applies the $g$ machine to the input $x$ and then uses the output $g(x)$ as the input for the $f$ machine.
Fig. 1(b)  Arrow diagram for $ f\circ g$. If $x$ is in the domain of $ g$ and $ g(x)$ is in the domain of $f$, then we may compose $f$ and $ g$ to form $ f\circ g$.
Let $f$ and $g$ be two functions. For those points $x$ in the domain of $g$ for which $g(x)$ lies in the domain of $f$, the composite function $f\circ g$ is defined by
\[
(f\circ g)(x)=f(g(x)).
\]
  • In set notation, the domain of $f\circ g$ is
    \[\bbox[#F2F2F2,5px,border:2px solid black]{Dom(f\circ g)=\{x|\ x\in Dom(g)\text{ and }g(x)\in Dom(f)\}.}\]
  • Note that the order of composition of two functions matters. In calculating $(f\circ g)(x)$, we first evaluate $g$ at $x$ and then use $g(x)$ as the input to calculate $f$ of the result. But in calculating $(g\circ f)(x)$, we first evaluate $f$ at $x$ and then calculate $g$ at the the point $f(x)$. Therefore, $(f\circ g)(x)$ is often quite different from $(g\circ f)(x)$.
Example 1
Let $f(x)=\sqrt{x}$ and $g(x)=1+x^{2}$. Find the following functions and their domains.
(a) $f\circ g$
(b) $g\circ f$
Solution
We have
\[
Dom(f)=\{x|\ x\geq0\}=[0,\infty),\qquad Dom(g)=\mathbb{R}.
\] (a)
\begin{align*}
(f\circ g)(x) & =f(g(x))\\
& =f(1+x^{2})\\
& =\sqrt{1+x^{2}}.
\end{align*}
Because $g(x)=1+x^{2}\geq1$ for all $x$, $g(x)$ is always in the domain of $f$. Therefore
\[
Dom(f\circ g)=\mathbb{R}.
\] (b) 
\begin{align*}
(g\circ f)(x) & =g(f(x))\\
& =g(\sqrt{x})\\
& =1+(\sqrt{x})^{2}\\
& =1+x  & (\text{ if } x\geq 0)
\end{align*}
Because the domain of $g$ is the entire set of real numbers, $f(x)$ always lies in the domain of $g$. However, $f$ is defined only for $x\geq0$. Therefore, the domain of $g\circ f$ is the same as the domain of $f$, namely
\[
Dom(g\circ f)=Dom(f)=\{x|\ x\geq0\}=[0,\infty).
\]
Example 2
If $f(x)=\dfrac{x+1}{x-1}$, find the function $f\circ f$ and its domain.
Solution
Let’s first find $f\circ f$
\begin{align*}
(f\circ f)(x) & =f(f(x))\\
& =f\left(\dfrac{x+1}{x-1}\right)\\
& =\dfrac{\dfrac{x+1}{x-1}+1}{\dfrac{x+1}{x-1}-1}\\
& =\dfrac{\dfrac{x+1+x-1}{x-1}}{\dfrac{x+1-(x-1)}{x-1}}\\
& =\frac{2x}{2}=x
\end{align*}
Just by looking at the above formula, one may be tempted to say that the domain of $f\circ f$ is $\mathbb{R}$. However, the domain of $f\circ f$ consists of all $x$ that 
(1) $x$ lies in the domain of $f$,
(2) $f(x)$ lies in the domain of $f$. 

The domain of $f$ is all real numbers except $x=1$ (recall that division by zero is not defined). So to satisfy the first condition we have to exclude $x=1$. To satisfy the second condition, we also have to exclude $x$ for which
\[
\frac{x+1}{x-1}=1.
\] It turns out that this equation has no solution because it is equivalent to $x+1=x-1$ or $-1=1$. This means that $f(x)$ is always in the domain of $f$, and the second condition is automatically satisfied without any further restriction on $x$. Thus in this specific example
\[
Dom(f\circ f)=Dom(f)=\{x|\ x\neq1\}=\mathbb{R}-\{1\}.
\]

Example 3
Find functions $f$ and $g$ such that $h=f\circ g$ given that
\[
h(x)=\cos(x^{2}).
\]
Solution
To calculate $h(x)$ we first square $x$ and then take the cosine of $x^{2}$. We can therefore set
\[
g(x)=x^{2} \qquad (\text{square the input})
\] and
\[
f(x)=\cos x \qquad  (\text{ take cosine of the input})
\] We observe that
\[
(f\circ g)(x)=f(g(x))=f(x^{2})=\cos(x^{2}).
\]
We can combine three or more functions by composing them two at a time. For example, the function $F$ given by
\[
F(x)=\frac{1}{2+x^{2}},
\] is a composition of three functions $F=f\circ g\circ h$, where
\[
h(x)=x^{2},\qquad g(x)=x+2,\qquad f(x)=\frac{1}{x}.
\] Notice that we can compose $f$ and $g$ first then compose $f\circ g$ with $h$ or we can first compose $g$ and $h$ and then compose $f$ and $g\circ h$. In general, composition is associative. That is
\[
f\circ g\circ h=(f\circ g)\circ h=f\circ(g\circ h)
\]
Example 5
Find three functions $f,g,$ and $h$ such that $F=f\circ g\circ h$ given that
\[
F(x)=\sqrt{|x|+3}.
\]
Solution
To calculate $F(x)$, we fist take the absolute value of $x$, add 3, and then take the square root. We can therefore set
\[
h(x)=|x|,
\] \[
g(x)=x+3,
\] and
\[
f(x)=\sqrt{x}.
\] With this choice of $f,g,$ and $h$, we observe that
\[
(f\circ g\circ h)(x)=f\left(g\left(h(x)\right)\right)=f\left(g\left(|x|\right)\right)=f\left(|x|+3\right)=\sqrt{|x|+3}.
\]