delta method

Posted on Edited on In

The Taylor series of a real or complex-valued function f (x) that is infinitely differentiable at a real or complex number \(a\) is the power series \[ f(x)\approx f(a)+\frac{f'(a)}{1}(x-a)+\frac{f''(a)}{2!}(x-a)^2+\frac{f'''(a)}{3!}(x-a)^3+\cdots \] Where \(n!\) denotes the factorial of \(n\). In the more compact sigma notation, this can be written as \[ \underset{n=0}{\overset{\infty}{\sum}}\frac{f^{(n)}(a)}{n!}(x-a)^n \] The property of taylor series determines two things:

  • The approximation would be more closer to the original value of \(f(x)\) if we add more terms
  • As \(x\) is more closer to the \(a\), the approximation will be more accurate

We provide the following example to illustrate above two properties \[ \begin{gather*} f(x)=lnx(x\in(0,1))\\ f(x)\approx lnx_0+(x-x_0)\frac{1}{x_0}(\text{first order})\\ f(x)\approx lnx_0+(x-x_0)\frac{1}{x_0}-\frac{1}{2}(x-x_0)^2\frac{1}{x_0^2}(\text{second order}) \end{gather*} \]

1
2
3
4
5
6
7
8
9
10
11
12
13
c1 <- curve(log(x))
c2 <- curve(log(0.5)+2*(x-0.5))
plot(curve(log(0.5)+2*(x-0.5)-0.5*(x-0.5)^2*4),
     col = 'red',type = 'l',xlab = 'x',ylab = 'y',
     ylim = c(-3,0),main = 'Taylor approximation for lnx at point 0.5')
lines(c1$x,c1$y,col = 'green')
lines(c2$x,c2$y,col = 'blue')
legend('topleft',
       c('lnx','first order','second order'),
       inset = 0.001,
       cex = 0.5,
       lty = c(1,1),
       col = c('green','blue','red'))

hi

Below is a good tip for explaining the delta method, from (Alan. H. Feiveson, NASA)

The delta method, in its essence, expands a function of a random variable about its mean, usually with a one-step Taylor approximation, and then takes the variance. For example, if we want to approximate the variance of \(f(X)\) where \(X\) is a random variable with mean \(\mu\) and \(f()\) is differentiable, we can try \[ f(x)=f(\mu)+(x-\mu)f'(\mu) \] so that \[ \begin{gather*} var[f(X)]=var(f(\mu)+(X-\mu)f'(\mu))\\ =var(X-\mu)[f'(\mu)]^2\\ =var(X)[f'(\mu)]^2 \end{gather*} \] This is a good approximation only if \(X\) has a high probability of being close enough to its mean(\(\mu\)) so that the Taylor approximation is still good.

Reference