Finite Difference Approximations of Derivatives

The NLMIXED Procedure

Finite Difference Approximations of Derivatives

The FD= and FDHESSIAN= options specify the use of finite difference approximations of the derivatives. The FD= option specifies that all derivatives are approximated using function evaluations, and the FDHESSIAN= option specifies that second-order derivatives are approximated using gradient evaluations.

Computing derivatives by finite difference approximations can be very time consuming, especially for second-order derivatives based only on values of the objective function (FD= option). If analytical derivatives are difficult to obtain (for example, if a function is computed by an iterative process), you might consider one of the optimization techniques that uses first-order derivatives only (QUANEW, DBLDOG, or CONGRA).

Forward Difference Approximations

The forward difference derivative approximations consume less computer time, but they are usually not as precise as approximations that use central difference formulas.

For first-order derivatives, n additional function calls are required:
$g_i = {\partial f \over \partial \theta_i} \approx {f(\theta + h_ie_i) - f(\theta) \over h_i}$
For second-order derivatives based on function calls only (Dennis and Schnabel 1983, p. 80), n+n²/2 additional function calls are required for dense Hessian:
${\partial^2 f \over \partial \theta_i \partial \theta_j} \approx {f(\theta+h_ie_i+h_je_j) - f(\theta+h_ie_i) - f(\theta+h_je_j) + f(\theta)\over h_i h_j}$
For second-order derivatives based on gradient calls (Dennis and Schnabel 1983, p. 103), n additional gradient calls are required:
${\partial^2 f \over \partial \theta_i \partial \theta_j} \approx {g_i(\theta + ... ... - g_i(\theta) \over 2h_j} + {g_j(\theta + h_ie_i) - g_j(\theta) \over 2h_i}$

Central Difference Approximations

Central difference approximations are usually more precise, but they consume more computer time than approximations that use forward difference derivative formulas.

For first-order derivatives, 2n additional function calls are required:

$g_i = {\partial f \over \partial \theta_i} \approx {f(\theta + h_ie_i) - f(\theta - h_ie_i) \over 2h_i}$
For second-order derivatives based on function calls only (Abramowitz and Stegun 1972, p. 884), 2n+4n²/2 additional function calls are required.

${\partial^2 f \over \partial \theta^2_i} \approx {-f(\theta + 2h_ie_i) + 16f(\... ...i) - 30f(\theta) + 16f(\theta - h_ie_i) - f(\theta - 2h_ie_i) \over 12h^2_i}$

${\partial^2 f \over \partial \theta_i \partial \theta_j} \approx {f(\theta+h_i... ..._i-h_je_j) - f(\theta-h_ie_i+h_je_j) + f(\theta-h_ie_i-h_je_j) \over 4h_ih_j}$
For second-order derivatives based on gradient calls, 2n additional gradient calls are required:

${\partial^2 f \over \partial \theta_i \partial \theta_j} \approx {g_i(\theta + ... ... h_je_j) \over 4h_j} + {g_j(\theta + h_ie_i) - g_j(\theta - h_ie_i) \over 4h_i}$

You can use the FDIGITS= = option to specify the number of accurate digits in the evaluation of the objective function. This specification is helpful in determining an appropriate interval size h to be used in the finite difference formulas.

The step sizes h_j, j = 1, ... ,n are defined as follows.

For the forward difference approximation of first-order derivatives using function calls and second-order derivatives using gradient calls, $h_j = \sqrt[2]{\eta} (1 + |\theta_j|)$ .
For the forward difference approximation of second-order derivatives using only function calls and all central difference formulas, $h_j = \sqrt[3]{\eta} (1 + |\theta_j|)$ .

The value of $\eta$ is defined by the FDIGITS= option:

If you specify the number of accurate digits using FDIGITS=r, $\eta$ is set to 10^-r.
If you do not specify the FDIGITS= option, $\eta$ is set to the machine precision $\epsilon$ .

Chapter Contents
Previous
Next
Top