Derivatives are required at the core of many numerical algorithms. Unfortunately, they are usually computed *inefficiently* and *approximately* by some variant of the finite difference approach
$$ f'(x) \approx \frac{f(x+h) - f(x)}{h}, h \text{ small }. $$
This method is *inefficient* because it requires \( \Omega(n) \) evaluations of \( f : \mathbb{R}^n \to \mathbb{R} \) to compute the gradient \( \nabla f(x) = \left( \frac{\partial f}{\partial x_1}(x), \cdots, \frac{\partial f}{\partial x_n}(x)\right) \), for example. It is *approximate* because we have to choose some finite, small value of the step length \( h \), balancing floating-point precision with mathematical approximation error.

One option is to explicitly write down a function which computes the exact derivatives by using the rules that we know from Calculus. However, this quickly becomes an error-prone and tedious exercise. **There is another way!** The field of automatic differentiation provides methods for automatically computing *exact* derivatives (up to floating-point error) given only the function \( f \) itself. Some methods use many fewer evaluations of \( f \) than would be required when using finite differences. In the best case, **the exact gradient of \( f \) can be evaluated for the cost of \( O(1) \) evaluations of \( f \) itself**. The caveat is that \( f \) cannot be considered a black box; instead, we require either access to the source code of \( f \) or a way to plug in a special type of number using operator overloading.

JuliaDiff is an informal organization which aims to unify and document packages written in Julia for evaluating derivatives. The technical features of Julia, namely, multiple dispatch, source code via reflection, JIT compilation, and first-class access to expression parsing make implementing and using techniques from automatic differentiation easier than ever before (in our biased opinion). Packages hosted under the JuliaDiff organization follow the same guidelines as for JuliaOpt; namely, they should be actively maintained, well documented and have a basic testing suite.

Below we list the packages that are currently included in the JuliaDiff organization and their testing status on the latest Julia release, if available.

ForwardDiff |
ForwardDiff provides user-friendly methods for performing forward-mode automatic differentiation on native Julia code. ForwardDiff offers several advanced features, such as nested differentiation, a non-allocating API, and a SIMD-vectorizable multidimensional dual number type. |

ReverseDiff |
ReverseDiff provides user-friendly methods for performing reverse-mode automatic differentiation on native Julia code. ReverseDiff implements many linear algebra optimizations, flexible performance annotations, and composes well with ForwardDiff (enabling mixed-mode AD). |

ReverseDiffSource |
ReverseDiffSource implements source-to-source reverse-mode automatic differentiation for native Julia functions and expressions. ReverseDiffSource supports gradients and higher-order derivatives, and accepts a subset of valid Julia syntax, including intermediate assignments. |

TaylorSeries |
TaylorSeries implements truncated multivariate power series for high-order integration of ODEs and forward-mode automatic differentiation of arbitrary order derivatives via operator overloading. |

DualNumbers |
DualNumbers provides a dual number type which can be used to take first-order scalar derivatives of native Julia code. |

HyperDualNumbers |
HyperDualNumbers provides a hyper-dual number type which can be used to take first-order and second-order scalar derivatives of native Julia code. |

These Julia packages also provide differentiation functionalities.

AutoDiffSource |
Fast reverse mode differentiation for scalar and tensor functions using source code transformation. |

Calculus |
Provides methods for symbolic differentiation and finite-difference approximations. |

PowerSeries |
Implements truncated power series type which can be used for forward-mode automatic differentiation of arbitrary order derivatives via operator overloading. |

ReverseDiffSparse |
Implements reverse-mode automatic differentiation for gradients and sparse Hessian matrices given closed-form expressions. |

SymEngine |
Implements symbolic differentiation. |

Packages implementing automatic differentiation techniques are already in use in the broader Julia ecosystem:

- Optim and NLsolve use
**ForwardDiff**to compute exact gradients and Jacobians of user-provided functions. Refer to each package's documentation to enable AD. - JuMP uses
**ReverseDiffSparse**and**ForwardDiff**to compute gradients and sparse Hessians, providing them to efficient interior-point solvers. - Klara, an MCMC engine, uses
**ForwardDiff**and**ReverseDiff**to compute gradients and Hessians of (possibly unnormalized) target distributions.

Discussions on JuliaDiff and its uses may be directed to the Julia Discourse forum or the julia-opt mailing list. The site autodiff.org serves as a portal for the academic community. For a well-written simple introduction to reverse-mode automatic differentiation, see Justin Domke's blog post. Finally, automatic differentiation techniques have been implemented in a variety of languages. If you would prefer not to use Julia, see the wikipedia page for a comprehensive list of available packages.

This website was made with Skeleton, based on Iain Dunning's work on JuliaOpt. Something wrong? Submit issue. Contact: Miles Lubin (mlubin AT mit DOT edu)