Hessian matrix

And it blew my mind because I thought, gee, it's absolutely brilliant.

It's doing Newton's method without calculating Hessian matrices of second derivatives.

You know, it's building up the quadratic information just by, as you move along the long pass, you can take a difference of gradients along a step over a difference in the variables, which is giving you some estimate of a second derivative.

And by putting that information in while you're doing your optimization, you actually build up a matrix, which you're using as an approximation to the Hessian and or its inverse.

And if you're doing this on a quadratic, you get the exact Hessian or inverse, whichever one you're actually approximating.

And that seemed almost like, wow, that's amazing.

And it's very efficient because you don't have to calculate second derivatives.

thesis