quasi-Newton method

A quasi-Newton method approximates Newton's method's curvature information without ever forming the true Hessian H^{(k)}. Each iteration updates an approximation B^{(k)} to the INVERSE Hessian using only the gradient change y_k=\nabla f(x^{(k+1)})-\nabla f(x^{(k)}) and the step already taken s_k=x^{(k+1)}-x^{(k)}, then steps as x^{(k+1)}=x^{(k)}-\alpha_k B^{(k)}\nabla f(x^{(k)}). BFGS and DFP are the two classic update rules for B^{(k)} — both recover Newton-like curvature purely from gradients already computed, at the cost of needing a few iterations before B^{(k)} is a good approximation.