Recipe

Run one cycle of Powell's method

Use when: you're minimizing a function using only function values, no gradient, and already have a set of n search directions to cycle through (initially the coordinate axes)

Goal: an updated point plus an updated, more effective direction set to start the next cycle from

Taught in Powell's method

1. 1

   Starting from P_0, for i=1,\dots,n minimize f along d_i to get P_i. Each one is an ordinary 1D line search using only function values, exactly like a single coordinate-descent step.

   Example

   From P_0=(4,-2), line search along d_1=(1,0) gives P_1=(1.9604,-2). Line search along d_2=(0,1) from P_1 gives P_2=(1.9604,-1.9216).

   Why: each individual search is exactly coordinate descent's own line search — nothing here yet requires a gradient or a Hessian, only comparisons of function values.

2. 2

   Set d_{\text{new}}=P_n-P_0: the TOTAL displacement across the whole cycle, not any single line search's own step.

   Example

   d_{\text{new}}=P_2-P_0=(1.9604-4,\ -1.9216-(-2))=(-2.0396,\,0.0784).

   Why: this direction captures how the objective's own valley actually runs, discovered purely by comparing endpoints — exactly what lets the direction set improve cycle over cycle without ever touching a derivative.

3. 3

   From P_n, line-search along d_{\text{new}} to get P_{n+1}, the cycle's actual result point.

   Example

   From P_2, the line search along d_{\text{new}} gives t^*\approx0.0407, so P_3\approx(1.8774,-1.9184).

   Why: Steps 1–2 only located a promising new direction; this step is what actually banks the cycle's progress along it.

4. 4

   Replace d_1, the oldest direction in the set, with d_{\text{new}}. Keep the rest in order and begin the next cycle from P_{n+1}.

   Example

   Cycle 2 continues from P_3\approx(1.8774,-1.9184) with the updated set \{d_2=(0,1),\,d_{\text{new}}\approx(-0.9996,0.0385)\}, replacing \{d_1,d_2\}.

   Why: replacing the oldest direction is what lets the set evolve away from the coordinate axes, cycle after cycle, toward directions built from the objective's own behavior — matching conjugate gradient's fast convergence with no gradient ever formed.