Optimization

Create a free account to open this chapter.

• Objective functions and decision variables ~16 min
• Constraints and the feasible region ~16 min
• Optimal solutions: local versus global ~15 min
• Continuous versus discrete optimization ~14 min
• Problem classes: LP, NLP, IP, MINLP ~17 min
• Why complexity matters: P, NP-complete, and NP-hard as vocabulary ~15 min
• The model-solve-refine cycle ~14 min
• Translating a problem into an optimization model: sets, parameters, variables, objective, constraints ~20 min
• Optimization across the sciences and industry ~13 min
• From problem to problem type: a first look at classifying what you're solving ~15 min

Create a free account to open this chapter.

• Critical points and the first derivative test, revisited ~16 min
• The second derivative test, revisited ~16 min
• Global extrema on closed, bounded regions, revisited ~15 min
• Optimization word problems, revisited and extended ~18 min
• Bracketing a minimum without derivatives ~16 min
• Golden-section search ~22 min
• Fibonacci search ~18 min
• Quadratic-fit search ~17 min
• The gradient and directional derivatives in several variables, revisited ~17 min
• The Hessian matrix and second-order conditions ~19 min
• Classifying critical points: bowls, domes, and saddles ~18 min

Create a free account to open this chapter.

• The descent-direction framework ~16 min
• Gradient descent: the update rule ~18 min
• Choosing a learning rate ~15 min
• Convergence, divergence, and overshoot ~18 min
• The zig-zag problem in narrow valleys ~18 min
• Exact versus inexact line search ~16 min
• The Armijo (sufficient decrease) condition ~17 min
• The Wolfe (curvature) conditions ~17 min
• Backtracking line search ~16 min
• Momentum and Nesterov momentum ~20 min

Create a free account to open this chapter.

• Newton's method for optimization ~21 min
• Quadratic convergence ~19 min
• When the Hessian is singular or indefinite ~22 min
• Quasi-Newton methods: DFP and BFGS ~23 min
• Limited-memory BFGS ~20 min
• Conjugate gradient descent ~21 min
• Preconditioning ~19 min
• Coordinate descent ~20 min
• Powell's method ~21 min
• Pattern search and Nelder-Mead ~24 min
• Choosing a method when you can't (or won't) compute derivatives ~20 min

Create a free account to open this chapter.

• Convex sets and convex combinations ~18 min
• The convex hull and conic hull ~16 min
• Convex cones ~14 min
• Convex functions and epigraphs ~18 min
• Jensen's inequality, revisited ~15 min
• The first-order (tangent-line) characterization of convexity ~17 min
• The second-order (Hessian) characterization of convexity ~17 min
• Operations that preserve convexity ~15 min
• Strictly and strongly convex functions ~16 min
• Why convexity rules out bad local minima ~16 min

Create a free account to open this chapter.

• Equality-constrained optimization, revisited ~16 min
• Lagrange multipliers as gradient parallelism, revisited ~15 min
• Shadow prices and the meaning of a multiplier ~15 min
• Inequality constraints: active versus inactive ~15 min
• The Lagrangian for mixed constraints ~16 min
• Constraint qualification and the Slater condition ~17 min
• The KKT conditions: stationarity ~18 min
• KKT primal and dual feasibility ~14 min
• KKT complementary slackness ~16 min
• Solving small problems by hand with KKT ~22 min
• Two constraints at once, revisited ~17 min
• When KKT conditions are — and aren't — sufficient ~17 min

Create a free account to open this chapter.

• Linear programs, revisited from a modeling lens ~16 min
• Standard and canonical form ~15 min
• Slack and surplus variables ~15 min
• Vertices and basic feasible solutions ~17 min
• The extreme point theorem ~15 min
• The simplex method: choosing an entering variable ~18 min
• The simplex method: the ratio test and pivoting ~20 min
• The Big-M method ~20 min
• Two-phase simplex ~20 min
• Degeneracy and cycling ~16 min

Create a free account to open this chapter.

• Constructing the dual of a linear program ~20 min
• Weak duality ~18 min
• Strong duality ~18 min
• Complementary slackness for linear programs ~18 min
• Shadow prices, revisited and formalized ~18 min
• Sensitivity to cost changes ~22 min
• Sensitivity to resource changes ~22 min
• Reading a sensitivity report ~20 min
• Duality as a certificate of optimality ~18 min

Create a free account to open this chapter.

• Optimization on graphs: a modeling lens ~15 min
• Shortest path, revisited ~16 min
• Minimum spanning trees, revisited ~15 min
• Max flow and min cut, revisited ~16 min
• The transportation problem ~18 min
• The assignment problem ~16 min
• Integer programming: why rounding fails ~16 min
• Branch and bound ~20 min
• Cutting planes ~17 min
• The 0-1 knapsack problem ~18 min
• Dynamic programming: optimal substructure and the Bellman equation ~18 min
• Dynamic programming: solving the knapsack problem ~20 min

Create a free account to open this chapter.

• Why inequality constraints resist a direct Lagrangian solve ~16 min
• Penalty methods ~18 min
• The augmented Lagrangian method ~18 min
• Barrier methods and the central path ~18 min
• Interior-point methods ~18 min
• Trust-region methods for constrained problems ~17 min
• Sequential quadratic programming: a first look ~15 min
• Choosing among constrained-optimization algorithms ~16 min
• A worked comparison: penalty versus barrier ~20 min

Create a free account to open this chapter.

• Stochastic gradient descent ~22 min
• Mini-batches and epochs ~20 min
• Learning-rate schedules ~20 min
• Adaptive methods: Adagrad, RMSProp, and Adam ~26 min
• Escaping saddle points with noise ~20 min
• Simulated annealing ~24 min
• Genetic algorithms ~26 min
• Particle swarm optimization ~24 min
• When a population method beats a gradient method ~20 min
• Choosing an algorithm family, revisited ~22 min

Create a free account to open this chapter.

• Least squares as optimization ~18 min
• Logistic regression as optimization ~20 min
• The loss-minimization framing of supervised learning ~16 min
• L2 regularization (ridge) ~20 min
• L1 regularization (lasso) and sparsity ~22 min
• Elastic net ~18 min
• Subgradients for nonsmooth objectives ~20 min
• Proximal methods and soft-thresholding ~20 min
• A first look at ISTA and FISTA ~16 min

Create a free account to open this chapter.

• Dominance and Pareto optimality ~18 min
• The weighted-sum method and its blind spot ~20 min
• The $\varepsilon$-constraint method ~20 min
• The lexicographic method ~18 min
• Goal programming ~19 min
• Optimizing under uncertain parameters ~18 min
• Worst-case (robust) optimization ~20 min
• Chance-constrained optimization ~20 min
• Value-at-risk and conditional value-at-risk ~20 min
• How much does robustness cost? ~19 min

Create a free account to open this chapter.

• Conditioning and ill-conditioned problems ~18 min
• Stopping criteria and convergence in practice ~16 min
• Comparing algorithms on the same problem ~18 min
• Production planning ~18 min
• Logistics and transportation ~18 min
• Portfolio optimization ~20 min
• Facility location ~18 min
• Scheduling ~16 min
• Classifying a new problem: which recipe applies? ~20 min
• A capstone: formulate, classify, solve ~25 min