Non-linear programming

Class date(s):	04 February 2015

Resources

Scroll down, if necessary, to see the resources.

Date	Class number	Topic	Slides/handouts for class	Video file	References and Notes
04 February	05A	Why consider unconstrained, single-variable problems Newton's method review to solve these problems	Handout from class	Video
09 February	06A	Newton's method reviewed again for unconstrained, single-variable problems Using finite differences instead in Newton's method Multivariate unconstrained optimization	Handout from class	Video
11 February	06B	Unconstrained single-variable optimization using gradient search Unconstrained multivariate optimization using gradient search Understanding the line search problem	Handout from class	Video
16 to 27 February	07	Reading week break and midterm
02 March	08A	Unconstrained optimization in two variables review Contrasting it back to the single variable case Extending to the multidimensional Newton's method	Handout from class	Video
04 March	08B	Examples on the multidimensional Newton's method Quasi Newton method in multiple dimensions Positive and negative definiteness of the Hessian	Handout from class	Video	Code used in class (see below)
09 March	09A	Constrained nonlinear optimization introduction Model formulation (convert a problem to mathematics)	Handout from class	Video
11 March	09B	Guest lecture	Handout from class	http://learnche.mcmaster.ca/media/2015-4G3-Class-09B.mp4 Video]
16 March	10A	Convexity, concavity Guarantees on when problems are globally optimal	Handout from class (continued with handout 09A)	Video
18 March	10B	Lagrange multiplier method for constrained optimization Interpretation of the Lagrange multiplier constraints	Handout from class	Video
23 March	11A	The Nelder-Mead method (several of you are using it in your projects) Practice with using the Nelder Mead method. Optimize this system: http://yint.org/nm	Handout from class	Video

Taking full Newton's steps to solve the class example

clear all;
close all;
clc;
[X1,X2] = meshgrid(-0.5:0.1:6, 0:0.01:9);
Z = func(X1,X2);
contour(X1, X2, Z)
hold on
grid on

x = [1,3]';
plot(x(1), x(2), 'o')
text(x(1)+0.2, x(2), '0')

for k = 1:10
   slope = -first_deriv(x)
   step = hessian(x)\slope;   % Solves the Ax=b problem, as x = A\b
   x = x + step;
   plot(x(1), x(2), '*')
   text(x(1)+0.1, x(2), num2str(k))
end

func.m

function y = func(x1,x2)
  y = 4.*x1.*x2 - 5.*(x1-2).^4 - 3.*(x2-5).^4;

first_deriv.m

function y = first_deriv(x)
  y = [4*x(2) - 20*(x(1)-2)^3; 
       4*x(1) - 12*(x(2)-5)^3];

hessian.m

function y = hessian(x)
  y = [-60*(x(1)-2)^2, 4; 
        4, -36*(x(2)-5)^2];