Difference between revisions of "Principal Component Analysis"

Revision as of 00:09, 31 October 2011

Class 2 (16 September)

<pdfreflow> class_date = 16 September 2011 [1.65 Mb] button_label = Create my projector slides! show_page_layout = 1 show_frame_option = 1 pdf_file = lvm-class-2.pdf </pdfreflow>

Download these 3 CSV files and bring them on your computer:
- Peas dataset: http://datasets.connectmv.com/info/peas
- Food texture dataset: http://datasets.connectmv.com/info/food-texture
- Food consumption dataset: http://datasets.connectmv.com/info/food-consumption

Background reading

Reading for class 2
Linear algebra topics you should be familiar with before class 2:
- matrix multiplication
- that matrix multiplication of a vector by a matrix is a transformation from one coordinate system to another (we will review this in class)
- linear combinations (read the first section of that website: we will review this in class)
- the dot product of 2 vectors, and that they are related by the cosine of the angle between them (see the geometric interpretation section)

This illustration should help better explain what I trying to get across in class 2B

$p_{1}$ and $p_{2}$ are the unit vectors for components 1 and 2.
$x_{i}$ is a row of data from matrix $X$ .
${\hat{x}}_{i, 1} = t_{i, 1} p_{1}$ = the best prediction of $x_{i}$ using only the first component.
${\hat{x}}_{i, 2} = t_{i, 2} p_{2}$ = the improvement we add after the first component to better predict $x_{i}$ .
${\hat{x}}_{i} = {\hat{x}}_{i, 1} + {\hat{x}}_{i, 2}$ = is the total prediction of $x_{i}$ using 2 components and is the open blue point lying on the plane defined by $p_{1}$ and $p_{2}$ . Notice that this is just the vector summation of ${\hat{x}}_{i, 1}$ and ${\hat{x}}_{i, 2}$ .
$e_{i, 2}$ = is the prediction error vector because the prediction ${\hat{x}}_{i}$ is not exact: the data point $x_{i}$ lies above the plane defined by $p_{1}$ and $p_{2}$ . This $e_{i, 2}$ is the residual distance after using 2 components.
$x_{i} = {\hat{x}}_{i} + e_{i, 2}$ is also a vector summation and shows how $x_{i}$ is broken down into two parts: ${\hat{x}}_{i}$ is a vector on the plane, while $e_{i, 2}$ is the vector perpendicular to the plane.

Class 3 (23 September)

I would advise printing the slides out no more than 2 per page (leaving space for extra notes in today's class) <pdfreflow> class_date = 23 September 2011 [580 Kb] button_label = Create my projector slides! show_page_layout = 1 show_frame_option = 1 pdf_file = lvm-class-3.pdf </pdfreflow>

Background reading

Least squares:
- what is the objective function of least squares
- how to calculate the regression coefficient $b$ for $y = b x + e$ where $x$ and $y$ are centered vectors
- understand that the residuals in least squares are orthogonal to $x$
Some optimization theory:
- How an optimization problem is written with equality constraints
- The Lagrange multiplier principle for solving simple, equality constrained optimization problems. (Understanding the content on this page is very important).

Class 4 (30 September)

Background reading

Reading on cross validation

@@ Line 1: / Line 1: @@
+== Class 2  (16 September)  ==
-== Class notes ==
 <pdfreflow>
@@ Line 10: / Line 9: @@
 </pdfreflow>
-* Also download these 3 CSV files and bring them on your computer:
+* Download these 3 CSV files and bring them on your computer:
 ** Peas dataset: http://datasets.connectmv.com/info/peas
 ** Food texture dataset: http://datasets.connectmv.com/info/food-texture
 ** Food consumption dataset: http://datasets.connectmv.com/info/food-consumption
+=== Background reading ===
+* [http://literature.connectmv.com/item/13/principal-component-analysis Reading for class 2]
+* Linear algebra topics you should be familiar with before class 2:
+** matrix multiplication
+** that matrix multiplication of a vector by a matrix is a transformation from one coordinate system to another (we will review this in class)
+** [http://en.wikipedia.org/wiki/Linear_combination linear combinations] (read the first section of that website: we will review this in class)
+** the dot product of 2 vectors, and that they are related by the cosine of the angle between them (see the [http://en.wikipedia.org/wiki/Dot_product geometric interpretation section])
+This illustration should help better explain what I trying to get across in class 2B
+*  $p_{1}$  and  $p_{2}$  are the unit vectors for components 1 and 2.
+*  $x_{i}$  is a row of data from matrix  $X$ .
+*  ${\hat{x}}_{i, 1} = t_{i, 1} p_{1}$  = the best prediction of   $x_{i}$  using only the first component.
+*  ${\hat{x}}_{i, 2} = t_{i, 2} p_{2}$  = the improvement we add after the first component to better predict  $x_{i}$ .
+*  ${\hat{x}}_{i} = {\hat{x}}_{i, 1} + {\hat{x}}_{i, 2}$  = is the total prediction of  $x_{i}$  using 2 components and is the open blue point lying on the plane defined by  $p_{1}$  and  $p_{2}$ . Notice that this is just the vector summation of  ${\hat{x}}_{i, 1}$  and  ${\hat{x}}_{i, 2}$ .
+*  $e_{i, 2}$   = is the prediction error '''''vector''''' because the prediction  ${\hat{x}}_{i}$  is not exact: the data point   $x_{i}$  lies above the plane defined by  $p_{1}$  and  $p_{2}$ . This  $e_{i, 2}$  is the residual distance after using 2 components.
+*  $x_{i} = {\hat{x}}_{i} + e_{i, 2}$  is also a vector summation and shows how  $x_{i}$  is broken down into two parts:  ${\hat{x}}_{i}$  is a vector on the plane, while  $e_{i, 2}$  is the vector perpendicular to the plane.
+[[Image:geometric-interpretation-of-PCA-xhat-residuals.png|500px]]
+== Class 3  (23 September)  ==
 I would advise printing the slides out no more than 2 per page (leaving space for extra notes in today's class)
@@ Line 26: / Line 46: @@
 </pdfreflow>
-== Class preparation ==
+===Background reading ===
-=== Class 2 (16 September) ===
-* [http://literature.connectmv.com/item/13/principal-component-analysis Reading for class 2]
-* Linear algebra topics you should be familiar with before class 2:
-** matrix multiplication
-** that matrix multiplication of a vector by a matrix is a transformation from one coordinate system to another (we will review this in class)
-** [http://en.wikipedia.org/wiki/Linear_combination linear combinations] (read the first section of that website: we will review this in class)
-** the dot product of 2 vectors, and that they are related by the cosine of the angle between them (see the [http://en.wikipedia.org/wiki/Dot_product geometric interpretation section])
-=== Class 3 (23 September) ===
 * [http://stats4eng.connectmv.com/wiki/Least_squares_modelling Least squares]:
@@ Line 42: / Line 52: @@
 ** how to calculate the regression coefficient  $b$  for  $y = b x + e$  where  $x$  and  $y$  are centered vectors
 ** understand that the residuals in least squares are orthogonal to  $x$
 * Some optimization theory:
 ** How an optimization problem is written with equality constraints
 ** The [http://en.wikipedia.org/wiki/Lagrange_multiplier Lagrange multiplier principle] for solving simple, equality constrained optimization problems. ('''''Understanding the content on this page is very important''''').
-=== Class 4 (30 September) ===
+== Class 4  (30 September) ==
+===Background reading ===
 * Reading on [http://literature.connectmv.com/item/12/cross-validatory-estimation-of-the-number-of-components-in-factor-and-principal-components-models cross validation]
-== Update ==
-This illustration should help better explain what I trying to get across in class 2B
-*  $p_{1}$  and  $p_{2}$  are the unit vectors for components 1 and 2.
-*  $x_{i}$  is a row of data from matrix  $X$ .
-*  ${\hat{x}}_{i, 1} = t_{i, 1} p_{1}$  = the best prediction of   $x_{i}$  using only the first component.
-*  ${\hat{x}}_{i, 2} = t_{i, 2} p_{2}$  = the improvement we add after the first component to better predict  $x_{i}$ .
-*  ${\hat{x}}_{i} = {\hat{x}}_{i, 1} + {\hat{x}}_{i, 2}$  = is the total prediction of  $x_{i}$  using 2 components and is the open blue point lying on the plane defined by  $p_{1}$  and  $p_{2}$ . Notice that this is just the vector summation of  ${\hat{x}}_{i, 1}$  and  ${\hat{x}}_{i, 2}$ .
-*  $e_{i, 2}$   = is the prediction error '''''vector''''' because the prediction  ${\hat{x}}_{i}$  is not exact: the data point   $x_{i}$  lies above the plane defined by  $p_{1}$  and  $p_{2}$ . This  $e_{i, 2}$  is the residual distance after using 2 components.
-*  $x_{i} = {\hat{x}}_{i} + e_{i, 2}$  is also a vector summation and shows how  $x_{i}$  is broken down into two parts:  ${\hat{x}}_{i}$  is a vector on the plane, while  $e_{i, 2}$  is the vector perpendicular to the plane.
-[[Image:geometric-interpretation-of-PCA-xhat-residuals.png|500px]]

Difference between revisions of "Principal Component Analysis"

Revision as of 00:09, 31 October 2011

Contents

Class 2 (16 September)

Background reading

Class 3 (23 September)

Background reading

Class 4 (30 September)

Background reading

Navigation menu

Difference between revisions of "Principal Component Analysis"

Revision as of 00:09, 31 October 2011

Class 2 (16 September)

Background reading

Class 3 (23 September)

Background reading

Class 4 (30 September)

Background reading

Navigation menu

Search