Difference between revisions of "Principal Component Analysis"

From Latent Variable Methods in Engineering
Jump to navigation Jump to search
m
 
(30 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{ClassSidebar
{{ClassSidebarYouTube
| date =
| date = 16, 23, 30 September 2011
| dates_alt_text =  
| vimeoID1 = 9QzNOz_7i6U
| vimeoID1 = 29191853
| vimeoID2 = qDiPZp-FWc4
| vimeoID2 = 29230126
| vimeoID3 = y0Alf0VZ-1E
| vimeoID3 =  
| vimeoID4 = XfH_p1WAydM
| vimeoID5 = QLB-UJ1dFiE
| vimeoID6 = bysqF41Mgc0
| vimeoID7 = p3i-XsviARM
| vimeoID8 = Qb28yc3eM0Q
| vimeoID9 =
| course_notes_PDF =  
| course_notes_PDF =  
| course_notes_alt = Course notes
| course_notes_alt = Course notes  
| overheads_PDF =  
| overheads_PDF =  
| assignment_instructions = <!-- -->
| overheads_PDF_alt = Projector notes
| assignment_solutions = <!-- -->
| assignment_instructions =  
| video_download_link_MP4 = http://connectmv.com/media/latent/video/Class-2A.mp4
| assignment_solutions =
| video_download_link_MP4_size = 290.1 Mb
| video_download_link_MP4 = http://learnche.mcmaster.ca/media/LVM-2011-Class-02A.mp4
| video_download_link2_MP4 = http://connectmv.com/media/latent/video/Class-2B.mp4
| video_download_link2_MP4 = http://learnche.mcmaster.ca/media/LVM-2011-Class-02B.mp4
| video_download_link2_MP4_size 305.5 Mb
| video_download_link3_MP4 = http://learnche.mcmaster.ca/media/LVM-2011-Class-02C.mp4
| video_download_link3_MP4 = http://connectmv.com/media/latent/video/Class-2C.mp4
| video_download_link4_MP4 = http://learnche.mcmaster.ca/media/LVM-2011-Class-03A.mp4
| video_download_link3_MP4_size =  Mb
| video_download_link5_MP4 = http://learnche.mcmaster.ca/media/LVM-2011-Class-03B.mp4
| video_download_link6_MP4 = http://learnche.mcmaster.ca/media/LVM-2011-Class-03C.mp4
| video_download_link7_MP4 http://learnche.mcmaster.ca/media/LVM-2011-Class-04A.mp4
| video_download_link8_MP4 = http://learnche.mcmaster.ca/media/LVM-2011-Class-04B.mp4
| video_download_link9_MP4 =
| video_download_link_MP4_size = 290 Mb
| video_download_link2_MP4_size = 306 Mb
| video_download_link3_MP4_size = 294 Mb
| video_download_link4_MP4_size = 152 Mb
| video_download_link5_MP4_size = 276 Mb
| video_download_link6_MP4_size = 333 Mb
| video_download_link7_MP4_size = 198 Mb
| video_download_link8_MP4_size = 180 Mb
| video_download_link9_MP4_size =  Mb
| video_notes1 =
| video_notes1 =
* 00:00 to 21:37 Recap and overview of this class
* 21:38 to 42:01 Preprocessing: centering and scaling
* 42:02 to 57:07 Geometric view of PCA
| video_notes2 =
| video_notes2 =
{{{!}}
| video_notes3 =
{{!}}-
| video_notes4 =
{{!}} 00:00 {{!}}{{!}}to{{!}}{{!}} 58:07 {{!}}{{!}} {{!}}{{!}} Details coming soon
| video_notes5 =
{{!}}-
| video_notes6 =
{{!}}}
}}__NOTOC__
|| video_notes3 =
== Class 2 (16 September 2011) ==
{{{!}}
{{!}}-
{{!}} 00:00 {{!}}{{!}}to{{!}}{{!}} 1:03:50 {{!}}{{!}} {{!}}{{!}} Details coming soon
{{!}}-
{{!}}}
}}


== Class notes ==


<pdfreflow>   
[[Image:Nuvola_mimetypes_pdf.png|20px|link=Media:Lvm-class-2.pdf]] [[Media:Lvm-class-2.pdf|Download the class slides]] (PDF)
class_date        = 16 September 2011 [1.65 Mb]
button_label      = Create my projector slides!
show_page_layout  = 1
show_frame_option = 1
pdf_file          = lvm-class-2.pdf
</pdfreflow>


* Also download these 3 CSV files and bring them on your computer:
** Peas dataset: http://datasets.connectmv.com/info/peas
** Food texture dataset: http://datasets.connectmv.com/info/food-texture
** Food consumption dataset: http://datasets.connectmv.com/info/food-consumption


== Class preparation ==
* Download these 3 CSV files and bring them on your computer:
** Peas dataset: http://openmv.net/info/peas
** Food texture dataset: http://openmv.net/info/food-texture
** Food consumption dataset: http://openmv.net/info/food-consumption
 
=== Background reading ===


=== Class 2 (16 September) ===
* [http://literature.connectmv.com/item/13/principal-component-analysis Reading for class 2]
* [http://literature.connectmv.com/item/13/principal-component-analysis Reading for class 2]
* Linear algebra topics you should be familiar with before class 2:
* Linear algebra topics you should be familiar with before class 2:
** matrix multiplication
** matrix multiplication
** that matrix multiplication of a vector by a matrix is a transformation from one coordinate system to another (we will review this in class)
** that matrix multiplication of a vector by a matrix is a transformation from one coordinate system to another (we will review this in class)
** [http://en.wikipedia.org/wiki/Linear_combination linear combinations] (read the first section of that website: we will review this in class)
** [https://en.wikipedia.org/wiki/Linear_combination linear combinations] (read the first section of that website: we will review this in class)
** the dot product of 2 vectors, and that they are related by the cosine of the angle between them (see the [http://en.wikipedia.org/wiki/Dot_product geometric interpretation section])
** the dot product of 2 vectors, and that they are related by the cosine of the angle between them (see the [http://en.wikipedia.org/wiki/Dot_product geometric interpretation section])
=== Class 3 (23 September) ===
* [http://stats4eng.connectmv.com/wiki/Least_squares_modelling Least squares]:
** what is the objective function of least squares
** how to calculate the two regression coefficients \(b_0\) and \(b_1\) for \(y = b_0 + b_1x + e\)
** understand that the residuals in least squares are orthogonal to \(x\)
* Some optimization theory:
** how an optimization problem is written with equality constraints
** the [http://en.wikipedia.org/wiki/Lagrange_multiplier Lagrange multiplier principle] for solving simple, equality constrained optimization problems
* Reading on [http://literature.connectmv.com/item/12/cross-validatory-estimation-of-the-number-of-components-in-factor-and-principal-components-models cross validation]
== Update ==


This illustration should help better explain what I trying to get across in class 2B
This illustration should help better explain what I trying to get across in class 2B


{|
|-
| [[Image:geometric-interpretation-of-PCA-xhat-residuals.png|700px]]
| valign="top"|
* \(p_1\) and \(p_2\) are the unit vectors for components 1 and 2.
* \(p_1\) and \(p_2\) are the unit vectors for components 1 and 2.
* \( \mathbf{x}_i \) is a row of data from matrix \( \mathbf{X}\).
* \( \mathbf{x}_i \) is a row of data from matrix \( \mathbf{X}\).
Line 87: Line 70:
* \(\mathbf{e}_{i,2} \)  = is the prediction error '''''vector''''' because the prediction \(\hat{\mathbf{x}}_{i} \) is not exact: the data point  \( \mathbf{x}_i \) lies above the plane defined by \(p_1\) and \(p_2\). This \(e_{i,2} \) is the residual distance after using 2 components.
* \(\mathbf{e}_{i,2} \)  = is the prediction error '''''vector''''' because the prediction \(\hat{\mathbf{x}}_{i} \) is not exact: the data point  \( \mathbf{x}_i \) lies above the plane defined by \(p_1\) and \(p_2\). This \(e_{i,2} \) is the residual distance after using 2 components.
* \( \mathbf{x}_i = \hat{\mathbf{x}}_{i} +  \mathbf{e}_{i,2} \) is also a vector summation and shows how \( \mathbf{x}_i  \) is broken down into two parts: \(\hat{\mathbf{x}}_{i}  \) is a vector on the plane, while \( \mathbf{e}_{i,2} \) is the vector perpendicular to the plane.
* \( \mathbf{x}_i = \hat{\mathbf{x}}_{i} +  \mathbf{e}_{i,2} \) is also a vector summation and shows how \( \mathbf{x}_i  \) is broken down into two parts: \(\hat{\mathbf{x}}_{i}  \) is a vector on the plane, while \( \mathbf{e}_{i,2} \) is the vector perpendicular to the plane.
|}
[[Image:geometric-interpretation-of-PCA-xhat-residuals.png|500px]]
 
 
== Class 3 (23, 30 September 2011) ==
 
[[Image:Nuvola_mimetypes_pdf.png|20px|link=Media:Lvm-class-3.pdf]] [[Media:Lvm-class-3.pdf|Download the class slides]] (PDF)
 
 
===Background reading ===
 
* Least squares:
** what is the objective function of least squares
** how to calculate the regression coefficient \(b\) for \(y =  bx + e\) where \(x\) and \(y\) are centered vectors
** understand that the residuals in least squares are orthogonal to \(x\)
* Some optimization theory:
** How an optimization problem is written with equality constraints
** The [https://en.wikipedia.org/wiki/Lagrange_multiplier Lagrange multiplier principle] for solving simple, equality constrained optimization problems.
 
 
===Background reading ===
* Reading on [http://literature.connectmv.com/item/12/cross-validatory-estimation-of-the-number-of-components-in-factor-and-principal-components-models cross validation]

Latest revision as of 14:04, 17 September 2018

Class date(s): 16, 23, 30 September 2011
Video material (part 1)
Download video: Link (plays in Google Chrome) [290 Mb]


Video material(part 2)
Download video: Link (plays in Google Chrome) [306 Mb]


Video material (part 3)
Download video: Link (plays in Google Chrome) [294 Mb]


Video material (part 4)
Download video: Link (plays in Google Chrome) [152 Mb]


Video material (part 5)
Download video: Link (plays in Google Chrome) [276 Mb]


Video material (part 6)
Download video: Link (plays in Google Chrome) [333 Mb]


Video material (part 7)
Download video: Link (plays in Google Chrome) [198 Mb]


Video material (part 8)
Download video: Link (plays in Google Chrome) [180 Mb]

Class 2 (16 September 2011)

Nuvola mimetypes pdf.png Download the class slides (PDF)


Background reading

  • Reading for class 2
  • Linear algebra topics you should be familiar with before class 2:
    • matrix multiplication
    • that matrix multiplication of a vector by a matrix is a transformation from one coordinate system to another (we will review this in class)
    • linear combinations (read the first section of that website: we will review this in class)
    • the dot product of 2 vectors, and that they are related by the cosine of the angle between them (see the geometric interpretation section)

This illustration should help better explain what I trying to get across in class 2B

  • \(p_1\) and \(p_2\) are the unit vectors for components 1 and 2.
  • \( \mathbf{x}_i \) is a row of data from matrix \( \mathbf{X}\).
  • \(\hat{\mathbf{x}}_{i,1} = t_{i,1}p_1\) = the best prediction of \( \mathbf{x}_i \) using only the first component.
  • \(\hat{\mathbf{x}}_{i,2} = t_{i,2}p_2\) = the improvement we add after the first component to better predict \( \mathbf{x}_i \).
  • \(\hat{\mathbf{x}}_{i} = \hat{\mathbf{x}}_{i,1} + \hat{\mathbf{x}}_{i,2} \) = is the total prediction of \( \mathbf{x}_i \) using 2 components and is the open blue point lying on the plane defined by \(p_1\) and \(p_2\). Notice that this is just the vector summation of \( \hat{\mathbf{x}}_{i,1}\) and \( \hat{\mathbf{x}}_{i,2}\).
  • \(\mathbf{e}_{i,2} \) = is the prediction error vector because the prediction \(\hat{\mathbf{x}}_{i} \) is not exact: the data point \( \mathbf{x}_i \) lies above the plane defined by \(p_1\) and \(p_2\). This \(e_{i,2} \) is the residual distance after using 2 components.
  • \( \mathbf{x}_i = \hat{\mathbf{x}}_{i} + \mathbf{e}_{i,2} \) is also a vector summation and shows how \( \mathbf{x}_i \) is broken down into two parts: \(\hat{\mathbf{x}}_{i} \) is a vector on the plane, while \( \mathbf{e}_{i,2} \) is the vector perpendicular to the plane.

Geometric-interpretation-of-PCA-xhat-residuals.png


Class 3 (23, 30 September 2011)

Nuvola mimetypes pdf.png Download the class slides (PDF)


Background reading

  • Least squares:
    • what is the objective function of least squares
    • how to calculate the regression coefficient \(b\) for \(y = bx + e\) where \(x\) and \(y\) are centered vectors
    • understand that the residuals in least squares are orthogonal to \(x\)
  • Some optimization theory:
    • How an optimization problem is written with equality constraints
    • The Lagrange multiplier principle for solving simple, equality constrained optimization problems.


Background reading