PCA — Principal Component Analysis
About this app
Principal axes of a 3D point cloud as eigenvectors of the covariance: projection onto k components with perpendiculars, reconstruction error = sum of the discarded λ, scree plot, standardising and the difference from the regression plane.
The idea
Principal component analysis (PCA) finds the directions in which a point cloud spreads the most. These directions are the eigenvectors of the covariance matrix C, and each eigenvalue λ is the variance along its direction. PC₁ carries the most variance, PC₂ the most of what remains, and all principal axes are mutually perpendicular. PCA therefore only rotates the coordinate system. Keeping the first k components and projecting onto them reduces the dimension; the mean squared reconstruction error is exactly the sum of the discarded eigenvalues.
What you see
A 3D point cloud (datasets “plane”, “cigar” and “disk”) with principal axes PC₁ to PC₃, the PC plane and a variance ellipsoid. “Project” pushes each point onto the first k components, and perpendicular lines show what is lost. A scree plot shows explained variance per component and cumulatively; a maths panel lists the covariance matrix, eigenvectors, eigenvalues and the trace Σλ. For comparison you can overlay a regression plane, which optimises a different criterion.
Things to try
Project with k = 2 and k = 1 and compare the mean squared perpendicular distance with the sum of discarded λ: the two numbers match. Increase the scale of x₁: its variance grows with the square of the factor and PC₁ swings onto the x₁ axis. “Standardise” gives every feature variance 1. On the “disk” dataset λ₁ and λ₂ are almost equal, so the direction within the plane is not well defined.
Frequently asked questions
- What is principal component analysis (PCA)?
- PCA is a dimensionality reduction method. It rotates the coordinate system so that the first axis captures the largest variance in the data, the second the largest remaining variance, and so on. Keeping only the first few axes describes the data with fewer features while losing as little information as possible.
- Why are principal components eigenvectors of the covariance matrix?
- The variance of the data along a unit vector v is vᵀCv. Subject to ‖v‖ = 1, this is maximised exactly when v is an eigenvector of C with the largest eigenvalue, and that eigenvalue is the variance in that direction. Because C is symmetric, its eigenvectors are orthogonal.
- How is PCA different from linear regression?
- Regression predicts one target feature from the others and minimises only the vertical errors in that target. PCA treats all features alike and minimises the perpendicular distance to the plane. The two planes therefore generally do not coincide.
- What can I do in this app?
- You choose a dataset, the number of kept components k, noise and the scale of x₁, then project the cloud. A variance panel compares reconstruction error with discarded eigenvalues, and a maths panel shows covariance, eigenvectors and eigenvalues. You can also overlay a regression plane and read off the angle between the two planes.
Subject: Machine Learning
More from Machine Learning
- AdaBoost
- Anatomy of a prompt
- Data privacy traffic lights
- Decision Tree
- Ensemble: Bagging vs. Boosting
- k-Means
- k-Nearest Neighbors (kNN)
- AI theatre
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps