Principal Component Analysis (PCA)

Suppose an energy research institute is comparing the energy use characteristics of 120 regions across the country. The institute collected several indicators for each region, including electricity use, natural gas use, summer cooling demand, winter heating demand and peak power demand.

But these indicators can be closely related to one another. Regions that consume a lot of electricity often also have high peak power demand, and regions with high heating demand may also show high natural gas use.

When variables carry similar information like this, it can be more useful to summarize their common variation with a few new indicators than to compare every indicator separately. Principal component analysis (PCA) is a method that reduces dimensions while retaining as much of the information in several numeric variables as possible.

In this article we use a single example to look at how principal components are built, standardization, eigenvalues and explained variance ratios, component loadings and scores, visualization, how PCA differs from factor analysis, and how to interpret results and what to watch out for.

The first principal component (PC1) is the direction that explains the most variation in the data, and the second principal component (PC2) is the direction, orthogonal to the first, that explains the most of the remaining variation. Later components are built on the same principle.

For example, if six energy indicators are summarized with two principal components, instead of comparing six values one by one for each region, you can look at the overall characteristics through two coordinates, PC1 and PC2.

However, using only two components loses some of the information in the original data. How much information was retained must be checked with the explained variance ratio.

PCA is a method for summarizing the variation in observed data. Principal components are not automatically given a causal meaning, and the researcher must interpret and name new components by looking at the contributions of the original variables.

Besides comparing regional energy characteristics, it can be used to explore patterns across many health checkup indicators, condense a product's many performance metrics, or reduce the dimensionality of features fed into a machine learning model.

However, reducing dimensions with PCA does not necessarily improve predictive performance. Information important for prediction may lie in a direction with small variance. Depending on the purpose of the follow-up analysis, compare the original variables with the principal components.

To reduce differences due to regional size, all variables were defined as indicators that take population or the size of relevant facilities into account.

The figures and result tables in this example were constructed to illustrate how PCA is interpreted; they are not actual regional energy statistics.

For example, cooling demand is in kWh, natural gas use in MWh, and variability is a unitless index, so analyzing them in their original units would let the choice of units itself affect the results.

For this reason, when analyzing indicators measured in different units together, standardization, which sets each variable's mean to 0 and standard deviation to 1, is often used.

Here x_ij is the value of variable j for region i, x̄_j is the mean of variable j, and s_j is its standard deviation. Applying PCA to standardized data is effectively the same as finding the principal components from the correlation matrix.

If all variables are in the same unit and differences in variance are themselves important information for the study, you might choose PCA based on the covariance matrix. So standardization is not a step to apply automatically; decide based on the variables' units and the purpose of the analysis.

The score on the first principal component can be written as follows.

The coefficients w_j are the weights given to each original variable, chosen so that the variance explained by the component is as large as possible. In standard PCA, the component directions are constructed to be orthogonal to one another.

In matrix terms, the principal components can be found by an eigendecomposition of the covariance or correlation matrix, or by a singular value decomposition (SVD) of the standardized data matrix. The eigenvectors give the directions of the components, and the eigenvalues give the variance explained along each direction.

The important point here is that the computed principal components are new composite variables, different from the original variables. Whether PC1 can be called 'scale of energy consumption' should be judged after checking the formula and the loadings.

Below are the results of PCA on the six standardized variables.

Because six standardized variables were used, the eigenvalues sum to 6. For example, the explained variance ratio of PC1 is 2.76 ÷ 6 = 0.46, or 46%. Using PC1 and PC2 together explains 71% of the total variance of the original variables.

In this example the goal is to look at the overall regional pattern with two components, so suppose we chose PC1 and PC2. In a study where preserving information matters more, however, you could keep up to PC3, which brings the cumulative share to 83%.

Rules such as '70% cumulative is enough' or 'always keep components with eigenvalues of 1 or more' should never be used as absolute criteria.

The table below is a conceptually simplified example of loadings expressed as correlation coefficients between the original variables and the components.

This table is an example pattern to explain the meaning of the components. It is not presented as the result of computing the exact same matrix back from the eigenvalues in the previous section.

PC1 is strongly positively related to the overall scale of energy use, including electricity use, peak power demand and natural gas use. So for the purposes of the study, it can be interpreted as 'overall energy consumption intensity'.

On PC2, the loading for cooling demand is positive, and the loadings for heating demand and natural gas use are negative. So it can be interpreted as 'cooling-oriented vs. heating-oriented usage'.

The signs of principal components in PCA are arbitrary. Even if all the signs of a component come out reversed when the same analysis is run in a different program, they can represent mathematically the same direction. So do not attach a fixed substantive meaning to a positive value itself.

If loadings describe the variables, component scores describe the observed units, here the regions. For example, suppose we got the following scores.

A score of 0 means being near the center of that component axis in the standardized data. A component score of +1.80 does not mean electricity use is 1.8 times higher. And because PCA scores are close to unitless composite coordinates, it is more natural to interpret the regions' relative positions and patterns than the absolute values.

With PC1 on the horizontal axis and PC2 on the vertical axis, moving right can be read as higher overall energy consumption intensity, and moving up as more cooling-oriented. This reading holds as long as the component directions defined above are kept.

However, the variance shown in the 2D plot is 71% in this example. So even if two regions appear close together in the plot, they may differ in the remaining 29% of the information. Similarity between regions should not be judged fully from a single PC1–PC2 scatterplot.

A biplot, which also draws the original variables' loadings as arrows, lets you look at the regions' scores and the variable directions at the same time. Here the interpretation of arrow lengths and angles can depend on the biplot scaling used, so check which scaling it is.

PCA creates new components that preserve as much of the total variance of the observed variables as possible. EFA estimates latent factors that explain the common variance of several items, and treats item-specific variation and measurement error separately.

For example, to condense six energy indicators into two composite indicators, PCA is a suitable starting point. If instead you want to find out how many psychological latent factors 12 survey items actually measure, EFA may be more suitable.

Being able to give components meaningful names does not by itself make PCA a latent variable model.

But dimensionality reduction does not always improve the quality of a follow-up analysis. For example, if a variable that is important for predicting a particular regional risk does not contribute much to the total variance, keeping only the top components can lose predictive information.

Also, the meaning of the components depends on the variables, sample and standardization method included in the analysis. If you rerun PCA after adding new variables or regions, the loadings and component scores can change, so keep the same transformation basis when comparing results.

That PC1 and PC2 explain 71% of the total variance means that 71% of the variation in the six energy indicators is represented by the two axes. It does not mean a later regression will predict a target variable with 71% accuracy.

Do not leave out the standardization decision.

If variables in different units are not standardized, their units of measurement can strongly influence the component directions. Conversely, if differences in variance matter for the study, you may choose not to standardize.

Do not confuse loadings and scores.

Loadings show which variables are related to a component, and scores show where each observed unit lies on the component axis.

Do not interpret components as causes.

A region scoring high on PC1 only reflects a pattern of variation shared by several energy indicators; it does not mean a factor called PC1 caused the region's electricity consumption.

Check the data structure and outliers.

Because PCA is based on means, variances and covariances, it can be strongly affected by extreme observations. Rather than automatically deleting outliers, distinguish data errors from genuinely unusual regions and, if needed, consider alternatives such as robust PCA.

If you used the scores in a follow-up analysis, also record how the scores were computed and the transformation basis.

The example loading pattern and the eigenvalue table were constructed separately to explain the principle.

Because it is constructed so that, starting with the first component, each explains as much variance as possible in turn, it is useful for organizing and visualizing patterns among correlated variables.

In the analysis, you first decide on the variables' units and whether to standardize, then use the eigenvalues and explained variance ratios to choose how many components to keep. You can then interpret what each component means from its loadings and look at the relative positions of the observed units from the component scores.

However, PCA is not a method for confirming latent factors or testing causal relationships. And a high explained variance ratio does not guarantee follow-up predictive performance, so use the results in a way that fits the purpose of the study.