Principal Component Regression (PCR): Simplifying Regression with PCA

34 minute read

Introduction: Regression in a Simplified Space

Imagine you’re trying to predict something complex, like customer satisfaction, based on many different factors – website clicks, time spent on site, number of purchases, customer reviews, and more. You might find that some of these factors are related to each other, making it hard to see the clear picture. Principal Component Regression (PCR) is a clever way to handle this complexity.

Think of PCR as a two-step process to make regression easier and more robust. First, it uses a technique called Principal Component Analysis (PCA) to simplify your input features. PCA is like finding the most important “directions” in your data. It combines your original features into a smaller set of new, uncorrelated features called principal components that capture most of the important information.

Second, instead of doing regression directly with your original, potentially messy features, PCR performs linear regression on these new, simplified principal components. It’s like doing regression in a cleaner, less cluttered space, which can lead to better predictions and a more stable model.

Real-world examples where PCR is useful:

Chemometrics and Spectroscopy: In chemistry and materials science, you might have spectra (like infrared or Raman spectra) with thousands of data points representing features. PCR is widely used to build regression models that predict chemical properties or concentrations from these complex spectra, by simplifying the spectral data first.
Image Analysis: When predicting something from images, the raw pixel data can be very high-dimensional and correlated. PCR can be applied to reduce the dimensionality of image features before using them in a regression model, making the model more efficient and robust.
Financial Forecasting: In finance, predicting stock prices or market movements can involve numerous economic indicators and market data. PCR can help reduce the dimensionality of these predictors and build more stable regression models.
Environmental Modeling: Predicting air quality or water pollution levels can involve many correlated environmental variables like temperature, humidity, wind speed, and concentrations of different pollutants. PCR can simplify these variables before building a regression model to forecast pollution levels.
Manufacturing Quality Control: In manufacturing, product quality might depend on many process parameters that are often inter-related. PCR can be used to reduce the number of variables used to predict quality, leading to a simpler and more robust quality control model.

Essentially, PCR is a go-to technique when you suspect that your input features are highly correlated and you want to reduce dimensionality and potentially improve the stability and interpretability of your regression model.

The Mathematics Behind Principal Component Regression

Let’s break down the mathematical steps involved in Principal Component Regression. It’s a combination of two key techniques: PCA and linear regression.

1. Principal Component Analysis (PCA): Dimensionality Reduction First

The first crucial step in PCR is to apply Principal Component Analysis (PCA) to your input features. PCA is a dimensionality reduction technique. (We have discussed PCA in detail in another blog post, if you want to review more about PCA, refer to that post). In brief, PCA does the following:

Standardization: It usually starts by standardizing your input features. This means transforming each feature to have a mean of 0 and a standard deviation of 1. This step is often important to ensure that features with larger scales don’t disproportionately influence PCA.

If we have a feature x, the standardized feature x’ is:
```
x' = (x - μ) / σ
```
where μ (mu) is the mean and σ (sigma) is the standard deviation of x.
Covariance Matrix: PCA calculates the covariance matrix of your standardized features. The covariance matrix shows how each pair of features varies together.
Eigenvalue Decomposition: PCA performs eigenvalue decomposition on the covariance matrix. This gives us two things:
- Eigenvectors: These are the principal components. They are new, uncorrelated features that are linear combinations of your original features.
- Eigenvalues: These represent the amount of variance (spread) in the data explained by each corresponding principal component.
If C is the covariance matrix, we find eigenvectors v and eigenvalues λ (lambda) such that:
```
C * v = λ * v
```
- v is an eigenvector (principal component direction).
- λ is the eigenvalue (variance explained).
Selecting Principal Components: PCA sorts the eigenvalues in descending order and selects the top ‘k’ eigenvectors (principal components) corresponding to the largest eigenvalues. These top ‘k’ principal components capture the most variance in your data. ‘k’ is usually much smaller than the original number of features, thus reducing dimensionality. Let’s denote the chosen principal components as PC₁, PC₂, …, PC_k.
Data Transformation: PCA transforms your original data by projecting it onto these chosen principal components. This means each original data point is now represented by its scores on the principal components:
```
PC_i = X * v_i
```
where:
- PC_i is the i-th principal component score for a data point.
- X is the original data point (represented as a row vector of feature values).
- v_i is the i-th eigenvector (principal component, represented as a column vector).

2. Linear Regression on Principal Components: The Second Step

After PCA, we have a new dataset consisting of the principal component scores (PC₁, PC₂, …, PC_k). The second step in PCR is to perform linear regression using these principal components as our new input features, and our original target variable ‘y’ remains the same.

We build a standard linear regression model of the form:

ŷ = β₀ + β₁ * PC₁ + β₂ * PC₂ + ... + β<sub>k</sub> * PC<sub>k</sub>

Where:

ŷ is the predicted value of ‘y’.
PC₁, PC₂, …, PC_k are the selected principal components scores.
β₀, β₁, β₂, …, β_k (beta values) are the regression coefficients that are learned during the linear regression step.
β₀ is the intercept.

The goal of this linear regression step is to find the best values for the coefficients (β₀, β₁, β₂, …, β_k) that minimize the error in predicting ‘y’ based on the principal component scores. Typically, Ordinary Least Squares (OLS) is used to perform this linear regression (as discussed in the OLS Regression blog post).

Putting it Together: PCR Algorithm Flow

In summary, the PCR algorithm combines PCA and linear regression in these steps:

Apply PCA to the input features ‘X’. Select ‘k’ principal components that capture a significant portion of the variance in ‘X’.
Transform the original feature data ‘X’ into principal component scores. This gives you a new feature matrix using PC₁, PC₂, …, PC_k as features.
Perform linear regression (OLS) using these principal component scores as predictors and the original target variable ‘y’ as the response.
The resulting linear regression model is your PCR model. It predicts ‘y’ based on a linear combination of the principal components.

Why PCR can be useful:

Handles Multicollinearity: PCA creates principal components that are uncorrelated with each other. Multicollinearity (high correlation between input features) can be a problem for standard linear regression, making coefficients unstable and hard to interpret. PCR addresses this by using uncorrelated principal components as predictors, thus mitigating multicollinearity issues in the regression step.
Dimensionality Reduction: By selecting a smaller number of principal components (‘k’ < original number of features), PCR reduces the dimensionality of the input space. This can simplify the model, improve computational efficiency, and sometimes improve generalization performance, especially when dealing with high-dimensional datasets.
Focus on Variance: PCA focuses on capturing the directions of maximum variance in the input features. By using principal components in regression, PCR prioritizes the dimensions of the data that contain the most information (in terms of variance).

Prerequisites and Preprocessing for Principal Component Regression

Before applying PCR, it’s important to understand the prerequisites and data preprocessing steps.

Assumptions of PCR:

PCR relies on the assumptions of both PCA and Linear Regression.

Linearity: PCR, being based on linear regression, assumes a linear relationship between the principal components and the target variable ‘y’. While PCA itself can handle non-linear data to some extent in terms of variance capture, the regression step is still linear in the principal component space. The overall effectiveness of PCR relies on the assumption that a linear model in the space of principal components is a reasonable approximation of the relationship between the original features and the target.
- Testing Linearity (in Principal Component Space - less direct): It’s less straightforward to directly test linearity in the principal component space compared to the original feature space. After performing PCR and getting predictions, you can examine residual plots (residuals vs. predicted values) to check for patterns that might suggest non-linearity in the overall model.
Relevance of Principal Components for Regression: PCR assumes that the principal components that capture the most variance in the input features are also the most relevant for predicting the target variable ‘y’. This is a crucial assumption, and it might not always hold true. It’s possible that components with lower variance (later principal components) might actually be more strongly related to ‘y’. If this assumption is significantly violated, PCR might discard important information by focusing only on top variance components.
Assumptions of Linear Regression (for the Regression Step): The linear regression step in PCR, which uses OLS, inherits the standard assumptions of linear regression (as discussed in the OLS Regression blog post). These include:
- Independence of Errors (Residuals).
- Homoscedasticity (Constant Variance of Errors).
- Normality of Errors (Residuals - less critical for large samples, more for statistical inference).
- No or Little Multicollinearity among the principal components. However, since principal components are designed to be uncorrelated, multicollinearity is generally not an issue among the PCs. Multicollinearity in the original features is addressed by PCA.

Testing Assumptions:

Linearity (Indirect): As mentioned, direct linearity tests are less common in PC space. Focus on residual analysis after fitting the PCR model.
Relevance of Top Variance Components: This assumption is harder to directly test a priori. It’s more of a judgment based on domain knowledge and evaluating PCR’s performance compared to other regression methods. If PCR performs poorly, it might indicate this assumption is not well met. You can also try varying the number of principal components to see how performance changes.
Linear Regression Assumptions (for Regression Step): Test the standard linear regression assumptions (independence, homoscedasticity, normality of residuals) after fitting the PCR model. These tests are performed on the residuals of the PCR model (predicted vs. actual ‘y’ values), just like you would for standard linear regression. Use residual plots, Durbin-Watson test, Breusch-Pagan test, normality tests (histograms, Q-Q plots, Shapiro-Wilk test) as described in the OLS Regression blog post.

Python Libraries Required for Implementation:

numpy: For numerical computations, array and matrix operations.
pandas: For data manipulation and analysis, using DataFrames.
scikit-learn (sklearn): Essential library:
- sklearn.decomposition.PCA for Principal Component Analysis.
- sklearn.linear_model.LinearRegression for Ordinary Least Squares Regression (used in the second step of PCR).
- sklearn.preprocessing.StandardScaler for data standardization.
- sklearn.model_selection.train_test_split for data splitting.
- sklearn.metrics for regression evaluation metrics (mean_squared_error, r2_score).
matplotlib and seaborn: For data visualization (scatter plots, residual plots, explained variance ratio plots, etc.).
statsmodels: Can be used for more detailed statistical analysis of the linear regression step in PCR, if desired (though typically sklearn.linear_model.LinearRegression is sufficient for PCR itself).

Data Preprocessing: Scaling is Crucial for PCR

Data Scaling (Standardization) is absolutely essential as a preprocessing step for Principal Component Regression. Let’s understand why it’s even more critical for PCR than for standard linear regression alone.

PCA’s Scale Sensitivity: As discussed in the PCA blog post, Principal Component Analysis is very sensitive to the scale of features. If features are on different scales, features with larger scales will dominate the variance calculation in PCA, and thus disproportionately influence the principal components. Features with smaller scales might get overshadowed, even if they contain important information for regression.

Example: Imagine you have features like “income” (ranging from $20,000 to $200,000) and “age” (ranging from 20 to 80). Income has a much larger scale and variance. If you run PCA without scaling, the first principal component will likely be heavily influenced by income, simply because it has a larger variance. Age might contribute very little to the first few principal components, even if age is relevant for predicting your target variable.

Standardization (making each feature have mean 0 and standard deviation 1) is crucial before PCA to put all features on a comparable scale. This ensures that PCA identifies principal components based on the underlying relationships in the data, not just driven by differences in feature scales.
Fairness in Variance Capture: Standardization ensures that each feature contributes to the variance calculation in PCA equally, regardless of its original scale. PCA then finds principal components that capture the maximum variance across all features after they are on a common scale. This leads to principal components that are more representative of the overall data structure, not just dominated by large-scale features.
Impact on Regression Step: If PCA is scale-biased due to unscaled features, the principal components you select will be skewed towards capturing variance driven by large-scale features. Consequently, when you perform linear regression on these biased principal components, the regression model might also be biased and underperform, especially for features that were originally on smaller scales but are actually important for prediction.

When Can Scaling Be (Potentially) Ignored for PCR? - Almost Never.

In almost all practical applications of PCR, you should always standardize your features before applying PCA. There are very few situations where skipping scaling would be advisable. Here are the extremely rare and theoretical cases where you might consider it (but even then, scaling is generally safer):

Features Already on Truly Comparable Scales and Variances (Extremely Rare): If you have absolute certainty and strong domain knowledge that all your features are already measured on exactly the same scale, have inherently similar variances and units, and scaling is fundamentally not meaningful in your domain, then you might theoretically consider skipping standardization. However, this scenario is exceptionally rare in real-world datasets. It’s almost always better to standardize.
If You Intentionally Want to Bias PCA by Feature Scale: In some very specific and unusual situations, you might have a deliberate reason to let features with larger scales dominate PCA. For example, if you have very strong prior knowledge that features with larger scales are inherently more important for your problem, and you want PCA to prioritize variance in those features directly due to their scale, then maybe you could skip scaling. But even in such cases, it’s usually better to first standardize and then perhaps consider weighting features based on domain knowledge in some other way if needed, rather than relying on the uncontrolled effect of scale on PCA.

Examples Where Scaling is Absolutely Essential for PCR:

Spectroscopic Data (Chemometrics): In spectroscopic applications (IR, Raman, NMR, etc.), spectral data points can have very different ranges of intensities and noise levels. Standardization is a mandatory preprocessing step before PCA in PCR for chemometrics to ensure that PCA captures meaningful spectral variations, not just scale-driven noise.
Financial Time Series Data: Financial indicators, stock prices, volumes, etc., are often measured on very different scales and units. Standardization is crucial before applying PCR for financial forecasting to prevent scale dominance issues.
Environmental Datasets: Environmental variables (temperature, rainfall, pollutant concentrations, wind speed, etc.) are measured in different units and ranges. Standardization is essential for PCR to work effectively in environmental modeling.
Image Data (Pixel Intensities - although other preprocessing also common): While pixel values might be in a similar range (e.g., 0-255), standardization (or other forms of scaling/normalization common in image processing) can still be beneficial before applying PCA to image features for regression tasks, especially if dealing with images with varying overall brightness or contrast.

In summary: Always, always standardize your features before applying Principal Component Regression. Standardization (using StandardScaler in scikit-learn) is a non-negotiable preprocessing step for PCR in virtually all practical scenarios to ensure that PCA works correctly and that your PCR model is not biased by feature scales.

Implementation Example: House Price Prediction with Principal Component Regression

Let’s implement PCR in Python to predict house prices using dummy data with multiple features. We will demonstrate how PCR addresses multicollinearity and performs regression in a reduced dimension space.

1. Dummy Data Creation (with multicollinearity):

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
import matplotlib.pyplot as plt

# Set seed for reproducibility
np.random.seed(42)

# Generate dummy data with multicollinearity
n_samples = 100
square_footage = np.random.randint(1000, 3000, n_samples)
bedrooms = np.random.randint(2, 6, n_samples)
bathrooms = 0.75 * bedrooms + np.random.normal(0, 0.5, n_samples) # Bathrooms correlated with bedrooms (multicollinearity)
location_index = np.random.randint(1, 10, n_samples)
age = np.random.randint(5, 50, n_samples)

# Price dependent on sqft, location, bedrooms, bathrooms, somewhat less on age
price = 200000 + 150 * square_footage + 30000 * location_index + 10000 * bedrooms + 8000 * bathrooms - 500 * age + np.random.normal(0, 30000, n_samples)

# Create Pandas DataFrame
data = pd.DataFrame({
    'SquareFootage': square_footage,
    'Bedrooms': bedrooms,
    'Bathrooms': bathrooms,
    'LocationIndex': location_index,
    'Age': age,
    'Price': price
})

# Split data into training and testing sets
X = data[['SquareFootage', 'Bedrooms', 'Bathrooms', 'LocationIndex', 'Age']] # Features
y = data['Price'] # Target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

feature_names = X.columns # Store feature names

print("Training data shape:", X_train.shape, y_train.shape)
print("Testing data shape:",  X_test.shape,  y_test.shape)
print("\nFirst 5 rows of training data:\n", X_train.head())

This code creates dummy house price data with 5 features. ‘Bathrooms’ is intentionally made correlated with ‘Bedrooms’ to introduce multicollinearity. ‘Price’ is generated based on a linear relationship with these features plus some noise.

2. Data Scaling (Standardization - Mandatory for PCR):

# Scale features using StandardScaler (mandatory for PCR)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) # Fit on training data and transform
X_test_scaled = scaler.transform(X_test)       # Transform test data using fitted scaler

X_train_scaled_df = pd.DataFrame(X_train_scaled, columns=feature_names, index=X_train.index) # for easier viewing
print("\nScaled Training Data (first 5 rows):\n", X_train_scaled_df.head())

Scale features using StandardScaler. This step is crucial for PCR.

3. Apply PCA and Transform Data:

# Apply PCA to scaled training data
pca = PCA() # Instantiate PCA (by default, keeps all components)
pca.fit(X_train_scaled)

# Determine explained variance ratio
explained_variance_ratio = pca.explained_variance_ratio_
cumulative_variance_ratio = np.cumsum(explained_variance_ratio)

print("Explained Variance Ratio per Principal Component:", explained_variance_ratio)
print("Cumulative Explained Variance Ratio:", cumulative_variance_ratio)

# Choose number of principal components to keep (e.g., based on explained variance)
n_components = 3 # Example: Keep 3 components (you would tune this in practice)
pca = PCA(n_components=n_components) # Instantiate PCA again with chosen n_components
pca.fit(X_train_scaled) # Fit PCA again

# Transform training and test data to principal component space
X_train_pca = pca.transform(X_train_scaled)
X_test_pca = pca.transform(X_test_scaled)

print("\nTransformed Training Data (PCA - first 5 rows, first 3 components):\n", X_train_pca[:5,:]) # Show first 5 rows and first 3 columns

We apply PCA using sklearn.decomposition.PCA. First, we fit PCA on the scaled training data to get the principal components and explained variance ratios for all components. We examine the explained variance ratios and cumulative variance ratios to decide how many components to keep (e.g., based on capturing a certain percentage of variance). In this example, we choose to keep n_components = 3 (you would tune this in practice). We then refit PCA with the chosen number of components and transform both training and test data to the principal component space.

4. Train Linear Regression Model on Principal Components (PCR Step):

# Train Linear Regression model on principal component scores
pcr_model = LinearRegression() # Instantiate Linear Regression model
pcr_model.fit(X_train_pca, y_train) # Train Linear Regression on PCA-transformed training data and original target

# Get coefficients and intercept of PCR model (coefficients are now for principal components, not original features)
pcr_coefficients = pcr_model.coef_
pcr_intercept = pcr_model.intercept_

print("\nPCR Model Coefficients (for Principal Components):", pcr_coefficients)
print("PCR Model Intercept:", pcr_intercept)

We instantiate a LinearRegression model and train it using the PCA-transformed training data (X_train_pca) as features and the original target y_train. The coefficients we get are now associated with the principal components, not the original features.

5. Evaluate PCR Model:

# Make predictions on test set using PCR model
y_pred_test_pcr = pcr_model.predict(X_test_pca)

# Evaluate PCR model performance
mse_pcr = mean_squared_error(y_test, y_pred_test_pcr)
r2_pcr = r2_score(y_test, y_pred_test_pcr)

print(f"\nPCR Regression - Test Set MSE: {mse_pcr:.2f}")
print(f"PCR Regression - Test Set R-squared: {r2_pcr:.4f}")


# For comparison, also train and evaluate a standard Linear Regression model directly on the *original scaled features* (without PCA):
linear_model_original_features = LinearRegression()
linear_model_original_features.fit(X_train_scaled, y_train)
y_pred_test_original_linear = linear_model_original_features.predict(X_test_scaled)
mse_original_linear = mean_squared_error(y_test, y_pred_test_original_linear)
r2_original_linear = r2_score(y_test, y_pred_test_original_linear)

print(f"\nStandard Linear Regression (on original scaled features) - Test Set MSE: {mse_original_linear:.2f}")
print(f"Standard Linear Regression (on original scaled features) - Test Set R-squared: {r2_original_linear:.4f}")

We evaluate the PCR model on the test set using MSE and R-squared. For comparison, we also train and evaluate a standard Linear Regression model directly on the original scaled features (without PCA dimensionality reduction). This helps to see if PCR provides any benefit in terms of performance or simplicity compared to standard linear regression in this example.

Understanding Output - Explained Variance Ratio:

The output includes “Explained Variance Ratio per Principal Component:” and “Cumulative Explained Variance Ratio:”. These are important for understanding PCA and choosing n_components.

Explained Variance Ratio per Principal Component: This array (e.g., [v1, v2, v3, v4, v5]) shows the proportion of the total variance in your scaled feature data that is explained by each principal component, in order. v1 is for PC₁, v2 for PC₂, and so on. Larger values mean the component captures more variance.
Cumulative Explained Variance Ratio: This array (e.g., [c1, c2, c3, c4, c5]) shows the cumulative proportion of variance explained as you include more principal components. c1 = v1, c2 = v1 + v2, c3 = v1 + v2 + v3, and so on. c5 (if you have 5 features and keep all 5 components) will usually be 1.0 (or very close to 1.0), meaning all variance is explained when you use all components.

By looking at these ratios, you can decide how many principal components to keep. A common approach is to choose enough components to capture a reasonably high percentage of the total variance (e.g., 80%, 90%, 95%). In the example, you would examine these ratios and then set n_components accordingly in the subsequent PCA step.

Saving and Loading the PCR Pipeline Components:

To save and load a complete PCR pipeline, you need to save the StandardScaler, the PCA model, and the LinearRegression model:

import joblib

# Save the StandardScaler, PCA model, and PCR Linear Regression model
joblib.dump(scaler, 'scaler_pcr.joblib')
joblib.dump(pca, 'pca_model_pcr.joblib')
joblib.dump(pcr_model, 'pcr_regression_model.joblib')

print("\nPCR pipeline components saved to 'scaler_pcr.joblib', 'pca_model_pcr.joblib', and 'pcr_regression_model.joblib'")

# To load them later:
loaded_scaler = joblib.load('scaler_pcr.joblib')
loaded_pca_model = joblib.load('pca_model_pcr.joblib')
loaded_pcr_model = joblib.load('pcr_regression_model.joblib')

# Now you can use loaded_scaler, loaded_pca_model, and loaded_pcr_model to preprocess new data, transform it with PCA, and make predictions using the PCR model.

We save each component of the PCR pipeline (StandardScaler, PCA, and LinearRegression model) separately using joblib. When loading, you’ll need to load all three components to reconstruct the full PCR prediction pipeline.

Post-Processing: Interpreting PCR Results and Component Analysis

Interpreting PCR Coefficients (Coefficients of Principal Components):

In PCR, the coefficients you get from the final linear regression model are associated with the principal components, not with the original features directly. This is a key difference from standard linear regression.

PCR Coefficients (β₁, β₂, …, β_k in ŷ = β₀ + β₁PC₁ + β₂PC₂ + … + β_kPC_k):
- These coefficients tell you the linear relationship between each principal component and the target variable ‘y’.
- Magnitude: The magnitude of a coefficient β_i reflects the strength of the relationship between the i-th principal component (PC_i) and ‘y’. Larger magnitudes mean PC_i has a greater influence on the predicted ‘y’.
- Sign: The sign (+ or -) indicates the direction of the relationship. Positive β_i means that as the score on PC_i increases, ‘y’ tends to increase. Negative β_i means as the score on PC_i increases, ‘y’ tends to decrease.
- Units: The PCR coefficients are unitless because principal component scores are also unitless (derived from standardized features).
Intercept (β₀): The intercept has the same interpretation as in standard linear regression – it’s the predicted value of ‘y’ when all principal component scores are zero.

Relating PCR Coefficients Back to Original Features (Loadings Analysis):

While PCR coefficients are for principal components, you can indirectly understand how the original features contribute to the PCR model by examining the PCA loadings.

PCA Loadings: The pca.components_ attribute in scikit-learn PCA gives you the eigenvectors (principal components). Each row of pca.components_ represents a principal component, and each column corresponds to an original feature. The values in this array are the loadings. A loading value for feature ‘j’ in principal component ‘i’ (e.g., pca.components_[i, j]) indicates how much feature ‘j’ contributes to that principal component PC_i.
- Magnitude of Loadings: Larger absolute loading values indicate a stronger contribution of that original feature to the principal component.
- Sign of Loadings: The sign of a loading indicates the direction of the feature’s contribution. Positive loadings mean the feature positively contributes to the component; negative loadings mean it negatively contributes.

Steps to Relate PCR Coefficients and Loadings for Interpretation:

Get PCR Coefficients (β₁, β₂, …, β_k) from pcr_model.coef_.
Get PCA Loadings (pca.components_) from the fitted PCA model.
To understand the combined effect of original features, you can think of ‘weighting’ the loadings by the PCR coefficients. For each original feature ‘j’, and for each principal component ‘i’, you have:
- PCR coefficient for PC_i: β_i
- Loading of feature ‘j’ on PC_i: pca.components_[i, j]
The product β_i * pca.components_[i, j] represents the contribution of original feature ‘j’ through principal component ‘i’ to the prediction.

Calculate Feature “Importance” Scores (Indirect and Approximate): To get a rough, indirect sense of feature “importance” in the PCR model, you can sum up these contributions across all principal components for each original feature. For each original feature ‘j’:

Feature_Importance_Score_j =  ∑_{i=1}^{k} |β_i * pca.components_[i, j]|

where:

∑ is the sum over all principal components i=1 to k.

denotes absolute value. We use absolute value because we are interested in magnitude of contribution, regardless of direction.

This score is a heuristic and approximate indicator of feature importance in PCR. It is not a formal feature importance measure in the same way as in tree-based models, but it can provide some insights into how original features are indirectly influencing predictions through the principal components.

Example (Conceptual - Illustrative, not exact code):

# (Assuming you have trained PCR model: pcr_model, pca, feature_names)

pcr_coefficients = pcr_model.coef_
pca_loadings = pca.components_

feature_importance_scores = {}

for j, feature_name in enumerate(feature_names): # Iterate through original features
    importance_score = 0
    for i in range(pca.n_components_): # Iterate through principal components
        contribution = pcr_coefficients[i] * pca_loadings[i, j]
        importance_score += abs(contribution) # Sum of absolute contributions
    feature_importance_scores[feature_name] = importance_score

# Create DataFrame to display feature importance scores
feature_importance_df = pd.DataFrame(list(feature_importance_scores.items()), columns=['Feature', 'PCR_Importance_Score'])
feature_importance_df = feature_importance_df.sort_values(by='PCR_Importance_Score', ascending=False)

print("\nPCR Feature Importance Scores (Approximate, based on loadings and coefficients):\n", feature_importance_df)

This conceptual code shows how you might calculate and visualize these approximate feature importance scores based on PCR coefficients and PCA loadings. Remember that this is an indirect and heuristic approach to feature importance in PCR.

Limitations of PCR Interpretation:

Indirect Relationship: PCR coefficients are for principal components, which are combinations of original features. Interpretation of coefficients and feature importance is less direct and intuitive than in standard linear regression with original features.
Combined Effect: Feature importance scores are approximate and represent a combined influence of a feature through all principal components. It’s not always easy to disentangle the specific contribution of each original feature.
Focus on Variance, Not Necessarily Relevance to Target: PCA selects components based on variance in features, not directly based on correlation with the target variable. While PCR uses these components in regression, the selected components and thus the interpretation are still primarily driven by feature variance structure, which might not perfectly align with relevance to the prediction task.

Despite these limitations, analyzing PCR coefficients in conjunction with PCA loadings can provide valuable, albeit indirect, insights into the relationships learned by the PCR model and the relative influence of original features.

Hyperparameter Tuning in Principal Component Regression (PCR)

The main hyperparameter to tune in PCR is the number of principal components to use in the regression step. This is typically controlled by the n_components parameter in sklearn.decomposition.PCA.

Hyperparameter: n_components (Number of Principal Components)

Effect: n_components determines how many principal components (from PCA) are used as features in the linear regression step of PCR. It controls the level of dimensionality reduction applied before regression.
- Small n_components: Aggressive dimensionality reduction. Only a few top principal components (capturing most variance) are used. Simpler regression model (fewer predictors), potentially more robust to overfitting, but might lose important information if too few components are kept, leading to underfitting and lower predictive performance if later principal components are also relevant for ‘y’.
- Large n_components (Up to the original number of features): Less dimensionality reduction (or no reduction if you keep all components, i.e., n_components = number of original features). More complex regression model (more predictors), retains more variance from the original features, might capture more subtle relationships if later components are relevant, but higher risk of overfitting, especially if the dataset is not very large or features are noisy. If you keep all components, PCR becomes mathematically equivalent to standard linear regression using the original features (assuming OLS is used in both cases).
Tuning n_components: Choosing the optimal n_components is crucial for PCR performance. You want to find a value that balances dimensionality reduction (to address multicollinearity and potentially overfitting) and retention of relevant information for prediction.

Hyperparameter Tuning Methods for n_components:

Cross-Validation (k-fold cross-validation is standard): The most reliable method to tune n_components.
- Process:
  - Choose a range of n_components values to try (e.g., from 1 up to the number of original features).
  - For each n_components value:
    - Use k-fold cross-validation (e.g., 5-fold or 10-fold) on your training data.
    - Within each cross-validation fold:
      1. Perform PCA with the current n_components on the training folds within this CV split.
      2. Transform the training and validation folds to principal component space.
      3. Train a Linear Regression model on the PCA-transformed training fold and evaluate its performance (e.g., MSE) on the PCA-transformed validation fold.
    - Average the performance metrics across all ‘k’ folds. This gives you an estimate of the validation performance for the chosen n_components.
  - Select the n_components value that yields the best average validation performance (e.g., lowest average MSE, highest average R-squared).
Validation Set Approach: Simpler but less robust than cross-validation. Split your training data into a training set and a validation set. Try different n_components values, train PCR models on the training set for each, evaluate on the validation set, and choose the n_components that gives the best validation performance.

Python Implementation - Hyperparameter Tuning using Cross-Validation (with sklearn):

from sklearn.model_selection import GridSearchCV, KFold
from sklearn.pipeline import Pipeline # Important for combining PCA and Regression steps in CV
from sklearn.decomposition import PCA
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler

# Create a pipeline combining StandardScaler, PCA, and LinearRegression
pcr_pipeline = Pipeline([
    ('scaler', StandardScaler()), # Step 1: Scaling
    ('pca', PCA()),           # Step 2: PCA (n_components will be tuned)
    ('linear_regression', LinearRegression()) # Step 3: Linear Regression
])

# Define the parameter grid to search for n_components
param_grid_pcr = {
    'pca__n_components': range(1, X_train.shape[1] + 1) # Try n_components from 1 to number of features
}

# Set up K-fold cross-validation
kf = KFold(n_splits=5, shuffle=True, random_state=42) # 5-fold CV

# Set up GridSearchCV with the PCR pipeline and parameter grid
grid_search_pcr = GridSearchCV(pcr_pipeline, param_grid_pcr, scoring='neg_mean_squared_error', cv=kf, return_train_score=False)
# scoring='neg_mean_squared_error' for GridSearchCV to minimize MSE (it maximizes score)

grid_search_pcr.fit(X_train, y_train) # Fit GridSearchCV on original *unscaled* training data and target - Pipeline handles scaling internally

# Best n_components found by cross-validation
best_n_components = grid_search_pcr.best_params_['pca__n_components']
print(f"\nBest n_components found by Cross-Validation: {best_n_components}")

# Best PCR model (trained with best n_components)
best_pcr_model_pipeline = grid_search_pcr.best_estimator_ # Best estimator is the Pipeline with best n_components

# Evaluate best model on test set
y_pred_test_best_pcr = best_pcr_model_pipeline.predict(X_test) # Predict on test data using best pipeline
mse_best_pcr = mean_squared_error(y_test, y_pred_test_best_pcr)
r2_best_pcr = r2_score(y_test, y_pred_test_best_pcr)

print(f"Best PCR Model - Test Set MSE: {mse_best_pcr:.2f}")
print(f"Best PCR Model - Test Set R-squared: {r2_best_pcr:.4f}")


# Access the PCA and LinearRegression models from the best pipeline, if needed for inspection:
best_pca_model_tuned = best_pcr_model_pipeline.named_steps['pca']
best_linear_regression_model_tuned = best_pcr_model_pipeline.named_steps['linear_regression']

This code uses GridSearchCV and Pipeline from sklearn to automate the cross-validation process for tuning n_components in PCR. We create a Pipeline to combine StandardScaler, PCA, and LinearRegression into a single model. GridSearchCV then searches through the n_components values, performing cross-validation for each value and selecting the n_components that minimizes the validation MSE. The best_n_components and best_pcr_model_pipeline (which is the entire tuned PCR pipeline) are extracted, and the best model is evaluated on the test set.

Accuracy Metrics for Principal Component Regression

The accuracy metrics used to evaluate PCR Regression are the same standard regression metrics we’ve discussed for other regression algorithms:

Common Regression Accuracy Metrics (Recap):

Mean Squared Error (MSE): Average of squared errors. Lower is better.
```
MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2
```
Root Mean Squared Error (RMSE): Square root of MSE. Lower is better. More interpretable units.
```
RMSE = \sqrt{MSE} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}
```
Mean Absolute Error (MAE): Average of absolute errors. Lower is better. Robust to outliers.
```
MAE = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|
```

R-squared (R²): Coefficient of Determination. Variance explained by the model. Higher (closer to 1) is better.

R^2 = 1 - \frac{SS_{res}}{SS_{tot}} = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2}

Using Metrics for PCR Evaluation:

Primary Metrics: MSE and RMSE are commonly used to evaluate PCR because PCR’s objective is often to minimize prediction error. RMSE is frequently preferred for its interpretability in the original units of the target variable.
R-squared: R-squared provides a measure of the proportion of variance in the target variable explained by the PCR model. It’s useful for comparing PCR to other regression models or assessing the overall goodness of fit in terms of variance explained.
MAE: MAE can be used if you want a metric that’s less sensitive to outliers in your dataset.

When reporting PCR performance, it’s good practice to include at least MSE, RMSE, and R-squared values on a held-out test set or from cross-validation to provide a comprehensive view of the model’s predictive accuracy.

Model Productionizing Principal Component Regression (PCR)

Productionizing a PCR model involves deploying the entire pipeline, including scaling, PCA transformation, and the linear regression model. Here are productionization steps for different environments:

1. Local Testing and Deployment (Python Script/Application):

Python Script Deployment: Deploy PCR as a Python script for local use or batch prediction jobs.

Load Pipeline Components: Load the saved scaler_pcr.joblib, pca_model_pcr.joblib, and pcr_regression_model.joblib files.
Define Prediction Function: Create a Python function that takes new data as input, applies scaling using the loaded scaler, transforms the scaled data using the loaded pca_model, and then makes predictions using the loaded pcr_regression_model.
Load New Data: Load new data for prediction.
Preprocess Data and Make Predictions: Use the prediction function to preprocess and predict on the new data.
Output Results: Output the predictions (save to file, print to console, etc.).

Code Snippet (Conceptual Local Deployment):

import joblib
import pandas as pd
import numpy as np

# Load saved PCR pipeline components
loaded_scaler = joblib.load('scaler_pcr.joblib')
loaded_pca_model = joblib.load('pca_model_pcr.joblib')
loaded_pcr_model = joblib.load('pcr_regression_model.joblib')

def predict_house_price_pcr(input_data_df): # Input data as DataFrame (with original features)
    scaled_input_data = loaded_scaler.transform(input_data_df) # Scale using loaded scaler
    pca_transformed_data = loaded_pca_model.transform(scaled_input_data) # Transform to PC space using loaded PCA model
    predicted_prices = loaded_pcr_model.predict(pca_transformed_data) # Predict using loaded PCR model
    return predicted_prices

# Example usage with new house data (original features)
new_house_data = pd.DataFrame({
    'SquareFootage': [3100, 1700],
    'Bedrooms': [4, 2],
    'Bathrooms': [3, 1.5],
    'LocationIndex': [9, 5],
    'Age': [15, 60]
})
predicted_prices_new = predict_house_price_pcr(new_house_data)

for i in range(len(new_house_data)):
    sqft = new_house_data['SquareFootage'].iloc[i]
    predicted_price = predicted_prices_new[i]
    print(f"Predicted price for {sqft} sq ft house: ${predicted_price:,.2f}")

# ... (Further actions) ...

Application Integration: Integrate the prediction function into a larger application.

2. On-Premise and Cloud Deployment (API for Real-time Predictions):

For applications needing real-time predictions from PCR, deploy as an API:

API Framework (Flask, FastAPI): Use a Python framework to create a web API.
API Endpoint: Define an API endpoint (e.g., /predict_house_price_pcr) to receive input data (house features).
Prediction Logic in API Endpoint: Within the API function:
1. Load the saved StandardScaler, PCA model, and PCR regression model.
2. Preprocess input data from the API request using the loaded scaler.
3. Transform preprocessed data to principal component space using the loaded PCA model.
4. Make predictions using the loaded PCR regression model on the PCA-transformed data.
5. Return predictions in the API response (JSON).
Server Deployment: Deploy the API application on a server (on-premise or cloud VM).
Cloud ML Platforms: Cloud platforms (AWS SageMaker, Azure ML, Google AI Platform) simplify deployment. Package your pipeline, deploy it using cloud ML services, and they handle API endpoint setup, scaling, monitoring, etc.
Serverless Functions: For event-driven predictions or lightweight APIs, serverless functions can be efficient.

Productionization Considerations Specific to PCR:

Pipeline Integrity: Ensure that in production, you apply the exact same preprocessing steps (scaling with the fitted scaler, PCA transformation with the fitted PCA model) as you did during training before making predictions. Use the saved scaler, PCA model, and regression model consistently.
Number of Components: Document the chosen number of principal components (n_components) used in your production PCR model. This is a key parameter for model understanding and maintenance.
Model Updates and Retraining: Like any ML model, PCR may need periodic retraining as data distributions change. Monitor model performance and retrain as needed, potentially revisiting the hyperparameter tuning for n_components if retraining.

Conclusion: Principal Component Regression - Addressing Multicollinearity and Simplifying Complexity

Principal Component Regression (PCR) is a valuable technique for linear regression, especially when dealing with multicollinearity among predictors and when dimensionality reduction is desired. It provides a way to build more stable and potentially simpler regression models by leveraging Principal Component Analysis.

Real-World Problem Solving with PCR:

Handling Multicollinearity Effectively: PCR is particularly useful when you have multicollinearity in your input features, as it addresses this issue by performing regression in the space of uncorrelated principal components. This leads to more stable and reliable regression coefficients compared to standard OLS in multicollinear scenarios.
Dimensionality Reduction for Regression: PCR reduces the dimensionality of the input feature space by using only a subset of principal components. This can simplify models, improve computational efficiency, and sometimes enhance generalization performance, especially in high-dimensional datasets.
Feature Extraction for Linear Models: PCA in PCR acts as a feature extraction step, creating new composite features (principal components) that capture the most variance in the original features. Linear regression is then performed on these extracted features.

Limitations and Alternatives:

Assumption: Variance = Relevance (Not Always True): A key assumption of PCR is that the principal components capturing the most variance are also the most relevant for predicting the target variable. This is not always the case. Components with lower variance could sometimes be more strongly related to ‘y’. If this assumption is significantly violated, PCR might discard important information by focusing only on top variance components and may not achieve optimal predictive performance.
Indirect Feature Interpretation: Interpretation of PCR coefficients and feature importance is less direct than in standard linear regression because coefficients are for principal components, which are combinations of original features. While you can analyze loadings to understand feature contributions indirectly, it’s less straightforward.
Alternatives for Feature Selection: If feature selection is the primary goal (identifying and using only the most important original features), techniques like Lasso Regression or tree-based feature importance methods might be more directly suitable than PCR, which primarily focuses on dimensionality reduction through variance capture rather than explicit feature selection in the original feature space.

Optimized and Newer Algorithms/Techniques:

Partial Least Squares Regression (PLSR): PLSR is a related technique that is often considered an improvement over PCR, particularly when the assumption that top variance components are most relevant to the target is not strongly met. PLSR, unlike PCR, considers the target variable ‘y’ during the component extraction process, aiming to find components that are most predictive of ‘y’, not just those with highest variance in ‘X’. PLSR often performs better than PCR in regression tasks because it directly focuses on components relevant to prediction.
Regularized Linear Regression (Ridge, Lasso, Elastic Net): Regularization techniques can also address multicollinearity and overfitting in linear regression without dimensionality reduction. Ridge Regression is particularly effective for handling multicollinearity by shrinking coefficients, and Lasso Regression can perform feature selection.

PCR remains a valuable technique, especially for addressing multicollinearity and dimensionality reduction in linear regression scenarios, and as a foundation for understanding more advanced regression methods like PLSR.

References

“Principal Component Regression” by Edward J. Dudewicz and Vidya S. Taneja (1994): A research paper discussing Principal Component Regression in detail.
“The Elements of Statistical Learning” by Trevor Hastie, Robert Tibshirani, and Jerome Friedman (2009): A comprehensive textbook with a chapter on Principal Components Regression and related dimensionality reduction techniques for regression. https://web.stanford.edu/~hastie/ElemStatLearn/
“Applied Predictive Modeling” by Max Kuhn and Kjell Johnson (2013): A practical guide to predictive modeling, with a chapter comparing and contrasting Principal Component Regression and Partial Least Squares Regression.
Scikit-learn Documentation for PCA: https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html
Scikit-learn Documentation for Linear Regression: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html

Share on

Twitter Facebook LinkedIn