Feature Selection and Basic Filtering for Cleaner ML Models

Previously in this course, we covered handling missing and inconsistent data. While that ensured our data was complete, it didn't necessarily ensure it was useful. Having more columns isn't always better; in fact, feeding a model "noisy" or redundant data often leads to overfitting and slower training times.

In this lesson, we focus on feature selection and data filtering. Our goal is to reduce dimensionality by keeping only the variables that actively contribute to the prediction task, ensuring our models remain performant and interpretable.

Why Less Is Often More

In production, every feature you pass to a model carries a "cost." It increases the complexity of the model's hypothesis space, requires more memory, and can introduce noise that distracts the algorithm from the underlying patterns.

Think of this as a data-cleaning audit. Just as we use REST API field selection to minimize bandwidth in web services, we perform feature selection in ML to minimize the "cognitive load" on our model.

1. Filtering by Relevance

Sometimes, a dataset contains columns that are metadata—IDs, timestamps, or system logs—that have no predictive value for the target. If you’re predicting house prices, an Internal_System_ID is not a feature; it’s noise.

PYTHON
import pandas as pd

# Assume df is our loaded dataset
# Drop columns that are irrelevant to the prediction target
features_to_drop = [CE9178">'id', CE9178">'timestamp', CE9178">'internal_code']
df_clean = df.drop(columns=features_to_drop)

2. Identifying and Removing Highly Correlated Features

When two features are highly correlated, they provide redundant information. For example, if you have both Square_Footage and Room_Count in a dataset, they likely move in tandem. Keeping both can inflate the variance of your model coefficients.

We use the correlation matrix to spot these relationships:

PYTHON
# Calculate the correlation matrix
corr_matrix = df_clean.corr().abs()

# Create a mask to identify highly correlated pairs(e.g., > 0.85)
upper = corr_matrix.where(pd.np.triu(pd.np.ones(corr_matrix.shape), k=1).astype(bool))

# Find features with correlation > 0.85
to_drop = [column for column in upper.columns if any(upper[column] > 0.85)]

# Drop these redundant features
df_final = df_clean.drop(columns=to_drop)

3. Renaming for Clarity

Production-grade code requires readability. If your CSV comes with messy headers like col_001_v2, rename them early. This makes your code self-documenting and saves hours of debugging later.

PYTHON
df_final = df_final.rename(columns={
    CE9178">'sq_ft_total': CE9178">'square_footage',
    CE9178">'yr_built_2023': CE9178">'year_built'
})

Hands-on Exercise

Using the dataset you’ve been preparing, perform the following steps in your Jupyter Notebook:

1. Identify three columns that are clearly irrelevant to your target variable and drop them.
2. Generate a correlation matrix using df.corr().
3. Identify any two features with a correlation coefficient greater than 0.9. Drop one of them.
4. Rename your remaining columns to use standard snake_case naming.

Common Pitfalls

• Dropping the Target Variable: It sounds obvious, but I’ve seen many engineers accidentally drop the column they are trying to predict during a bulk drop() operation. Always check your columns after filtering.
• Assuming Correlation = Causation: Just because two features are correlated doesn't mean one causes the other. However, for the purpose of dimensionality reduction, we only care about the redundancy they create, not the underlying causality.
• Over-Filtering: Don't delete features just because they have a low correlation with the target. Some features interact with others to create predictive power. If you aren't sure, keep it—feature selection is an iterative process.

Recap

Feature selection is the art of removing the "dead weight" from your dataset. By filtering for relevance, removing redundant correlated features, and cleaning up your column names, you prepare your data for the modeling phase. We have now moved from raw data loading—which you mastered in Loading and Inspecting Datasets with Pandas—to creating a refined, production-ready input set.

Up next: We will perform the final audit on our data and save it to begin our project modeling phase.