Mastering Regression Evaluation Metrics: RMSE, MAE, and R-squared

Previously in this course, we covered training the baseline linear model. Now that you have a model spitting out predictions, you need a way to determine if those numbers are actually good. In this lesson, we move beyond "it looks okay" and into the quantitative world of regression metrics.

When predicting continuous values like house prices or temperature, we measure performance by calculating the distance between our predicted values ($\hat{y}$) and the actual ground truth values ($y$).

Why We Use Multiple Regression Metrics

There is no "perfect" metric. A single number often hides the nature of your model's errors. We use a combination of metrics to understand the magnitude, direction, and explanatory power of our predictions.

1. Mean Absolute Error (MAE)

MAE is the simplest metric: it is the average of the absolute differences between predictions and actuals. If your model predicts a house price is $500k but it's actually $520k, the error is $20k.

Mathematically, it's the mean of $|y - \hat{y}|$. Because it uses absolute values, it doesn't penalize large errors more than small ones. It tells you, on average, how many units "off" your model is.

2. Root Mean Squared Error (RMSE)

RMSE is the standard for most regression tasks. We take the difference, square it, find the average, and then take the square root.

The squaring step is critical: it heavily penalizes large errors. If your model is consistently off by a small amount, your RMSE will be low. If your model is mostly accurate but makes a few massive "outlier" mistakes, your RMSE will spike.

3. R-Squared ($R^2$)

While MAE and RMSE tell you about the scale of your error, $R^2$ tells you about the quality of your fit. Often called the "Coefficient of Determination," it represents the proportion of variance in the target variable that your model explains.

An $R^2$ of 1.0 is a perfect model; an $R^2$ of 0.0 is equivalent to a model that simply guesses the average value of the target every single time.

Worked Example: Calculating Metrics with Scikit-Learn

In practice, you shouldn't calculate these by hand. Use scikit-learn to ensure precision and consistency.

PYTHON
import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

# Simulated data from our project
y_true = np.array([100, 150, 200, 250, 300])
y_pred = np.array([105, 145, 210, 240, 310])

# Calculate metrics
mae = mean_absolute_error(y_true, y_pred)
rmse = np.sqrt(mean_squared_error(y_true, y_pred))
r2 = r2_score(y_true, y_pred)

print(f"MAE: {mae:.2f}")
print(f"RMSE: {rmse:.2f}")
print(f"R-squared: {r2:.2f}")

In this example, the MAE tells us we are off by about 7 units on average. The RMSE is slightly higher, which alerts us that our errors are spread out.

Hands-on Exercise

Using the dataset you cleaned in project dataset initialization, load your test set predictions from your previous model.

1. Calculate the MAE and RMSE.
2. Interpret the $R^2$ value. If it's negative, don't panic—it just means your model is performing worse than a horizontal line representing the mean of your data.
3. Compare the MAE and RMSE. If RMSE is significantly larger than MAE, search your predictions for large outliers that might be skewing the results.

Common Pitfalls

• Ignoring the units: Always keep your MAE and RMSE in the context of the target variable. An RMSE of 500 is "good" for a house price model but "catastrophic" for a model predicting the number of people in a room.
• Over-relying on $R^2$: A high $R^2$ does not mean your model has zero error. It only means your model captures a large portion of the trend. Always check your error metrics (MAE/RMSE) alongside it.
• Sensitivity to scale: Remember that RMSE is sensitive to outliers. If your data has extreme values, consider scaling or capping them before training to prevent your metric from being dominated by a few bad rows.

Recap

Regression evaluation is about understanding the "distance" between your predictions and reality. Use MAE for an intuitive look at average error, RMSE to catch and penalize large mistakes, and R-squared to judge the overall explanatory power of your model. By mastering these regression metrics, you gain the ability to diagnose model performance objectively.

Up next: We will dive into the confusion matrix to see how we evaluate models that categorize data rather than predicting continuous numbers.