Forecasting is an exercise in managing uncertainty. To improve, we must first learn to measure exactly how far off our predictions are.
1The Distance of Error
MAE (Mean Absolute Error) is the simplest measure of error; it tells you the average 'raw' distance from the truth. RMSE (Root Mean Square Error), however, squares the errors before averaging them. This makes RMSE much more sensitive to Outliers. If your business loses a massive amount of money on a single large forecasting mistake (e.g., running out of inventory), you should optimize for RMSE to force the model to avoid those 'big misses.'
2Relative Performance (MAPE)
MAPE (Mean Absolute Percentage Error) is the most common metric for communicating with non-technical stakeholders. Knowing that your error is '50 units' is meaningless without context; knowing that your error is '5%' is immediately understandable. However, MAPE has a weakness: it can't handle zero values in the actual data, as you can't divide by zero. In such cases, we often use sMAPE (Symmetric MAPE).
3Detecting Bias
Beyond the magnitude of error, we must look for Bias. If your model consistently predicts values that are *higher* than reality, it has a Positive Bias. If it predicts *lower*, it has a Negative Bias. We measure this using the Mean Error (ME). A good model should have balanced errors that average out to zero over time. Consistent bias usually indicates that your model is missing a key explanatory feature.
