Saturday, 21 March 2009

Abusing Simple Linear Regression Models

I was very excited about the book I recently bought. However, after a brief first read I'm not sure I'll continue reading it.

Here are two examples of "evidence" presented in the book.



There is a lot to be said about this. For example, that the regression line is obviously a bad fit for the data.

A very quick and rough analysis in Matlab yields the following:



The red line is the regression line the authors use. The blue line is the regression line you get if you drop a single data point (in particular the US). The model the authors use suggests that the homicides go from 10 (low inequality) to about 30 (high inequality) per million (a 200% increase). Dropping the US results in a model that predicts 13 to 22 (an increase of less than 100%). The authors don't bother to point out any of this.

Furthermore, the (95% confidence level) prediction boundaries (dotted lines) indicate that the models need to be treated carefully. The confidence levels for predictions range up to 34 for low inequality and down to 6 for high inequality. The overlap suggests that the authors should gather more data if they want to prove that "Homicides are more common in more unequal countries".

The second example:



The authors briefly mention that there is an issue when using only so few data points. On the other hand they pretend that the data is very clear. Removing the UK and US will completely change the results of their regression model. The data gives their conclusion ("Social mobility is lower in more unequal countries") only very weak support. I also don't like how they scaled it. If I assume that the "low-high" ranges are comparable in both cases I would expect this to be reflected in the plot. Instead the mobility dimension seems visually about 30% less important than the inequality dimension. Anyway, I'm not really sure how useful "low" and "high" without exact definitions are in the first place.

No comments:

Post a Comment