Lies, damned lies and statistics

By Neil Patrick

We all know this saying. But it came into sharp focus last week, when I was researching what the numbers of companies recorded at Companies House actually means.  Answer - not as much as some people would have us believe.

These types of numbers are routinely quoted by politicians as evidence of the correctness of their argument or viewpoint. But politicians are rarely even numerate, let alone statistical experts.

In 2011, Ipsos MORI carried out a test to find out how good with numbers our politicians actually are.

A total of 97 MPs were asked this probability problem: If you spin a coin twice, what is the probability of getting two heads?*

According to the BBC report, among Conservative members, 47% gave the wrong answer, which is disappointing enough. But of the 44 Labour MPs who took part, 77% answered incorrectly!

*The correct answer is of course 25%.

This is truly shocking I think. Whilst MPs of all parties are quick to claim that the nation’s general standards of numeracy are not high enough, their own levels of numeracy are truly frightening. If you or I are not great with numbers, we are the main losers. When it comes to MPs, if they cannot understand numbers they will make potentially disastrous decisions that affect us all.

According to Wikipedia:

"Lies, damned lies, and statistics" is a phrase describing the persuasive power of numbers, particularly the use of statistics to bolster weak arguments. It is also sometimes colloquially used to doubt statistics used to prove an opponent's point.

The term was popularised in the United States by Mark Twain (among others), who attributed it to the 19th-century British Prime Minister Benjamin Disraeli (1804–1881): "There are three kinds of lies: lies, damned lies, and statistics."

I never really attributed much significance to the saying. It was merely a witticism to me. But last week, after spending an inordinate amount of time trying to find some simple numbers in the website of the Office for National Statistics (ONS), I began to see deep meaning and wisdom in the saying.

Then on Friday night I sat down in my local pub with a well-earned pint and encountered a friend. I recalled he worked for the ONS as a compiler of government data.

At first as I was feeling off-duty, I merely ribbed him about how hopelessly impenetrable his employer’s website was. Instead of pushing back, he actually agreed with me.

A rather more earnest conversation ensued. What emerged was less sensational than conspiracy theorists would have us believe, but also much more illuminating than anything I had thought of.

And it boils down to this. The ONS who are the main compliers of all UK government data which is used to inform decisions made by many including the government and Bank of England, depends on millions of inputs every month from individuals and organisations. A few, (my friend suggested less than 1%), take their reporting of data very seriously. But the vast majority see it as a difficult and burdensome task and consequently give it little priority or care (there is no benefit to them in being super diligent after all).

In most cases the task gets delegated to a very junior person who also cares little about accuracy, and probably doesn’t even know for sure what the correct figures are. According to my friend, the usual outcome is that a number similar to the last one provided is simply provided. After all if a big change is reported, even if it is true, the number may get queried, creating even more unwanted work. So the default is to just put in a similar number to the last one provided, even if this bears no resemblance to the truth.

I’m not suggesting that this means all government statistics are meaningless. On the contrary, they are probably the best indicators we have. But they are not correct and will never be. However, even if they over or under report the actual figures, the inherent errors will be consistent. This means that the most important information i.e. the trends rather than absolute numbers will be broadly informative.

So next time you see a press headline which quotes an actual number rather than a change in a number, remember this number is almost certainly wrong.

And if it’s quoted by a politician, remember they almost certainly have little idea of what the number actually means.

As everyone in IT is constantly telling is, “ If c**p goes in, c**p comes out”. And this is clearly what is happening with our national statistics.