I was reading a study about some studies recently…
A team at Harvard published four papers suggesting that people living near nuclear power stations were more likely to die of cancer.
Makes sense on the face of it, doesn’t it?
But then two other researchers took the same method and tried it on other things instead. Universities, state capitals, Costco. By the same sums, Costco is responsible for more than a fifth of all American cancer deaths between 2000 and 2018. Harvard didn't come out of it well either.
Nobody thinks Costco causes cancer, obviously. The method seems to find something wherever you point it. And nobody had checked, because nuclear power came with a story already attached and the result fitted the story.
Lies, damned lies and statistics. I don't think anyone was lying, really. The numbers were real but nobody had asked whether the method could ever say no.
My “profession” does this. Funds launch with lovely backtests (the ugly ones never get launched), and there's always a market pattern that would have made you rich if you'd spotted it beforehand. Sell in May and go away, and so on.
Truth be told, I don't look at the charts I am sent showing a whizz-bang new fund or unbeatable strategy.
But if I did, there's one question I'd want answered. If you ran the same numbers on something daft, say every company whose name begins with B, or the price of milk, would they still show a result? I suspect quite often they would.
It's one reason we keep the investment side fairly dull and let the plan do the heavy lifting. Dull doesn't need a story to prop it up.
I've tried the test on a few of my own ideas. It's not gone brilliantly.