Crisis? What Replication Crisis?

As humans, we believe in certain things; that the sun will rise in the east, we’ll have to pay taxes, and; ‘nope, this ain’t going to be the Warriors’ year neither!’

Our trust in these axioms is the bedrock of our belief system, the very spine of our existence. Knowing these things to be true gives us the confidence to get up every morning secure in the knowledge that the Sun will be there to meet us, as will the IRD bill in the email account, and that our wonderful Warriors will be dropping balls on the practice fields at Mt Smart.

Sure, magical axioms like these may not all be universally unchanging, as rules can get tweaked a bit in certain obscure fields of physics every so often – like whether dark matter really exists somewhere out in the distant icy expanses of space or not and whether gravity is a ‘thing’ at all. But on the whole, we are able to stride forward confidently to face an unknown future from within the comfort of our known, solidly grounded paradigm of truth.

From missteps to straight out stumbling

But… what if a lot of the things we’ve believed in for ages simply aren’t true at all? I know that’s a shocking thought – you may as well challenge those well-known truisms that: blackcurrant drinks have four times the amount of Vitamin C as orange based beverages, the Earth is flat and that St. Donald Trump always has the interests of ordinary citizens at heart.
Yet, as hard as it may be for us to even imagine, all of those things mentioned above, are no longer regarded as ‘facts’.

Why? Because we’ve tested them recently to see if they were still true – and they weren’t. What motivated us to test them? About 10 or so years ago, questions began to arise about why the results of various scientific studies from the past – often in the fields of psychology, medicine, social sciences and biology – couldn’t be replicated. Modern researchers looking to build on or refine those earlier studies just couldn’t get anywhere near the same results when they tried re-booting them in contemporary times.

This Replication Crisis of course began to cast doubt on the veracity of the previous results – i.e. it has exposed them for the pack of lies they really were.

Why on Earth did this happen?

The same reason everything happens – money. Tech or biotech startups rake in investment but in a scientific institution or university context, this usually takes the form of study grants. So, if you want to keep getting funds to pay your researchers, hire your equipment and kit out your labs – then you have to give your Ivory Tower masters what they want to hear: positive results.
Plus, as scientific journals are only ever interested in ‘significant’ results, you’re less likely to get a promotion or become a team leader if your work only ever returns ‘meh’ or inconclusive results. Horrible questions like, ‘What are you guys actually doing in those labs anyway?’ start getting asked.

Thus, you fudge a number here, tweak a reaction there – and straight out fib in the conclusion over there – and boom! – your grant money to do another study magically appears! Rinse and repeat until a whole culture of results-fudging develops.

Worldwide.

For decades.
But what harm can a few little white lies do? It’s not like anyone actually reads these highbrow journals anyway…

The lasting effects of a crisis

The Peloponnesian War happened ages ago, between 431–404 BC to be exact. At the time, it just seemed like a local scrap between the two cities of Athens and Sparta. Not a biggie – Ancient Greeks will be Ancient Greeks and tried to kill each other all the time. Yet this petty-seeming, silly little war had ramifications that have lasted all the way down the centuries until today.
Bullyboy Athens’ defeat made their hard-won utopian version of democracy seem weak and opened the door for autocrats to take charge – sound familiar? The war also weakened Greece allowing the Macedonians Philip II and his famous kid, Alexander the Great, to take over and go on to conquer all other the ruling empires of the time – including the , Egyptian, Persian and Indian ones. So, as an indirect result of the Peloponnesian War, the entire Known World was reshaped and reconstituted, influencing every global empire since…

The scary levels of damage

The Replication Crisis has a similar potential for long-lasting damage and for a tidal wave of distrust to cascade into different corners of our lives too. For example, a 2015 project by the Open Science Collaboration found that only 36 out of 100 selected psychology studies could be replicated. Yeow!

But this might not have been so bad if it were only inane business self-help theories that were in the gun. You know, theorems like Facial Feedback Hypothesis – where Strack et al. claimed that if you hold a pen in your mouth to force a ‘smile’ you would feel happier; or Carney, Cuddy & Yap’s zinger that holding a ‘power pose’ like Wonder Woman would boost your testosterone and confidence levels before you present to a meeting!

Scientifically, both are apparently a load of nonsense, but unfortunately it all isn’t just; ‘Ha ha, look at the silly pseudo-scientists!’ kind of stuff. The real problems start when processes like the Hare Psychopathy Checklist-Revised (PCL-R) get discredited. ThePCL-R is still being used – selectively – in our criminal justice system to assess psychopathy in criminals and to determine their risk for reoffending(and thus, whether they should be released into society or not).

So potentially violent and definitely anti-social criminals could be dumped back into our communities on the back of a shonky grant-grasping assessment tool?! Oh yes.

It’s a similar story with Emotional Processing for Sex Offenders. The idea here was that sex offenders lacked empathy so by getting them to ‘feel’ the victim’s pain they would be cured. The problem is that no one’s ever been able to come up with any evidence to prove that increasing empathy actually reduces repeat crimes. One thing they did discover though – many of the offenders lied to pass their evaluations. Funny that!

Some bitter pills to swallow

It gets even worse when the Replication Crisis creeps into medicine. Star performer here is the Theranos debacle where Elizabeth Holmes’ biotech startup swindled investors out of nearly $1 bn by claiming her device could perform hundreds of blood tests from a single drop. This was not just non-replicable but based on completely fake tech!

Then over we go to Big Pharma where, extremely worryingly, the Amgen and Bayer studies found that only a small percentage of landmark cancer research could be replicated. Amgen tried replicating 53 cancer studies whose results had been proudly paraded in top journals like Nature, Science and Cell. Yet only six of these studies could be reproduced, and even then, with results feebler than originally claimed.

It was a similar story with Bayer; they reviewed 67 pre-clinical projects and found that only 25% were replicable. Plus, in 65% of the cases, Bayer couldn’t get findings strong enough to justify drug development.

Whaaat? But this is really important stuff! People can – and do – die from misdiagnosis and medication errors. How can this happen? Again, the usual suspects of small sample sizes, cherry-picking data and missing methodology; along with journal print deadline pressure, hunger for ‘novelty’ and positive results seem to at least be partially at fault.

Sadly, the Crisis has spread almost everywhere in medicine too with observational studies on disease risk factors in epidemiology, efficacy in certain surgical procedures, and some pain management trials in anaesthesiology all in doubt due to non-replicable trials.

Even those good old sayings like ‘red wine is good for your heart’ can’t be replicated. No matter how hard a multitude of lab assistants worldwide tested the theory on a Friday night, they just couldn’t find any cardiovascular benefit – which is a real bummer as we’ve all lost an excuse for reaching for that third glass!

An axe to grind?

Of course, some will say that these studies have been nobbled by commercial rivals or the findings shot down via bitter ex-lab employees with an axe to grind.

Uh, not really. GlaxoSmithKline’s claim that their Ribena blackcurrant drink had four times the Vitamin C of oranges was disproved by some Pakuranga high school kids simply for their class science project! The original claim was so far from being a ‘fact’ that the kids were easily able to disprove it by showing that even cheap orange juice from the dairy had more Vitamin C than Ribena. An incensed Commerce Commission subsequently fined GlaxoSmithKline $217,500 for false advertising.

Opinion ain’t proof

Science really, really doesn’t need this Replication Crisis. There are already more than enough people out there eager to throw whatever findings climate and environmental scientists, vaccine developers and sex and race researchers come up with – under the bus. Having entire disciplines being eclipsed by a massive yawning credibility gap only gives these conspiracy theorists yet more ammunition.

Pop culture isn’t helping either. Every day it seems there is some new ‘expert’ espousing the exact opposite advice to what yesterday’s expert gobbed off. For every Neil de Grasse Tyson and Brian Cox there are on social media there are ten more dubious Deepak Chopras and Jessica Knuricks too. Whether these guys are like Graham Hancock and genuinely believe in what they’re peddling, or if they’re just cashing in on consumer gullibility – it doesn’t really matter. Everyone’s got the right to their opinion and to make a buck. But the combined effect of all these so-called ‘experts’ continually sounding off only further erodes our trust in Science in general.

Keeping the faith

Science needs to take some of its own hard medicine and win our trust back. To do that it needs to re-test everything with the attitude of: ‘if it can’t be replicated, it’s junk’. Bin it and develop a new theory. Anything that CAN be replicated gets a version of the Heart Tick as ‘Scientifically ProvenTM’. Everything else is just ‘opinion’. It’ll take a while to build a new brand, but it’ll work.
It’s one thing to hold doubts about psychology – I think most of us sneer at some of their findings, whether they can be scientifically proven or not – social sciences and biology possibly less so. But should we be like Sting and lose our faith in medicine, then we’re straight back to worshipping fire and sacrificing virgins in wicker effigies – and that freaks me all the way out. I thought the Ancient Greeks had pointed us in better directions – even from times well before the Peloponnesian War.

This Replication Crisis is a big, big problem and we need to address it pronto. Of course, this could also mean the end of certain disciplines as they are today – here’s looking at you psychologists! But they’ll bounce back, perhaps even with more credibility, on the back of better – proven – theorems and processes.

Addressing this Crisis ASAP is necessary because we have to cut out the cancer of mistrust in science before it harms medicine any further.