Thinking critically about Lost Connections 2: The evidencing double standard

Part 2 of my Lost Connections review. Back to part 1

I spent many years of my life in varying levels of depression. Sometimes it was really bad: the ugly, messy shit, the lying in bed too tired to cry, too tired to sleep, subsisting off a diet of Haribo, because it was closest to the bed. Much of the time, it was less bad than that: a nagging numbness, a constant feeling of a dragging weight, a listless lack of enjoyment of things that ought to be fun, problem drinking, problem screwing, problems all over the shop. The usual. I knew the depression was there, and I knew there would be ways of helping myself out if I’d just go and see a fucking doctor.

Once, back when I was at uni, the head of department gently suggested to me, as I was falling very much behind on my workload, a course of citalopram to get myself back on my feet. I pretended I’d consider it as my brain told her to go and stuff herself, because I’d read the research and knew they weren’t something that would magically make me an effective worker (which, it was transparently obvious, was her motivation for the recommendation!). I tried a couple of courses of therapy, and they weren’t particularly helpful, being too short, and not being all that suitable for someone who, it turns out, is physically impaired at visualising.

I never took meds, because I had a low opinion of them, and felt like they’d probably do more harm than good for me.

Meanwhile, my epilepsy also took a turn for the worse, and last year I began finally dealing with that by taking an anticonvulsant medication called lamotrigine. Now, lamotrigine works pretty well at controlling epilepsy. It also has another medical function: treating the depression symptoms side of bipolar.

And would you guess what? An unexpected side effect of my epilepsy medications was that my depression has subsided. I’ve felt this strange sense of energy over the last year, a feeling that I am no longer dragging an ever-increasing weight with me wherever I go. I’m getting out of bed every day, and eating meals. I’m not feeling a crushing insurmountable despair. The drug I was prescribed for different reasons has, it seems, also treated the depression I lived with for years.

Now you’ve read my story, are you planning on popping lamotrigine to deal with your depression?

No?

Then you, my friend, are a sensible person who understands on some level that anecdote isn’t data. That the experience of one person is not the same as others. That, no matter how much a story may resonate with you, it’s something you’d probably want to research a bit more independently and chat to a doctor before seeing if the drug I take (which has a small chance that it might make your skin fall off) is suitable for you.

I opened this section with a personal story because I figured we might as well start on a level playing field of evidence.

Evidencing is important, and doubly important if you were exposed as a fabricator and plagiarist.

When writing about science, there are certain standards of evidence that are better than others, and certain types of reference where you’d be chased out of any research institute with pitchforks if you put them down. The former is stuff that’s published in journals or edited academic books: meta-analyses, experiments, rigorous population research and the like. The latter is stuff like popular science books, blog posts, asking someone who you reckon knows a bit more about the subject than you do, personal anecdotes, and so forth. I’m pleased to report that Lost Connections contains some of the former. It also contains a lot of the latter.

We also talk about levels of evidence, because that’s important too. All evidence is not equal. For example, a single study containing 20 people is less good evidence than a study containing 200 people. A study using 200 monkeys is less good evidence than a study using 200 people, if you want your findings to apply to  humans. A study which finds a correlation between one variable and another is less good evidence than a study where you manipulate one variable and measure its effect on the other. One study finding something is less good evidence than 15 studies that find the same thing. A meta-analysis is great evidence: that’s where you put together findings from lots of studies on the same topic to check if the results still hold up. When you are looking at the evidence, these are all things to bear in mind.

The funny thing is, Hari understands this… to a point. He is very clear on all of these points when presenting the evidence that antidepressants don’t work very well, and explains them reasonably well. Now, I’m not going to get into a point by point analysis of the early sections of the book and why I disagree with his conclusion that the drugs don’t work, because I covered pretty much all of my problems with it in my post about the extract, and every point besides the first remains in place for me as I read the text in its entirety.

In the psychological sciences, we often reference “in-line”, so it makes it easier to pick up on the references.  APA formatting is pretty standard for if you’re publishing a psychology study: when you’re referencing a statement, you’d name the authors and the date they wrote what you’re referencing within the sentence. They’re preferable to footnotes as it makes our jobs a lot easier to look at a reference quickly and check if you’re citing a journal article or whether it’s a forum post by DongSmoker6969.

Lost Connections uses footnotes. Lots and lots and lots of footnotes. And sometimes the footnote will go to a proper scientific study. Sometimes it will not. It’s a crapshoot, and I will confess that I did not bother checking every footnote, because I value my own sanity. I often just checked the footnote if something seemed a little bit off to me. So, obviously draw healthy scepticism about what I say throughout, because no, I did not check every single reference. I also found Hari’s approach to evidencing particularly irksome as he frequently refers to researchers by their first names, which makes it a little harder to follow who he’s talking about. For example, in Chapter 7, he often alludes to the work of a researcher called John, and so I spent half the chapter thinking “who’s John, and why should I care?”. Flicking back, I finally found it was John Cacioppo, a researcher so well-known they teach his work about loneliness in Psychology A-Level: his work is familiar to many–and I’ll be writing a bit more about that in a later section of this review.

The thing is, though, Hari’s own talk of high standards of evidencing completely collapses as he writes about his own views on the causes of and solutions for depression. There, we suddenly see conclusions drawn from a study involving animals. We see for-profit companies making a profit on selling the idea of a cause of a sickness for which they provide a cure. We see grand conclusions being drawn from one-off studies involving sample sizes of 20, with very non-representative samples.

A substantial portion of what Hari is proposing as “reconnections” to alleviate depression are alternative remedies, be it lifestyle changes, meditation or, indeed, a herbal supplement.

Hari dismisses pharmaceutical interventions for depression, suggesting they’re no better than the placebo, appealing to the authority of Professor Irving Kirsch. Kirsch gives us a suggestion, due to the side effects of antidepressant medication, of an alternative placebo:

“We could be giving people the herb St John’s Wort, Irving says, and we’d have all of the placebo effects and none of these drawbacks. Although–of course–St John’s Wort isn’t patented by the drug companies, so nobody would be making much profit off it.”

I want to focus on this quote because it highlights a lot of problems rather neatly. Firstly, St John’s Wort is not an inert substance. It actually is effective in treating mild-to-moderate depression, and Mind conclude you can use it as an alternative to antidepressant drugs. It affects serotonin levels. However, being not inert, there are also some dangerous problems with St John’s Wort. It has side effects, like any other medicine. And it also interacts with a lot of other medicines to stop them working as well: important drugs such as contraceptives, HIV medications, blood-thinners and heart medicine.

Another problem with herbal remedies is you might get the dose wrong. While pharmaceutical drugs must be tested very rigorously to find the right doses that won’t kill people or make them sicker, this is not the case for herbal medicines. That’s worrying.

You can take St John’s Wort, and it might work for you. However, as with any drug, you should see a doctor so they can make sure you’re doing it safely and you don’t need anything else, to help you.

Of course, St John’s Wort is a herbal remedy that actually works, but we can divide most alternative remedies into two categories: the ones which do work, and therefore, because they’re doing something, can come with side effects and you need medical monitoring; and the ones which do precisely nothing, like homeopathy, which is literally a sugar pill.

Medical professionals’ tendency to avoid alternative remedies is frequently treated as Big Pharma suppressing the secret treatments that really work. And that’s not true. The truth is that they’re often not recommended because they’ll either not do anything, or have an effect that’s wildly unpredictable and possibly dangerous, and therefore, it’s considered better to go with drugs where we know all about safe dosage, side effects and interactions.

The pharmaceutical industry is, of course, not spotless. It’s pretty evil, and I’m eternally grateful to Martin Shkreli for putting a punchable face to everything I hate about it. However, I am highly surprised by the suggestion that nobody is profiting from St John’s Wort; alternative health industry is every bit as lucrative as the pharmaceutical industry, and every bit as evil.

I would imagine the profit margin for the alternative health industry is comparable to, or possibly larger, than that in the pharmaceutical industry. If you’re selling crushed-up flowers or a meditation tape, you don’t have to spend a large amount of money in research and development, testing, and ensuring your remedy is safe. You might sell less of the product, but you’re going to make a killing. It becomes even more unethical when the product is ineffective, as this gives false hope to sick people, which is diabolically cruel.

In the UK, there is a further issue with recommending alternative remedies over what’s available from a doctor: money, dear boy. Here in the UK, the patient doesn’t have to pay full price for medicines, only a prescription fee. Talking therapies are provided free of charge, as are many other services for mental health. Meanwhile, a month’s supply of St John’s Wort would set you back £15, while an SSRI on prescription would be £8.60. Which, by the way, is still revoltingly expensive, and too many chronically ill people are paying through the nose for their prescription medicines. As a patient, it’s probably cheaper for you to stick with the NHS.

I use St John’s Wort as an example here, although perhaps that is unfair, as there is stronger evidence for its efficacy as a treatment for depression than many of the other things presented in the book: for example, I checked the references on “sympathetic joy” meditation. It’s effective. At some things. None of which are treating depression.

Let me provide some concrete examples. I had particular trouble with Hari’s Cause 6: Disconnection From The Natural World. A substantial portion of the evidence presented pertains to bonobos. Now, I hope I don’t have to tell all of you that bonobos aren’t people, and are instead kind of a nicer version of chimpanzees that fuck a lot. We then have a bit of correlational evidence about mental health evidence upon moving from somewhere green to a city, or vice versa. And finally, we have a bit of experimental evidence from a study where people went for a walk in either a natural or a rural setting which found that the walks in nature were superior. A grand claim, so let’s follow that footnote, shall we? It leads us to a 2012 study from Berman and colleagues. The experiment had a sample size of 20 people, which is pretty small, and all of the sample had diagnoses of the same type of depression, which means they’re not very representative of the population.

Had that been a trial for a drug intervention, nobody on this earth would ever be given that drug, and rightly so. Had this been a test of an antidepressant, Johann Hari himself would have noticed that this was not a very good study, and certainly not anything which ought to influence your medical decisions. We see this throughout. For example, we are treated to a dismantling of the serotonin theory of depression, and shown it was a marketing tactic from pharmaceutical companies. This is fair enough. Then, in chapter 7, we are treated to an explanation of how internet addiction is a real thing and very bad, from someone who works at a for-profit internet rehab clinic.

One cannot have it both ways. One cannot take a critical approach to evidence you disagree with, and then turn around and accept evidence–often of a poorer quality–to something that you do agree with. This is called “cherry picking”, and it’s generally frowned upon–indeed, Irving Kirsch, who features heavily in early chapters of the book dismantling evidence of the effectiveness of antidepressant drugs, worked very hard to minimise the cherry picking effect of pharmaceutical companies’ publication of drug trials on the evidence of effectiveness.

Yes, I’m going to take a second to giggle about a man called Kirsch working against cherry picking, because I’ve just read a book I hate and I need a bit of levity in my life.

By all means, be critical of evidence, any evidence. It’s how science thrives and how medical treatments improve. But this rigorous approach needs to be applied to everything you are writing. The high standards of evidencing introduced by Hari at the beginning do not hold up in the slightest when we’re looking at his proposed alternative model, and his remedies.

The evidencing double standard is a vast problem within the text and pollutes everything within. However, the rot is even worse than that: the entire argument presented in the book rests on a strawman, and that’s what we’ll be looking at tomorrow.

Part 3: Defining depression and the false dichotomy >

_

Enjoyed what you read? Consider becoming a Patron, support me on Liberapay, or leave a tip