Below, I discuss a study that, as it turns out, basically says the opposite of what it claims in its abstract. The study made me more angery than I initially expected, which means it's time for a mini-rant about what I'm calling Drunk Mormon Hypotheses.
In short: If a group sets a specific goal, the default hypothesis should be that they are not further from the goal than they would be if they did not have that goal.
The titular example is Mormons and alcohol. It should surprise nobody that Utah has the lowest alcohol consumption in the nation, because Mormons famously don't drink alcohol while just about everyone else in the US doesn't have a problem with it. If you spot a report that mentions a lot of drunk Mormons kicking around, it should register as something odd and unexpected.
Climate change is another example. Since the Republican Party has no stated goal to reduce carbon emissions, the default hypothesis should be that Democrats are more likely to reduce carbon emissions than Republicans. This should be the default whether or not you consider reducing carbon emissions to be an important goal.
A Drunk Mormon Hypothesis, then, is one that states the opposite: A group is worse at achieving their own goal than they would be without that goal. It's completely possible for this to be true, of course, but it requires a heavy burden of proof. If a claim like this is made in the absence of proof, doubt it.
This is applicable in a broad range of contexts, but because of the study linked above, I'm thinking primarily of examples surrounding online claims about socially conservative policies.
Examples:
-
Prohibition was a failure (well, it decreased exactly the things it meant to decrease)
-
Conservative policies don't decrease abortion rates (ehhhh)
-
Tracking doesn't help advanced students (many words to say "teaching someone specifically what they're ready for does, in fact, help them")
And, returning to the claim that started me off on this whole chain, the initial question of the linked study:
- Why do states with larger proportions of religious conservatives have higher divorce rates than states with lower proportions of religious conservatives?
Then you dive into the study and find out that the answer is, according to their own data, "all religions seem to provide a strong influence against divorce compared to a lack of religion, but evangelical faiths as a group are somewhat less effective than other religions at achieving this goal. Also, regions that exhibit strong compliance with religious values such as high proportion of marriage versus cohabitation tend to have dramatically lower divorce rates." This is almost the direct opposite of the claim they were strongly implying.
That's why I'm angry right now, to be clear. I don't think the writers were being actively malicious, but burying the largest effects (religious unaffiliation, cohabitation rates) in a few paragraphs in the data section while structuring their entire paper around the implication that conservative religious people have higher divorce rates than everyone else is about as extreme of narrative stretching as you can get without actively falsifying data. "People in cultures strongly opposed to divorce will likely divorce less than people in cultures with weaker opposition" should be a deeply unsurprising observation, but it gets obscured by things like this.
Now, it's important to emphasize something obvious here. A group setting a goal doesn't mean that goal is good. It doesn't mean working towards that goal won't cause terrible consequences for other goals. It doesn't mean they're working towards the goal in the most efficient possible way, or that other methods won't also work to progress towards that goal. Accepting that they're progressing towards their goals, then (or not regressing), is not tantamount to saying, "You're right and I'm wrong." It's largely just the first step towards a meaningful conversation about which goals are important to pursue and what we should do in pursuit of those goals. So—religious people tend to get divorced less than others. What tradeoffs and costs are associated with this? Do they have higher rates of unhappy marriages? So forth.
In addition, some Drunk Mormon Hypotheses are right, and they're incredibly important when they are. If someone genuinely wants to accomplish something, and what they think is helping is having the opposite effect, they should want to know. My commentary about them isn't intended to demonstrate that every counterintuitive result in this vein is false, only that if you see a situation where it's claimed that a group working towards a goal is actively worse at it than those without that goal, you should demand a particularly high burden of proof.
Drunk Mormon Hypotheses have an innate strong appeal, since there's something incredibly satisfying about uncovering outgroup hypocrisy. "We're better than them at the things they care about" plays really well in just about every group, and if broadly accepted, it kneecaps the opposition. My instinct is that, because of this appeal and rhetorical strength, a lot of these ideas stick around without ever being seriously investigated. They often become absurd on closer examination, but they're useful enough that nobody really cares to look deeply at them.
For contrarians like me, that's actually pretty useful. It means you can make a surprising amount of progress towards interesting ideas pretty quickly simply by noticing things that might be Drunk Mormon Hypotheses. If done right, your conclusions should appear blindingly obvious to everyone in the group being discussed ("Look, of course we don't drink much. It's a huge sin!") while countering some prevailing wisdom elsewhere. I would say around half of the things I like to dive into lots of obscure data on arise only because I spot a Drunk Mormon Hypothesis in the wild and get really irritated.
After all, it means that someone's wrong on the internet. And that simply cannot stand.
Cheers!