Last week a piece of science news popped up: The jet stream had apparently crossed the equator and thus created a “climate emergency” that announced a catastrophic collapse of the food supply. It sounded preposterous, but as what little climate expertise I have is limited to the northern latitudes (I’m not an atmospheric scientist), I didn’t think I could do more than make skeptical noises against the many quotes and re-tweets the articles received on social media.
The claims were invalidated a day later in, eg. “Claim that jet stream crossing equator is ‘climate emergency’ is utter nonsense” in the Washington Post or “No, the Earth’s jet streams are not spinning out of control” in Ars Technica.
Meanwhile the original piece made its way, substantially unaltered, from relatively niche media into mainstream outlets such as the Independent. Nonetheless, it is possible to spot junk science a little earlier on, if we give some thought to how to assess how much trust we should place into a piece of science reporting. Here are, roughly, the steps that I follow.
Step 1: Who is reporting, and more importantly, what are their sources? Yes, I usually look at the context before I look at the content. Unfortunately, you will find junk science reported in the most reputable papers, and reliable facts in niche magazines, even in magazines with a political slant. Still, overall there is a small credibility bonus on news that is reported by a variety of outlets, including big mainstream ones.
The real reason I look at the source first, though, is to figure out two things: a) is it from a peer-reviewed journal, and if the not, what else? b) did the journalist rely on a single source or get multiple statements to elucidate the topic?
The first point is less meaningful than many think. Unreliable results are published in peer-reviewed journals every day (and that’s not even necessarily a bad thing), while a huge amount of great science news comes from non-peer-reviewed sources: a volcano is erupting; a new particle accelerator just started service; a rare bird was spotted; a spacecraft is flying by a comet; the previous month broke weather records. It doesn’t even have to be a professional source to be credible, but one thing to keep in mind is whether the source and the content are compatible: amateur astronomy or zoology is a thing, but amateur particle physics really isn’t; and a lab result that would question everything we know about the chemistry of life should come from a peer-reviewed source, and not be published via press release.
The second test, however, is a pretty clear-cut one. Good news has corroborating voices from other members of the discipline. In the “jet stream crossing the equator” case, the original reports relied on blog posts and videos from an instructor at the University of Ottawa who has yet to get his PhD in climatology, and from an author of, it seems, climate catastrophe fiction. Far be it from me to belittle the expertise of PhD candidates — being one myself — but no corroborating voices were cited in the piece, even though the discipline is chock-full with established scientists who are only too ready to be interviewed.
Step 2: Does it pass the smell test? That is, does it, just overall, feel plausible? Another news item I saw last week reported on a skeleton found on the shores of Loch Ness. In the accompanying picture, the pile of bones didn’t look like any wild animal carcass I’ve ever seen, but like something form a butcher’s shop. The rib cage was about cow-sized, probably two or three individual animals’ worth. So… probably not Nessie, right?
This test, however, must be wielded with care. We all have a tendency to nod along with news that reinforce what we already believe — either from what we were taught about the world, or from our general philosophy — and to be skeptical about what contradicts it. This is called confirmation bias, and it’s strong and insidious. So we first need to take a hard look at the contours and limits of our expertise.
Step 3: What is the tone of the report? Two types of science reports make my senses tingle: horror stories and feel-good news. Reliable science reporting isn’t out to speak first and foremost to our emotions, but to help us understand something about the physical world. Of course, once we have understood it, we’re perfectly at liberty to attach an emotional value to it. Some things are genuinely frightening (or hopeful). But an article about, say, landslide hazard in some area of the world needs to talk facts and rules and laws and models and model uncertainty. Even if the outcome is “crap, someone needs to get these people out of there ASAP”.
A different red flag goes up if the article’s tone is mocking or dripping with scientific superiority. For news from some disciplines, like agricultural bio, it has become rare to find news that aren’t trying to influence your attitude to further a political goal. More generally, the attitude is annoying and unhelpful.
Step 4: Do I have people with more expertise in my social network? So let’s say you’ve gone through steps 1 to 3 and are still not sure what to think. Maybe you know someone else who does! I’m frequently astonished about the breadth and depth of expertise among my Facebook friends and Twitter network. Instead of re-posting the article with the caption “OMG the food supply is going to hell”, why not try “Hmm, this would be pretty bad if it’s true. Does anyone have some insight?”
Step 5: Maybe just … wait a little while? It’s a humbling thought, but if the jet streams are going to crap, chances are neither you nor I will be the ones whose social media post is going to save the world. If the report is true, it would with near certainty be picked it up. NOAA or the UK Met Office or the German weather service (DWD). The major science writers would dig into it. Within a few days, we’ll all have a much more complete account of what’s going on. And then we can panic. (And if they don’t, it’s still ample time to find out why not.)
So the bottom line is: If the report is from a single unconfirmed source, and sensational (be it positive or negative!), it’s time to send it out for a second opinion.
