How to Evaluate a Claim: A Practical Checklist

Every day brings new claims: a miracle cure, a shocking statistic, a photo that seems to prove something. Most of us do not have time to investigate each one, and few of us were taught how to do it quickly.

In 1995 the astronomer Carl Sagan offered a toolkit in The Demon-Haunted World, a book written to explain how science works to general readers. He called it a baloney detection kit, and its nine tools still work, whether the claim is about aliens, health or politics.

Science as a habit, not a body of facts

Sagan's central idea was that science is less a collection of facts than a way of thinking, which he described as a built-in error-correcting machine. It assumes that we will sometimes be wrong and builds in ways to find out. The checklist below is that habit applied to everyday claims.

Four of the book's chapters were written with Ann Druyan, and she later said the name of the kit came from a friend of theirs, Arthur Felberbaum.

1. Look for independent confirmation

A claim that only one source makes is weak. Before you believe it, look for independent confirmation of the facts: does anyone without a stake in the story report the same thing? Several outlets repeating one original report do not count as independent.

This is the single most useful habit. Many viral stories collapse as soon as you look for a second source and find there is none.

2. Encourage debate, and weigh authority lightly

Sagan's second tool is to encourage substantive debate on the evidence by knowledgeable people of all points of view. His third is that arguments from authority carry little weight: experts have been wrong before, and in science an argument has to stand on evidence, not on who makes it.

That does not mean ignoring expertise. A broad agreement among people who have studied a question deserves weight, for reasons we explain in what scientific consensus means. It means that a single famous name is not evidence.

3. Consider other explanations

Spin more than one hypothesis, Sagan advised, and think of tests that could decide between them. Then avoid falling in love with your own favourite: be as hard on the explanation you like as on the ones you do not.

Many mysteries dissolve at this step. Lights in the sky could be aircraft, satellites, weather balloons or reflections before they need to be anything stranger; a photo could be edited, staged or simply mislabelled. Our guide on how experts check if an old photo is fake shows the same discipline applied to images.

4. Quantify, and test every link

Wherever possible, put numbers on things. A claim that a risk has doubled sounds alarming until you learn it went from one in a million to two in a million. Numbers make claims testable and comparable.

And if an argument depends on a chain of reasoning, every link has to work, not just most of them. A conspiracy theory that requires thousands of people to keep a secret for decades fails at the link where any one of them talks.

5. Prefer the simpler explanation, and ask if it can be tested

Occam's razor says that when two explanations fit the facts equally well, choose the simpler one. It is not a law of nature, but it is a sensible rule for where to place your bet.

The last tool is the most powerful: ask whether the claim could, even in principle, be shown to be false. Sagan illustrated this with a story of a dragon in his garage that is invisible, floats in the air and breathes heatless fire, so that no test could ever detect it. If nothing could count against a claim, it is not telling you anything about the world.

The kit in action

Take a famous case and run it through the kit: the 1934 Surgeon's Photograph of the Loch Ness Monster. Was there independent confirmation? No; one photograph, from one source, with a respectable name attached. Were other explanations considered? A small object close to the camera, a model, a bird or a log would all produce a similar shape.

Could the claim be tested? Only by finding the creature, or the photographer's materials. Sixty years later a confession provided the second: the monster was a toy submarine with a putty head. Applied in 1934, the kit would not have proved the photo was fake, but it would have stopped anyone treating it as strong evidence. Our account of the Loch Ness monster and the Surgeon's Photograph tells the whole story.

Common traps

The second half of Sagan's kit lists about twenty logical fallacies to watch for. Among the most common are attacking the person instead of the argument, assuming that because one thing followed another it was caused by it, and arguing against a weaker version of a position than the one actually held, the straw man.

You do not need to memorise them all. Asking where a claim came from, whether anyone independent confirms it, what else could explain it and whether it could be tested will catch most of what crosses your screen. For a case where all of these failed for a long time, see why people believe the Earth is flat.