When you see headlines about changes in something scary like crime or disease, make sure it’s an apples-to-apples comparison. Otherwise, you might get fooled by faulty statistics.
Let’s say you’re new to birdwatching. You want to know whether birds of a certain color—gray, for example—are most common near your house. You grab your notebook and head outside. Out of the 20 birds you see, 2 of them, or 10%, are gray.
You compare notes with your more experienced bird nerd friend across town and are shocked to see that more than half of the birds they counted are gray! Is the wildlife in your neighborhood really that different from theirs? Should you be worried about a dangerous predator going after gray birds near you? Or could something simpler be going on?
It might seem silly, but before you can count gray birds, you have to decide what a gray bird is. Can it be mostly gray but have a white belly? What about speckles of another color? How dark can it be before you consider it black, not gray? You had one definition, and your friend had another. This led you two to make an apples-to-oranges comparison, where you’re trying to compare two things that aren’t the same. If you’d agreed on a definition beforehand, you might have noticed that the two samples of birds are actually identical!
This is a low-stakes scenario (sorry, backyard birders!), but it walks you through a much bigger issue. People often think of data as being objective, but that’s rarely the case. Each data point you see online or in the news is actually made up of lots of decisions made by lots of different people. Sometimes those decisions change, or aren’t the same from one place to another, or are updated with new knowledge.
Can you think of any higher-stakes scenarios where the same thing happens? Our recent video on whether there’s an epidemic of autism diagnoses touches on a similar situation: the definition of autism has changed to become more broad, and that’s part of why rates of diagnoses have gone up. That change isn’t part of a conspiracy or a ploy. It’s just part of how science works: we find new information, we broaden our understanding, and we make adjustments. We discover that maybe speckles are part of the coloring of a bird, or that behavioral differences might be more subtle than we once thought.
So the next time you get worried about scary-sounding numbers, make sure you know what’s being measured, by whom, and how it’s defined. Otherwise, someone might be hoping you won’t ask what exactly is a gray bird.
