{"id":3692,"date":"2025-05-09T07:02:21","date_gmt":"2025-05-09T07:02:21","guid":{"rendered":"https:\/\/mailitics.com\/index.php\/2025\/05\/09\/the-dangers-of-deceptive-data-part-2-base-proportions-and-bad-statistics\/"},"modified":"2025-05-09T07:02:21","modified_gmt":"2025-05-09T07:02:21","slug":"the-dangers-of-deceptive-data-part-2-base-proportions-and-bad-statistics","status":"publish","type":"post","link":"https:\/\/mailitics.com\/index.php\/2025\/05\/09\/the-dangers-of-deceptive-data-part-2-base-proportions-and-bad-statistics\/","title":{"rendered":"The Dangers of Deceptive Data Part 2\u2013Base Proportions and Bad Statistics"},"content":{"rendered":"

The Dangers of Deceptive Data Part 2\u2013Base Proportions and Bad Statistics
\n \t

\n
<\/BR>
\n
\n \t

\n
<\/BR><\/p>\n

\n

This is a follow<\/mdspan>-up to my earlier article: The Dangers of Deceptive Data\u2013Confusing Charts and Misleading Headlines<\/a>. My first article focused on how visualizations<\/em> can be used to mislead, diving into a form of data presentation widely used in public matters.<\/p>\n

In this article, I go a bit deeper, looking at how a misunderstanding of statistical ideas is breeding ground for being deceived by data. Specifically, I\u2019ll walk through how correlation, base proportions, summary statistics, and misinterpretation of uncertainty can lead people astray.<\/p>\n

Let\u2019s get right into it.<\/p>\n

Correlation \u2260 Causation<\/h2>\n

Let\u2019s start with a classic to get in the right frame of mind for some more complex ideas. From the earliest statistics classes in grade school, we are all told that correlation is not equal to causation.<\/p>\n

If you do a bit of Googling or reading, you can find \u201cstatistics\u201d that show a high correlation between cigarette consumption and average life expectancy [1]. Interesting. Well, does that mean we should all start smoking to live longer?<\/p>\n

Of course not. We\u2019re missing a confounding factor: buying cigarettes requires money, and countries with higher wealth understandably have higher life expectancies. There is no causal link between cigarettes and age. I like this example because it is so blatantly misleading and highlights the point well. In general, it\u2019s important to be wary of any data that only shows a correlational link.<\/p>\n

From a scientific standpoint, a correlation can be identified via observation, but the only way to claim causation is to actually conduct a randomized trial controlling for potential confounding factors\u2014a fairly involved process.<\/p>\n

I chose to start here because while being introductory, this concept also highlights a key idea that underpins understanding data effectively: The data only shows what it shows, and nothing else.<\/strong><\/p>\n

Keep that in mind as we move forward.<\/p>\n

Remember Base Proportions<\/h2>\n

In 1978, Dr. Stephen Casscells and his team famously asked a group of 60 physicians, residents, and students at Harvard Medical School the following questions:<\/p>\n

\n

\u201cIf a test to detect a disease whose prevalence is 1 in 1,000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming you know nothing about the person\u2019s symptoms or signs?\u201d<\/p>\n<\/blockquote>\n

Though presented in medical terms, this question is really about statistics. Accordingly, it also has connections to data science. Take a second to think about your own answer to this question before reading further.<\/p>\n

\"\"<\/a>
Photo by Getty Images<\/a> on Unsplash<\/figcaption><\/figure>\n

The answer is (approximately) 2%. Now, if you looked through this quickly (and aren\u2019t up to speed with your statistics), you may have guessed significantly higher.<\/p>\n

This was certainly the case with the medical school folks. Only 11\/60 people correctly answered the question, with 27\/60 going as high as 95% in their response (presumably just subtracting the false positive rate from 100).<\/p>\n

It is easy to assume that the actual value should be high due to the positive rest result, but this assumption contains a crucial reasoning error: It fails to account for the extremely low prevalence of the disease in the population.<\/p>\n

Said another way, if only 1 in every 1,000 people has the disease, this needs to be taken into account when calculating the probability of a random person having the disease. The probability does not rely only on the positive test result. As soon as the test accuracy falls below 100%, the influence of the base rate comes into play quite significantly.<\/p>\n

Formally, this reasoning error is known as the base rate fallacy<\/strong>.<\/p>\n

To see this more clearly, imagine that only 1 in every 1,000,000 people had the disease, but the test still has a false positive rate of 5%. Would you still assume that a positive test result immediately indicates a 95% chance of having the disease? What if it was 1 in a billion?<\/p>\n

Base rates are extremely important. Remember that.<\/p>\n

Statistical Measures Are NOT Equivalent to the Data<\/h2>\n

Let\u2019s take a look at the following quantitative data sets (13 of them, to be precise), all of which are visualized as a scatter plot. One is even in the shape of a dinosaur.<\/p>\n

\"\"
Image By Author. Generated using code available under MIT license at https:\/\/jumpingrivers.github.io\/datasauRus\/<\/a><\/figcaption><\/figure>\n

Do you see anything interesting about these data sets?<\/p>\n

I\u2019ll point you in the right direction. Here is a set of summary statistics for the data:<\/p>\n

\n\n\n\n\n\n\n\n
X-Mean<\/td>\n54.26<\/td>\n<\/tr>\n
Y-Mean<\/td>\n47.83<\/td>\n<\/tr>\n
X-SD (Standard Deviation)<\/td>\n16.76<\/td>\n<\/tr>\n
Y-SD<\/td>\n26.93<\/td>\n<\/tr>\n
Correlation<\/td>\n-0.06<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

If you\u2019re wondering why there is only one set of statistics, it\u2019s because they\u2019re all the same. Every single one of the 13 Charts<\/a> above has the same mean, standard deviation, and correlation between variables.<\/p>\n

This famous set of 13 data sets is known as the Datasaurus Dozen<\/em> [5], and was published some years ago as a stark example of why summary statistics cannot always be trusted. It also highlights the value of visualization as a tool for data exploration. In the words of renowned statistician John Tukey,<\/p>\n

\n

\u201cThe greatest value of a picture is when it forces us to notice what we never expected to see.<\/strong>\u201c<\/p>\n<\/blockquote>\n

Understanding Uncertainty<\/h2>\n

To conclude, I want to talk about a slight variation of deceptive data, but one that is equally important: mistrusting data that is actually correct.<\/strong> In other words, false deception.<\/p>\n

The following chart is taken from a study analyzing the sentiments of headlines taken from left-leaning, right-leaning, and centrist news outlets [6]:<\/p>\n

\"\"<\/a>
\u201cAverage yearly sentiment of headlines grouped by the ideological leanings of news outlets\u201d by Authors of the study: David Rozado, Ruth Hughes, Jamin Halberstadt is licensed under CC BY 4.0. To view a copy of this license, visit https:\/\/creativecommons.org\/licenses\/by\/4.0\/?ref=openverse.<\/figcaption><\/figure>\n

There is quite a bit going on in the chart above, but there is one particular aspect I want to draw your attention to: the vertical lines extending from each plotted point. You may have seen these before. Formally, these are called error bars<\/em>, and they are one way that scientists often depict uncertainty in the data.<\/p>\n

Let me say that again. In statistics and Data Science<\/a>, \u201cerror\u201d is synonymous with \u201cuncertainty.\u201d Crucially, it does not mean something is wrong or incorrect about what is being shown<\/strong>. When a chart depicts uncertainty, it depicts a carefully calculated measure of the range of a value and the level of confidence at various points within that range. Unfortunately, many people just take it to mean that whoever made the chart is essentially guessing.<\/p>\n

This is a serious error in reasoning, for the damage is twofold: Not only does the data at hand get misinterpreted, but the presence of this misconception also contributes to the dangerous societal belief that science is not to be trusted. Being upfront about the limitations of knowledge should actually increase our confidence in a claim\u2019s reliability, but mistaking that limitation as admission of foul play leads to the opposite effect.<\/p>\n

Learning how to interpret uncertainty is challenging but incredibly important. At the minimum, a good place to start is realizing what the so-called \u201cerror\u201d is actually trying to convey.<\/p>\n

Recap and Final Thoughts<\/h2>\n

Here\u2019s a cheat sheet for being wary of deceptive data:<\/p>\n