Manipulating Statistics Study Guide (page 2)
Figures don't lie, but liars figure.
Mark Twain, American writer and humorist (1835–1910)
Numbers are facts, so numbers are always true, right? Well, not always. Sometimes people use numerical information incorrectly, either innocently or with a motive to mislead. In this lesson, we'll explore some common ways numbers are misused, including incorrectly gathering the figures, drawing the wrong conclusion, and misrepresenting the data.
We're bombarded with facts and figures every day—numerical information about what's going on in the world, who we should vote for, what we should buy, and even what we should think. The problem is, facts and figures aren't always factual. You've probably heard the old saying, "numbers don't lie." Well, they do, or rather the people who use them do!
Numbers are manipulated all the time, whether by deliberate misuse, negligence, or plain incompetence, so that what we see, hear, and read isn't always the truth. If we rely on numbers in statistics, polls, or percentages as a basis for decisions and opinions, we could be making a serious mistake. After all, people who work with numbers and those who analyze or interpret them are people. They may be biased, less than competent, or negligent, so you have to be just as concerned with the sources and quality of numbers as you are with words.
Numbers can be misused. It all happens in one, or both, of two key areas. First, numbers must be gathered. If they are collected incorrectly, or by someone with an agenda or bias, you need to know that. Second, numbers must be analyzed or interpreted. Again, this process can be done incorrectly, or misused by an individual or group. Once you learn what to look for in these two areas, you can evaluate the numerical data you encounter and rely on it only when it is objective and correct.
Authors, advertisers, businesses, and politicians rely on surveys, polls, and other statistics to make their points of view appear more credible and important. The problem is, it's just as easy to mislead with numbers as with words. Numbers must be gathered correctly so they can be trusted. Here are a few examples, including how numbers are manipulated so they can't be trusted.
- Use an appropriate sample population that is
- large enough—if the sample number is too low, it won't be representative of a larger population; asking just two people if they like a new ice cream flavor and finding that one person does doesn't mean that 50% of all ice cream eaters, a number in the millions, will like the flavor.
- similar to the target population—if the target population includes ages 10–60, your sample can't be taken just from a junior high school
- random—asking only union members about labor laws is not random; asking one hundred people whose phone numbers were picked by a computer is
- Remain un-biased. Ask objective questions and create a non-threatening, non-influencing atmosphere. Compare, "Do you think people should be allowed to own dangerous firearms if they have innocent young children at home?" to "Do you think people should be allowed to exercise their Second Amendment right to own a firearm?" Also, if the person asking the question is wearing a "Gun Control Now!" or "Gun Freedom Now!" button, his or her bias pollutes the environment and will influence the answers received.
Imagine an ad that reads, "Eighty percent of respondents in a recent survey liked Smilebright toothpaste better than Brightsmile." The high percentage is meant to convince readers that most people prefer Smilebright, so you will, too. But how was that percentage figured? The survey consisted of asking five people, who already said they preferred a gel-type toothpaste, whether they liked Smilebright or Brightsmile. There was no random sampling—everyone had the same preference, which is probably not true for a larger population.
Remember, surveys can't prove anything with 100% certainty unless the sample questions 100% of the population.
Margin of Error
Most survey results end with a statement such as "there is a margin of error of three percentage points." What does this mean? It tells how confident the surveyors are that their results are correct. The lower the percentage, the greater their confidence. A 3% margin of error means that the sample population of the survey could be different from the general population by 3% in either direction. Let's say a survey concluded that "55% of Americans want to vote for members of the Supreme Court." If there is a 3% margin of error, the results could be 58%, 52%, or anywhere in between, if you conducted the identical survey asking another group of people.
Knowing the margin of error is important, especially in political polls. Imagine a headline that reads, "Smith's lead slips to 58%; Manotti gaining momentum at 37%." The accompanying article states that last week, the results were 61% to 34%, with a 4% margin of error. That means there's really no difference between the two polls. No one is "slipping" or "gaining momentum." The margin of error tells the real story; the news article is wrong.
Once numbers are gathered, they must be interpreted or evaluated, and this step affords many opportunities to distort the truth. For example, researchers often do correlation studies to find out if a link exists between two sets of data. Here are two questions someone might use for a correlation study:
- Is there a connection between the full moon and an increase in birth rates?
- Does having a high IQ indicate that you will have a high income level?
Imagine that research at five area hospitals shows that during a full moon, an average of 4% more babies are born than on nights with no full moon. You could then say there's a small but positive correlation between full moons and birth rates. But many studies have shown that any correlation is really due to chance. No evidence has been found to support the theory that the moon's phases affect human behavior in any way. So, even though you found a positive correlation, it doesn't necessarily mean there's a cause-and-effect relationship between the two elements in the study.
For the second question, if a study showed that Americans with the top 5% of IQ scores made an average of $22,000 a year, while those in the middle 5% made an average of $40,000, you would say there is a negative correlation between IQ and income levels. To describe the results of the study, you could say that there is no evidence that IQ determines income level. In other words, you do not need to have a high IQ to make a lot of money.
This conclusion is obvious. But let's look at how these same correlation study results can be used to come up with a ridiculous conclusion. The second example shows that there is no connection between a high IQ and a high income level. Is that the same as saying that "the dumber you are, the more money you will make?" Of course it isn't. This type of conclusion shows one of the dangers of correlation studies. Even if the study uses accurate data, the way in which it is interpreted can be wrong, and even foolish. When you encounter a correlation study, as with survey and poll results, do not assume the numbers and conclusion are correct. Ask questions, and look at supporting data. Does the study make sense? Or does it seem too convenient for the advertiser/politician/reporter/author who is using it? Think critically, and do not rely on anyone's numbers until you determine they are true and valid.