Manipulating Statistics Study Guide (page 3)
Figures don't lie, but liars figure.
Mark Twain, American writer and humorist (1835–1910)
Numbers are facts, so numbers are always true, right? Well, not always. Sometimes people use numerical information incorrectly, either innocently or with a motive to mislead. In this lesson, we'll explore some common ways numbers are misused, including incorrectly gathering the figures, drawing the wrong conclusion, and misrepresenting the data.
We're bombarded with facts and figures every day—numerical information about what's going on in the world, who we should vote for, what we should buy, and even what we should think. The problem is, facts and figures aren't always factual. You've probably heard the old saying, "numbers don't lie." Well, they do, or rather the people who use them do!
Numbers are manipulated all the time, whether by deliberate misuse, negligence, or plain incompetence, so that what we see, hear, and read isn't always the truth. If we rely on numbers in statistics, polls, or percentages as a basis for decisions and opinions, we could be making a serious mistake. After all, people who work with numbers and those who analyze or interpret them are people. They may be biased, less than competent, or negligent, so you have to be just as concerned with the sources and quality of numbers as you are with words.
Numbers can be misused. It all happens in one, or both, of two key areas. First, numbers must be gathered. If they are collected incorrectly, or by someone with an agenda or bias, you need to know that. Second, numbers must be analyzed or interpreted. Again, this process can be done incorrectly, or misused by an individual or group. Once you learn what to look for in these two areas, you can evaluate the numerical data you encounter and rely on it only when it is objective and correct.
Authors, advertisers, businesses, and politicians rely on surveys, polls, and other statistics to make their points of view appear more credible and important. The problem is, it's just as easy to mislead with numbers as with words. Numbers must be gathered correctly so they can be trusted. Here are a few examples, including how numbers are manipulated so they can't be trusted.
- Use an appropriate sample population that is
- large enough—if the sample number is too low, it won't be representative of a larger population; asking just two people if they like a new ice cream flavor and finding that one person does doesn't mean that 50% of all ice cream eaters, a number in the millions, will like the flavor.
- similar to the target population—if the target population includes ages 10–60, your sample can't be taken just from a junior high school
- random—asking only union members about labor laws is not random; asking one hundred people whose phone numbers were picked by a computer is
- Remain un-biased. Ask objective questions and create a non-threatening, non-influencing atmosphere. Compare, "Do you think people should be allowed to own dangerous firearms if they have innocent young children at home?" to "Do you think people should be allowed to exercise their Second Amendment right to own a firearm?" Also, if the person asking the question is wearing a "Gun Control Now!" or "Gun Freedom Now!" button, his or her bias pollutes the environment and will influence the answers received.
Imagine an ad that reads, "Eighty percent of respondents in a recent survey liked Smilebright toothpaste better than Brightsmile." The high percentage is meant to convince readers that most people prefer Smilebright, so you will, too. But how was that percentage figured? The survey consisted of asking five people, who already said they preferred a gel-type toothpaste, whether they liked Smilebright or Brightsmile. There was no random sampling—everyone had the same preference, which is probably not true for a larger population.
Remember, surveys can't prove anything with 100% certainty unless the sample questions 100% of the population.
Margin of Error
Most survey results end with a statement such as "there is a margin of error of three percentage points." What does this mean? It tells how confident the surveyors are that their results are correct. The lower the percentage, the greater their confidence. A 3% margin of error means that the sample population of the survey could be different from the general population by 3% in either direction. Let's say a survey concluded that "55% of Americans want to vote for members of the Supreme Court." If there is a 3% margin of error, the results could be 58%, 52%, or anywhere in between, if you conducted the identical survey asking another group of people.
Knowing the margin of error is important, especially in political polls. Imagine a headline that reads, "Smith's lead slips to 58%; Manotti gaining momentum at 37%." The accompanying article states that last week, the results were 61% to 34%, with a 4% margin of error. That means there's really no difference between the two polls. No one is "slipping" or "gaining momentum." The margin of error tells the real story; the news article is wrong.
Once numbers are gathered, they must be interpreted or evaluated, and this step affords many opportunities to distort the truth. For example, researchers often do correlation studies to find out if a link exists between two sets of data. Here are two questions someone might use for a correlation study:
- Is there a connection between the full moon and an increase in birth rates?
- Does having a high IQ indicate that you will have a high income level?
Imagine that research at five area hospitals shows that during a full moon, an average of 4% more babies are born than on nights with no full moon. You could then say there's a small but positive correlation between full moons and birth rates. But many studies have shown that any correlation is really due to chance. No evidence has been found to support the theory that the moon's phases affect human behavior in any way. So, even though you found a positive correlation, it doesn't necessarily mean there's a cause-and-effect relationship between the two elements in the study.
For the second question, if a study showed that Americans with the top 5% of IQ scores made an average of $22,000 a year, while those in the middle 5% made an average of $40,000, you would say there is a negative correlation between IQ and income levels. To describe the results of the study, you could say that there is no evidence that IQ determines income level. In other words, you do not need to have a high IQ to make a lot of money.
This conclusion is obvious. But let's look at how these same correlation study results can be used to come up with a ridiculous conclusion. The second example shows that there is no connection between a high IQ and a high income level. Is that the same as saying that "the dumber you are, the more money you will make?" Of course it isn't. This type of conclusion shows one of the dangers of correlation studies. Even if the study uses accurate data, the way in which it is interpreted can be wrong, and even foolish. When you encounter a correlation study, as with survey and poll results, do not assume the numbers and conclusion are correct. Ask questions, and look at supporting data. Does the study make sense? Or does it seem too convenient for the advertiser/politician/reporter/author who is using it? Think critically, and do not rely on anyone's numbers until you determine they are true and valid.
Statistics is simply a mathematical science that gathers information about a population so that population may be described usefully. Statistics are often used to draw conclusions and make decisions based on that information. So, what's the problem?
Statistics are complicated and their problems can be numerous. In general, though, problems with statistics are similar to those of other types of numerical data; namely, they can be gathered, analyzed, and/or interpreted incorrectly, or mishandled by someone with a bias. Let's look at two common problems with statistics.
- Is it meaningful? Many parents worry when they hear that the average baby walks at 13 months. They may conclude that something must be wrong with their 18-month-old who is still crawling. But, studies have proven that at age two, there are no developmental differences between early walkers and babies who walk later. So the statistic is not meaningful; there is nothing abnormal about an 18-month-old who still crawls. Here is another example. When national standardized test scores were analyzed, it was concluded that since students from wealthy communities had higher scores, they were smarter than students in poorer communities. Is that a meaningful, accurate conclusion? Probably not; it doesn't take into account the many other variables that can explain lower test scores, such as teacher quality, less test preparation, fatigue, and even breakfast on the day of testing.
Always check the source of data. Is the source credible? Look for another source that gives similar data to strengthen the credibility.
Never trust numbers alone to prove a point. Ask yourself, "Why should I believe these statistics?" Always look for some other kind of evidence or an example to reinforce the figures.
It is just as easy to deceive with numbers as it is with words. Surveys, studies, and statistics are conducted and interpreted by researchers who might have a bias, or simply lack the skills necessary to do their jobs properly. Therefore, it is important to evaluate numbers before accepting them as truth. Ask questions about how the information was gathered, what its margin of error is, and how meaningful it is. Does the conclusion make sense, or does it seem to distort the findings? Thinking critically about the many numbers you encounter will help you to rely only on information that is objective and accurate.
Skill Building Until Next Time
- Watch a news broadcast and listen for the results of a survey or poll. Does the newscaster tell the margin of error? Why is it important to know this number?
- Look for a print or article that includes a statistic. Why was it included? Does it seem accurate and objective? How else could the advertiser or writer have made the point without using numbers?
Exercises for this concept can be found at Manipulating Statistics Practice Exercises.
Add your own comment
Today on Education.com
- Kindergarten Sight Words List
- The Five Warning Signs of Asperger's Syndrome
- What Makes a School Effective?
- Child Development Theories
- Why is Play Important? Social and Emotional Development, Physical Development, Creative Development
- 10 Fun Activities for Children with Autism
- Test Problems: Seven Reasons Why Standardized Tests Are Not Working
- Bullying in Schools
- A Teacher's Guide to Differentiating Instruction
- Steps in the IEP Process