Reasoning Skills and Statistics Help (page 3)
Introduction to Reasoning Skills and Statistics
"He uses statistics as a drunken man uses lampposts—for support rather than for illumination."
—Andrew Lang, Scottish poet and novelist (1844–1912)
Statistics are often used to strengthen arguments—but they aren't always trustworthy. This lesson will show you how to judge the validity of statistics and how to make sure that any statistics you cite are credible.
There's strength in numbers. Whether on the battlefield or in the boardroom, the more people you have fighting for a cause, the more likely you are to win. There's strength in numbers in arguments, too—statistics generally carry more weight and sound more valid than opinions. That's because numbers look concrete, factual, and objective. But numbers are not always to be trusted. Like words, numbers can be—and often are—manipulated. As a critical thinker, you need to beware of the kinds of tricks numbers can play, and you need to know how to evaluate surveys, statistics, and other figures before you accept them as valid.
Consider the Source
One of your first priorities when you come across a figure or statistic is to consider the source. Where is this information coming from? You need to know the source so you can consider its credibility.
Figures are often cited without naming their source. This should automatically raise a red flag. When there's no source acknowledged, that figure could come from anywhere. Here's an example:
Eighty percent of all Americans believe that there is too much violence on television.
Our immediate reaction might be to say "Wow! Eighty percent! That's an impressive statistic." But because this claim does not indicate a source, you have to fight your instinct to accept the number as true. The question, "Who conducted this survey?" must be answered in order for you to be able to assess the validity of the figure. A figure that isn't backed by a credible source isn't worth much and can't be accepted with confidence. Unfortunately, you have to consider that the claimant could have made it up to give the appearance of statistical support for his argument.
If the claimant does provide a source, then the next step is to consider the credibility of that source. Remember, to determine credibility, look for evidence of bias and level of expertise.
Here's that statistic again attributed to two different sources:
- According to Parents Against Television Violence (PATV), 80 percent of Americans believe that there is too much violence on TV.
- According to a recent University of Minnesota survey, 80 percent of Americans believe there is too much violence on TV.
Would you accept the statistic as offered by source number 1? How about by source number 2?
While both sources may have a respectable level of expertise, it should be acknowledged that the people who conducted the university study probably have a higher level of expertise. More importantly, the source in number 1—Parents Against Television Violence—should encourage you to consider their statistics with caution. Is a group such as PATV likely to be biased in the issue of television violence? Absolutely. Is it possible, then, that such an organization could offer false or misleading statistics to support its cause? Yes. Would it be wise, therefore, to accept this statistic only with some reservations? Yes.
The university's study, however, is much more likely to have been conducted professionally and accurately. Scholarly research is subject to rigorous scrutiny by the academic community, so the university's findings are probably quite accurate and acceptable. There's less reason to suspect bias or sloppy statistical methods.
The Importance of Sample Size
In the ideal survey or opinion poll, everyone in the population in question would be surveyed. But since this is often impossible, researchers have to make do by interviewing a sample of the population. Unfortunately, this means that their results do not always reflect the sentiment of the entire population.
Obviously, the larger the sample size, the more reflective the survey will be of the entire population. For example, let's say you want to know how parents of children in grades 6–9 in Pennsylvania public schools feel about removing vending machines from school cafeterias. If there are two million parents that fall into this category, how many should you survey? Two? Two hundred? Two thousand? Twenty thousand? Two hundred thousand?
Indeed, how many people you survey depends upon the time and money you have to invest in the survey. But under no circumstances would surveying two or two hundred people be sufficient—these numbers represent far too small a percentage of the population that you're surveying. Twenty thousand is a much better sample, although it constitutes only one percent of the population you are trying to reach. Two hundred thousand, on the other hand, reaches ten percent of the population, making it much more likely that the results of your survey accurately reflect the population as a whole.
On CNN's online site, a poll is taken several times a day on different topics. Readers can vote on an issue and look to see how many others have voted also. In this manner, CNN lets you know the exact sample size. This practice helps make the reported results more credible and enables you to judge for yourself whether a sample is large enough to be representative of the sentiments of the entire country.
You're probably wondering how much is enough when it comes to sample size. There's no hard and fast rule here except one: The larger your sample size, the better. The bigger the sample, the more likely it is that your survey results will accurately reflect the opinions of the population in question.
Sample size is important in surveys. A reminder of this can be found on any nutritional label. Pick up a box of your favorite chips or cookies, for example. Check out how many grams of carbohydrates it has. How much sugar? Protein? Salt? Now, before you think you will buy one product over another, check out the serving size. Are those numbers for three chips or a dozen? One cookie or four? It makes a big difference in food and in reasoning.
Representative, Random, and Biased Samples
Let's say you want to conduct the "tuition/sports arena" survey but don't have any budget. Since you are on a tennis team with 50 players, you decide to simply poll the players on your team. Will your results accurately reflect the sentiment on your campus?
Regardless of how the players feel about this issue, it'd be nearly impossible for your survey results to accurately reflect the sentiments of the student body. Why? Because your sample is not representative of the population whose opinion you wish to reflect. In order for your sample to be representative, it should include all the various groups and subgroups within the student population. That is, the people in your sample group should represent the people in the whole group. That means, for one thing, that you need to survey players from several different sports teams, not just yours. In addition, your sample group needs to include members from all different campus organizations—student government, sororities, political groups, various clubs, and so on.
Furthermore, the sample should include respondents from these groups in approximately the same proportion that you would find them on campus. That is, if 50 percent of the students belong to fraternities or sororities, then approximately 50 percent of your respondents should be members of fraternities or sororities. If 20 percent are members of an athletic group, then approximately 20 percent of your respondents should be athletes, and so on. In this way, your survey results are more likely to be proportionate to the results you'd get if you were able to survey everyone on campus.
But how do you get a representative sample for larger populations such as two million parents or one billion Chinese? Because the range of respondents is so wide, your best bet is to get a random sample. By randomly selecting participants, you have the best chance of getting a representative sample because each person in the population has the same chance of being surveyed. Representative and random samples help prevent you from having a biased sample. Imagine you read the following:
In a survey of 6,000 city residents, 79 percent of the respondents say that the Republican mayor has done an outstanding job.
This claim tells us the sample size—6,000—which is a substantive number. But it doesn't tell how the 6,000 residents were chosen to answer the survey. Because the political affiliation and socioeconomic standing of the respondents could greatly influence the results of the survey, it is important to know if those 6,000 people are varied enough to accurately reflect the sentiment of an entire city.
For example, if all of those 6,000 surveyed were Republicans, of course the percentage of favorable votes would be high; but that doesn't tell much about how people from other political parties feel. Survey another 6,000 residents who are Democrats and you'd come up with a much, much lower number. Why? Because members of this sample group might be biased against a Republican mayor. Thus, it's critical that the sample be as representative as possible, including both Democrats and Republicans, the wealthy and the poor.
How do you know, though, that a survey has used a representative sample? Surveys that have been conducted legitimately will generally be careful to provide you with information about the sample size and population so that their results are more credible to you. You might see something like the following, for example:
- In a recent survey, 500 random shoppers were asked whether they felt the Food Court in the mall provides a sufficient selection.
- A survey of 3,000 men between the ages of 18 and 21 found that 72 percent think either that the drinking age should be lowered to 18 or that the draft age should be raised to 21.
Notice how these claims let you know exactly who was surveyed.
Beware of call-in surveys and polls that are conducted by mail or that otherwise depend upon the respondents to take action. Results of these surveys tend to be misleading because those who take the time to return mail-in surveys or make the effort to call, fax, or e-mail a response are often people who feel very strongly about the issue. To assume that the opinions of those people who feel strongly about the issue represent how the entire population feels is risky because it's not very likely that most people in the population feel that way.
Comparing Apples and Oranges
In 1972, a Hershey's chocolate bar cost only 5 cents. Today, the same bar costs at least 50 cents. That's an increase of over 1,000 percent!
This increase sounds extreme, doesn't it? But is it really as severe as the math makes it seem? Not quite.
The problem with this claim is that while the actual price of a Hershey's bar may have increased 1,000 percent, it's not a fair comparison. That's because 5 cents in 1972 had more market value than 5 cents today. In this situation, the actual costs can't legitimately be compared. Instead, the costs have to be compared after they've been adjusted for inflation. Because there has been such a long time span and the value of the dollar has declined in the last 30 years, maybe 50 cents today is actually cheaper than 5 cents was in 1972.
It's important, therefore, to analyze comparisons like this to be sure the statistics are indeed comparable. Any monetary comparison needs to take into consideration market value and inflation. When dealing with figures other than money, however, there are other important concerns. For example, read the following argument:
In 1990, there were 100 unemployed people in Boone County. In 2000, there were 250. That's an increase of 150 percent in just ten years. Unemployment in this country is becoming an epidemic!
What's wrong with this argument? Clearly, there has been a sharp rise in unemployment in the last decade. But what the claim doesn't tell you is that during that same time period, the population of Boone County increased by 250 percent. Now how does that affect the argument?
If the population increased from 100,000 to 350,000, is the rise in unemployment still evidence that can be used to support the claim "Unemployment in this country is becoming an epidemic"? No. In fact, this means that that the number of unemployed per capita (that is, per person) has actually decreased. This is a case of comparing apples to oranges because the population in 1990 was so different than the population in 2000.
You should beware of any comparison across time, but the same problems can arise in contemporary comparisons. Take the following statistic, for example:
Charleston Medical Center physicians perform more arthroscopic knee operations than St. Francis physicians, who use a technique that requires a large incision.
If you need to have knee surgery, should you go to Charleston Medical Center? Not necessarily. Consider this fact, first: St. Francis physicians specialize in complicated knee surgeries that cannot be performed arthroscopically. Because their pool of patients is different from those of Charleston Medical Center, so will the number of nonarthroscopic knee operations.
Numbers don't lie … but resources can. When putting together statistics to use in your argument or to analyze in someone else's, make sure that they come from sources that you can trust. Your mother's guess on the current unemployment rate or your younger brother's estimate on how many days it has rained this year may be convenient to obtain but not likely very accurate. Take the time to get numbers from reliable sources.
Reasoning Skills and Statistics In Short
The truth about statistics is that they can be very misleading. When you come across statistics, check the source to see whether or not it's credible. Then find out the sample size and decide whether it's substantial enough. Look for evidence that the sample is representative of the population whose opinion you wish to reflect, or randomly selected and not biased. Finally, beware of statistics that compare apples to oranges by putting two unequal items side by side.
Skill Building until Next Time
- Look for survey results in a reputable newspaper with a national circulation, like The New York Times, Washington Post, or San Francisco Chronicle. Notice how much information they provide about how the survey was conducted. Then, look for survey results in a tabloid or a less credible source. Notice how little information is provided and check for the possibility of bias.
- Think about a survey that you would like to conduct. Who is your target population? How would you ensure a representative sample? How large should your sample be?
Exercises for this concept can be found at Reasoning Skills and Statistics Practice.
Add your own comment
Local SAT & ACT Classes
- Kindergarten Sight Words List
- The Five Warning Signs of Asperger's Syndrome
- What Makes a School Effective?
- Child Development Theories
- Why is Play Important? Social and Emotional Development, Physical Development, Creative Development
- 10 Fun Activities for Children with Autism
- Test Problems: Seven Reasons Why Standardized Tests Are Not Working
- Bullying in Schools
- A Teacher's Guide to Differentiating Instruction
- Steps in the IEP Process