Reasoning Skills and Statistics Help (page 3)

Introduction to Reasoning Skills and Statistics

"He uses statistics as a drunken man uses lampposts—for support rather than for illumination."

—Andrew Lang, Scottish poet and novelist (1844–1912)

Lesson Summary

Statistics are often used to strengthen arguments—but they aren't always trustworthy. This lesson will show you how to judge the validity of statistics and how to make sure that any statistics you cite are credible.

There's strength in numbers. Whether on the battlefield or in the boardroom, the more people you have fighting for a cause, the more likely you are to win. There's strength in numbers in arguments, too—statistics generally carry more weight and sound more valid than opinions. That's because numbers look concrete, factual, and objective. But numbers are not always to be trusted. Like words, numbers can be—and often are—manipulated. As a critical thinker, you need to beware of the kinds of tricks numbers can play, and you need to know how to evaluate surveys, statistics, and other figures before you accept them as valid.

Consider the Source

One of your first priorities when you come across a figure or statistic is to consider the source. Where is this information coming from? You need to know the source so you can consider its credibility.

Figures are often cited without naming their source. This should automatically raise a red flag. When there's no source acknowledged, that figure could come from anywhere. Here's an example:

Eighty percent of all Americans believe that there is too much violence on television.

Our immediate reaction might be to say "Wow! Eighty percent! That's an impressive statistic." But because this claim does not indicate a source, you have to fight your instinct to accept the number as true. The question, "Who conducted this survey?" must be answered in order for you to be able to assess the validity of the figure. A figure that isn't backed by a credible source isn't worth much and can't be accepted with confidence. Unfortunately, you have to consider that the claimant could have made it up to give the appearance of statistical support for his argument.

If the claimant does provide a source, then the next step is to consider the credibility of that source. Remember, to determine credibility, look for evidence of bias and level of expertise.

Here's that statistic again attributed to two different sources:

1. According to Parents Against Television Violence (PATV), 80 percent of Americans believe that there is too much violence on TV.
2. According to a recent University of Minnesota survey, 80 percent of Americans believe there is too much violence on TV.

Would you accept the statistic as offered by source number 1? How about by source number 2?

While both sources may have a respectable level of expertise, it should be acknowledged that the people who conducted the university study probably have a higher level of expertise. More importantly, the source in number 1—Parents Against Television Violence—should encourage you to consider their statistics with caution. Is a group such as PATV likely to be biased in the issue of television violence? Absolutely. Is it possible, then, that such an organization could offer false or misleading statistics to support its cause? Yes. Would it be wise, therefore, to accept this statistic only with some reservations? Yes.

The university's study, however, is much more likely to have been conducted professionally and accurately. Scholarly research is subject to rigorous scrutiny by the academic community, so the university's findings are probably quite accurate and acceptable. There's less reason to suspect bias or sloppy statistical methods.

The Importance of Sample Size

In the ideal survey or opinion poll, everyone in the population in question would be surveyed. But since this is often impossible, researchers have to make do by interviewing a sample of the population. Unfortunately, this means that their results do not always reflect the sentiment of the entire population.

Obviously, the larger the sample size, the more reflective the survey will be of the entire population. For example, let's say you want to know how parents of children in grades 6–9 in Pennsylvania public schools feel about removing vending machines from school cafeterias. If there are two million parents that fall into this category, how many should you survey? Two? Two hundred? Two thousand? Twenty thousand? Two hundred thousand?

Indeed, how many people you survey depends upon the time and money you have to invest in the survey. But under no circumstances would surveying two or two hundred people be sufficient—these numbers represent far too small a percentage of the population that you're surveying. Twenty thousand is a much better sample, although it constitutes only one percent of the population you are trying to reach. Two hundred thousand, on the other hand, reaches ten percent of the population, making it much more likely that the results of your survey accurately reflect the population as a whole.

On CNN's online site, a poll is taken several times a day on different topics. Readers can vote on an issue and look to see how many others have voted also. In this manner, CNN lets you know the exact sample size. This practice helps make the reported results more credible and enables you to judge for yourself whether a sample is large enough to be representative of the sentiments of the entire country.

You're probably wondering how much is enough when it comes to sample size. There's no hard and fast rule here except one: The larger your sample size, the better. The bigger the sample, the more likely it is that your survey results will accurately reflect the opinions of the population in question.

Tip

Sample size is important in surveys. A reminder of this can be found on any nutritional label. Pick up a box of your favorite chips or cookies, for example. Check out how many grams of carbohydrates it has. How much sugar? Protein? Salt? Now, before you think you will buy one product over another, check out the serving size. Are those numbers for three chips or a dozen? One cookie or four? It makes a big difference in food and in reasoning.

Representative, Random, and Biased Samples

Let's say you want to conduct the "tuition/sports arena" survey but don't have any budget. Since you are on a tennis team with 50 players, you decide to simply poll the players on your team. Will your results accurately reflect the sentiment on your campus?

Regardless of how the players feel about this issue, it'd be nearly impossible for your survey results to accurately reflect the sentiments of the student body. Why? Because your sample is not representative of the population whose opinion you wish to reflect. In order for your sample to be representative, it should include all the various groups and subgroups within the student population. That is, the people in your sample group should represent the people in the whole group. That means, for one thing, that you need to survey players from several different sports teams, not just yours. In addition, your sample group needs to include members from all different campus organizations—student government, sororities, political groups, various clubs, and so on.

Furthermore, the sample should include respondents from these groups in approximately the same proportion that you would find them on campus. That is, if 50 percent of the students belong to fraternities or sororities, then approximately 50 percent of your respondents should be members of fraternities or sororities. If 20 percent are members of an athletic group, then approximately 20 percent of your respondents should be athletes, and so on. In this way, your survey results are more likely to be proportionate to the results you'd get if you were able to survey everyone on campus.

But how do you get a representative sample for larger populations such as two million parents or one billion Chinese? Because the range of respondents is so wide, your best bet is to get a random sample. By randomly selecting participants, you have the best chance of getting a representative sample because each person in the population has the same chance of being surveyed. Representative and random samples help prevent you from having a biased sample. Imagine you read the following:

In a survey of 6,000 city residents, 79 percent of the respondents say that the Republican mayor has done an outstanding job.

This claim tells us the sample size—6,000—which is a substantive number. But it doesn't tell how the 6,000 residents were chosen to answer the survey. Because the political affiliation and socioeconomic standing of the respondents could greatly influence the results of the survey, it is important to know if those 6,000 people are varied enough to accurately reflect the sentiment of an entire city.

For example, if all of those 6,000 surveyed were Republicans, of course the percentage of favorable votes would be high; but that doesn't tell much about how people from other political parties feel. Survey another 6,000 residents who are Democrats and you'd come up with a much, much lower number. Why? Because members of this sample group might be biased against a Republican mayor. Thus, it's critical that the sample be as representative as possible, including both Democrats and Republicans, the wealthy and the poor.

How do you know, though, that a survey has used a representative sample? Surveys that have been conducted legitimately will generally be careful to provide you with information about the sample size and population so that their results are more credible to you. You might see something like the following, for example:

• In a recent survey, 500 random shoppers were asked whether they felt the Food Court in the mall provides a sufficient selection.
• A survey of 3,000 men between the ages of 18 and 21 found that 72 percent think either that the drinking age should be lowered to 18 or that the draft age should be raised to 21.

Notice how these claims let you know exactly who was surveyed.

Special Note

Beware of call-in surveys and polls that are conducted by mail or that otherwise depend upon the respondents to take action. Results of these surveys tend to be misleading because those who take the time to return mail-in surveys or make the effort to call, fax, or e-mail a response are often people who feel very strongly about the issue. To assume that the opinions of those people who feel strongly about the issue represent how the entire population feels is risky because it's not very likely that most people in the population feel that way.