Collecting Data: Surveys, Experiments, Observational Studies for AP Statistics
In the preceding section, we discussed data analysis and inferential statistics. A question not considered in many introductory statistics courses (but considered in detail in AP Statistics) is how the data are collected. Often times we are interested in collecting data in order to make generalizations about a population. One way to do this is to conduct a survey. In a well-designed survey, you take a random sample of the population of interest, compute statistics of interest (like the proportion of baseball fans in the sample who think Pete Rose should be in the Hall of Fame), and use those to make predictions about the population.
We are often more interested in seeing the reactions of persons or things to certain stimuli. If so, we are likely to conduct an experiment or an observational study. We discuss the differences between these two types of studies in Chapter 8, but both basically involve collecting comparative data on groups (called treatment and control) constructed in such a way that the only difference between the groups (we hope) is the focus of the study. Because experiments and observational studies are usually done on volunteers, rather than on random samples from some population of interest (it's been said that most experiments are done on graduate students in psychology), the results of such studies may lack generalizability to larger populations. Our ability to generalize involves the degree to which we are convinced that the only difference between our groups is the variable we are studying (otherwise some other variable could be producing the responses).
It is extremely important to understand that data must be gathered correctly in order to have analysis and inference be meaningful. You can do all the number crunching you want with bad data, but the results will be meaningless.
In 1936, the magazine The Literary Digest did a survey of some 10 million people in an effort to predict the winner of the presidential election that year. They predicted that Alf Landon would defeat Franklin Roosevelt by a landslide, but the election turned out just the opposite. The Digest had correctly predicted the outcome of the preceding five presidential elections using similar procedures, so this was definitely unexpected. Its problem was not in the size of the sample it based its conclusions on. Its problem was in the way it collected its data—the Digest simply failed to gather a random sample of the population. It turns out that its sampling frame (the population from which it drew its sample) was composed of a majority of Republicans. The data were extensive (some 2.4 million ballots were returned), but they weren't representative of the voting population. In part because of the fallout from this fiasco, the Digest went bankrupt and out of business the following year. If you are wondering why the Digest was wrong this time with essentially the same techniques used in earlier years, understand that 1936 was the heart of the Depression. In earlier years the lists used to select the sample may have been more reflective of the voting public, but in 1936 only the well-to-do, Republicans generally, were in the Digest's sample taken from its own subscriber lists, telephone books, etc.
We look more carefully at sources of bias in data collection in Chapter 8, but the point you need to remember as you progress through the next couple of chapters is that conclusions based on data are only meaningful to the extent that the data are representative of the population being studied.
In an experiment or an observational study, the analogous issue to a biased sample in a survey is the danger of treatment and control groups being somehow systematically different. For example, suppose we wish to study the effects of exercise on stress reduction. We let 100 volunteers for the study decide if they want to be in the group that exercises or in the group that doesn't. There are many reasons why one group might be systematically different from the other, but the point is that any comparisons between these two groups is confounded by the fact that the two groups could be different in substantive ways.
Today on Education.com
- Kindergarten Sight Words List
- Signs Your Child Might Have Asperger's Syndrome
- Coats and Car Seats: A Lethal Combination?
- Child Development Theories
- GED Math Practice Test 1
- The Homework Debate
- Graduation Inspiration: Top 10 Graduation Quotes
- Social Cognitive Theory
- 10 Fun Activities for Children with Autism
- First Grade Sight Words List