Statistical Significance for AP Statistics

By — McGraw-Hill Professional
Updated on Feb 5, 2011

Practice problems for these concepts can be found at:

Statistical Significance

One of the desirable outcomes of a study is to help us determine cause and effect. We do this by looking for differences between groups that are so great that we cannot reasonably attribute the difference to chance. We say that a difference between what we would expect to find if there were no treatment and what we actually found is statistically significant if the difference is too great to attribute to chance.

An experiment is a study in which the researcher imposes some sort of treatment on the experimental units (which can be human—usually called subjects in that case). In an experiment, the idea is to determine the extent to which treatments (the explanatory variable(s)) affects outcomes (the response variable (s)). For example, a researcher might vary the rewards to different work group members to see how that affects the group's ability to perform a particular task.

An observational study, on the other hand, simply observes and records behavior but does not attempt to impose a treatment in order to manipulate the response.

    example: A group of 60 exercisers are classified as "walkers" or "runners." A longitudinal study (one conducted over time) is conducted to see if there are differences between the groups in terms of their scores on a wellness index. This is an observational study because, although the two groups differ in an important respect, the researcher is not manipulating any treatment. "Walkers" and "runners" are simply observed and measured. Note that the groups in this study are self-selected. That is, they were already in their groups before the study began. The researchers just noted their group membership and proceeded to make observations. There may be significant differences between the groups in addition to the variable under study.
    example: A group of 60 volunteers who do not exercise are randomly assigned to one of the two fitness programs. One group of 30 is enrolled in a daily walking program, and the other group is put into a running program. After a period of time, the two groups are compared based on their scores on a wellness index. This is an experiment because the researcher has imposed the treatment (walking or running) and then measured the effects of the treatment on a defined response.

It may be, even in a controlled experiment, that the measured response is a function of variables present in addition to the treatment variable. A confounding variable is one that has an effect on the outcomes of the study but whose effects cannot be separated from those of the treatment variable. A lurking variable is one that has an effect on the outcomes of the study but whose influence was not part of the investigation. A lurking variable can be a confounding variable.

    example: A study is conducted to see if Yummy Kibble dog food results in shinier coats on Golden Retrievers. It's possible that the dogs with shinier coats have them because they have owners who are more conscientious in terms of grooming their pets. Both the dog food and the conscientious owners could contribute to the shinier coats. The variables are confounded because their effects cannot be separated.

A well-designed study attempts to anticipate confounding variables in advance and control for them. Statistical control refers to a researcher holding constant variables not under study that might have an influence on the outcomes.

    example: You are going to study the effectiveness of SAT preparation courses on SAT score. You know that better students tend to do well on SAT tests. You could control for the possible confounding effect of academic quality by running your study with groups of "A" students, "B" students, etc.

Control is often considered to be one of the three basic principles of experimental design. The other two basic principles are randomization and replication.

The purpose of randomization is to equalize groups so that the effects of lurking variables are equalized among groups. Randomization involves the use of chance (like a coin flip) to assign subjects to treatment and control groups. The hope is that the groups being studied will differ systematically only in the effects of the treatment variable. Although individuals within the groups may vary, the idea is to make the groups as alike as possible except for the treatment variable. Note that it isn't possible to produce, with certainty, groups free of any lurking variables. It is possible, through the use of randomization, to increase the probability of producing groups that are alike. The idea is to control for the effects of variables you aren't aware of but that might affect the response.

Replication involves repeating the experiment on enough subjects (or units) to reduce the effects of chance variation on the outcomes. For example, we know that the number of boys and girls born in a year are approximately equal. A small hospital with only 10 births a year is much more likely to vary dramatically from 50% each than a large hospital with 500 births a year.

Practice problems for these concepts can be found at:

Add your own comment

Ask a Question

Have questions about this article or topic? Ask
150 Characters allowed