Conducting a Study and Statistics Planning Study Guide

Analyzing the Data

Before data are collected, the analysis should be outlined. With the analysis and potential conclusions in mind, the research question should be reviewed to confirm that the planned study has the potential of answering the question. Too often, studies are conducted before the researchers realize they have no idea how to analyze the data or that the collected data cannot be used to answer the research question. The statistician should verify that the data collection protocol was properly followed. Each analysis should begin by summarizing the data graphically and numerically. Then the appropriate statistical analyses should be conducted.

Answering the Question

Through interpretation of the analysis results, we learn what conclusions can be drawn from the study. The aim is to answer the research question using the conclusions drawn from the study. Sometimes, we are unable to answer the question or are able to only partially answer it. At the conclusion of any study, the research team should reflect on what was learned from the study and use that to direct future research.

Selecting the Sample

Most of the inferential methods introduced in the text are based on random selection. The simplest form of random selection is simple random sampling. A simple random sample of size n is one drawn in such a manner that every possible sample of size n has an equal chance of being chosen.

It is important to realize that, if every unit in the population has an equal chance of being included in a sample, the sample may still not be a simple random one. To see this, suppose that a company has two divisions, A and B. There are 700 employees in division A and 300 in division B. The management decides to take a sample of 100 employees. To do this, they write each employee's name on a chip and put the chip in bowl A or B, depending on whether the employee is in division A or division B, respectively. The chips are thoroughly mixed in each bowl. Seventy chips are drawn from bowl A and 30 chips are drawn from bowl B, and the employees whose names are on the selected chips comprise the sample. Each employee has a 1 in 100 chance of being included in the sample; however, this is not a random sample.

Only samples with 70 division-A employees and 30 division-B employees are possible; it would not be possible to have, for instance, a sample with 50 division-A employees and 50 division-B employees. Because not all samples of size 100 are equally likely to be selected, this is not a random sample. In Lesson 14, we will discuss other methods of random selection.

Care must be taken in selecting a sample so that it is not biased. Bias is the tendency for a sample to differ in some systematic manner from the population of interest. Some common sources of bias are selection bias, measurement bias, response bias, and nonresponse bias. Selection bias occurs when a portion of the population is systematically excluded from the sample. For example, suppose a company wants to estimate the percentage of adults in a community who smoke. If a telephone poll is conducted, adults without telephones would be excluded from the sample, and selection bias would be introduced.

Measurement bias, or response bias, occurs when the method of observation tends to produce values that are consistently above or below the true value. For example, if a scale is inaccurately calibrated, observed weights could be consistently greater than true weights, resulting in a measurement bias. The way in which a survey question is worded could influence the response, leading to bias. For example, suppose that a survey question was stated as follows: "Many people think driving motorcycles is dangerous. Do you agree?" When stated in this way, the proportion of those agreeing will tend to be larger than would have been the case if the question had been phrased in a neutral way. The tendency of people to lie when asked about illegal behavior or unpopular beliefs, characteristics of the interviewer, and the organization taking the poll could be other sources of response bias.

Often in surveys, some people refuse to respond. Nonresponse bias is present if those who respond differ in important ways from those who do not participate in the survey. In a survey of gardeners, those with smaller gardens were much more likely to respond than those with large gardens. Because some of the questions were related to the size of the garden, this nonresponse resulted in response bias.

Types of Studies

In determining what types of conclusions can be made from a study, two primary considerations are (1) how the units are selected for inclusion in the study and (2) how the treatments are assigned to the units.

If the study units are selected at random from a population, then inference can be made to the population from which the units were drawn. Inference can only be drawn to the units in the study if units were not randomly selected from some population. If a researcher gets volunteers to participate in a study, then conclusions can be made only for those volunteers. Often, an effort is made to argue that the units in the study are representative of some larger population. However, if someone disagrees with the results and claims that the units in the study are different and that this affected the outcome, then there is no statistical foundation upon which we could argue otherwise.

In some studies, treatments can be assigned at random to units. For example, the treatments could be a new type and a standard type of dog food. Half of the dogs available for the study could be randomly assigned to the new type of dog food; the other half would get the standard type. If the assignment is made at random, then the study is called an experiment, and thus, a cause-and-effect relationship can be claimed. Returning to the dog food study, if the dogs on the new type of dog food had improved health compared to those on the standard dog food, we could conclude that the type of food caused the difference. (More advanced methodology may be used to derive casual relationships, but they are beyond the scope of this book.)

If treatments are not assigned at random to the study units, then we can discuss associations but not cause and effect. For a long time, it was observed that people who smoked were more likely to develop lung cancer than those who did not smoke. However, smoking is not a treatment that can be randomly assigned (at least ethically) to people. Therefore, it could not be claimed that smoking caused cancer, only that the two were associated with each other.

If treatments are assigned at random to units that were randomly selected from a population of interest, then the study is an experiment with a broad scope of inference. The term broad scope of inference means that inference can be drawn beyond the study units to the whole population.

If treatments are assigned at random but the units were not randomly selected from some population, then the study is an experiment with a narrow scope of inference. Because no random selection of the study units occurred, inference can be made only to the units in the study, and this is called a narrow scope of inference.

If treatments are not assigned at random but units are randomly selected from a population of interest, then the study is a sample survey. Notice that when conducting a survey, it is not possible to assign certain treatments at random. Gender, age, and dominant hand are only three examples. Associations, but not cause-and-effect conclusions, can be concluded for the population.

If treatments are not assigned at random and units are not randomly selected from a population of interest, then the study is an observational study. Here, associations can be drawn, but only for units in the study.

The discussion in the previous paragraphs is summarized in Table 2.1. To illustrate using Table 2.1, consider the following study. Suppose the goal is to determine whether the nicotine patch increases the proportion of heavy smokers (those smoking at least a pack a day) who are able to stop smoking. The study could be conducted in several ways. First, suppose that an advertisement is placed in a newspaper asking heavy smokers who want to quit smoking to participate in a study. All interested participants are screened to be sure that they are heavy smokers and to confirm a genuine interest in quitting. Half of these are randomly assigned to wear a nicotine patch; the other half are given a patch that has no nicotine. Every participant wears a patch for six weeks. Two months after the patch is removed (eight weeks after the start of the study), each study participant is assessed to determine whether or not he or she is smoking. Because study participants are volunteers and all volunteers meeting the study criteria were included, there was no random selection of units (people) for inclusion in the study. This would correspond to the second row in the body of Table 2.1.

Because there was no random selection of units, inference can be made only to the people included in the study. In practice, the argument is often made that the study participants are no different from other heavy smokers, and conclusions are made more broadly. However, if someone claims that these study participants are not representative of the population of heavy smokers and thus the conclusions do not apply to the whole population, there is no solid foundation by which to refute the claim.

The treatments (nicotine patch and no nicotine patch) were assigned at random. This corresponds to the second column in the table, so cause-and-effect conclusions can be made. That is, if the proportion of study participants who stopped smoking is significantly greater for those who wore the nicotine patch than the proportion of those who did not wear the nicotine patch, we conclude that the difference is due to the nicotine patch. The patch without nicotine is called a placebo patch because it has no active ingredients. Often, people who receive a treatment respond whether or not the treatment has any active ingredient. To be sure that a treatment, such as a patch or a pill, is effective, a treatment that appears the same but has no active ingredient is also given. The patch or pill or other item with no active ingredient is called a placebo.

In summary, the nicotine patch study is an experiment with a narrow scope of inference. It is an experiment because treatments are assigned at random. The scope of inference is narrow because people were not randomly selected from the population of heavy smokers for inclusion in the study, and thus, inference can be made only to the people in the study.

The nicotine patch study was a blinded one because the study participants did not know whether or not they had a patch with the active ingredient. A double-blinded study is one in which neither the study participant nor the individual determining the value of the response variable knows which treatment each person has received. By blinding, any tendency to favor one treatment over the other can be eliminated.

Conducting a Study and Statistics Planning In Short

In this lesson, we have discussed the key steps in conducting a study. Every decision made during a study has an impact on the analysis and interpretation of the results. The strongest conclusions are from experiments where treatments have been applied at random, allowing cause-and-effect conclusions to be made. Otherwise, we are limited to discussing observed associations between variables. By drawing the study units at random from a population, we are able to draw inference beyond the units used in the study to the population from which the units were drawn.

Find practice problems and solutions for these concepts at Conducting a Study and Statistics Planning Practice Questions.