Join us for conversations that inspire, recognize, and encourage innovation and best practices in the education profession.
Available on Apple Podcasts, Spotify, Google Podcasts, and more.
Private: Learning Math: Data Analysis, Statistics, and Probability
Designing Experiments Part A: Comparative Studies (15 minutes)
Many statistics problems require you to make comparisons. You may be interested in comparing salaries for different professions, for example, or you might want to compare test scores of children in different reading programs. Comparative studies like this fall into two categories — experimental studies and observational studies. See Note 2 below.
The purpose of a comparative experimental study is to determine the “cause and effect” of an action. In this situation, an experiment deliberately imposes a particular treatment on a group of individuals in order to gauge their responses. This allows the investigator to determine if the treatment caused a change in the individuals’ responses.
For example, to determine the effect of taking aspirin on heart disease, an experiment was conducted using a large group of doctors as subjects. Half the doctors took aspirin every other day; the other half took a pill that looked and tasted like aspirin, but was not. The experimenters imposed the treatment (aspirin) on the individuals by deciding through random selection which half would take aspirin and which half would not. After several years, the incidence of heart attacks for the two groups was compared. The analysis of the results suggested that the aspirin dosage had a positive effect — that is, there was a lower rate of heart attacks among the doctors who received aspirin.
The purpose of a comparative observational study is to determine if there is a difference in measurements between groups of individual subjects. An observational study does not impose a treatment on the subjects — it observes them as they are. These types of studies do not allow the investigator to determine whether or not a treatment caused a change in the subjects’ responses — since there is no treatment!
For example, studies of the effects of smoking on people are necessarily observational. For many reasons, we cannot select a group of subjects and force them to smoke for a long period of their lives and then select another group of subjects and not allow them to smoke. Instead, we would select a group of people who are smokers, select another group who are not smokers, and compare the health characteristics of the two groups.
The data for comparative studies like these (either observational or experimental) consist of at least two sets of measurements. Although there may be more than one variable in a study, we will restrict our attention to the analysis of data collected on one variable for now. We will use Five-Number Summaries and comparative box plots to analyze and interpret data from several different comparative studies.
Problem A1
Suppose a professor wanted to decide whether courses taught online are as effective as more traditional methods of instruction. The professor divides a class into two groups. One group receives traditional instruction, while the other takes the course online. At the end of the course, each group takes the same comprehensive exam.
a. Is this an experimental study or an observational study? Explain.
b. What treatment is under study?
Problem A2
Suppose you were designing a study to answer the question “Do people who choose to take a two-week vacation in the winter feel more positive upon returning to work than people who choose to take a two-week vacation in the summer?”
a. Describe how such a study might be designed.
b. Would this be an experimental study or an observational study? Why?
c. What treatment is under study?
Problem A3
Suppose you were designing a study to answer the question “Do people with diets high in fiber, fruits, and vegetables have a lower risk of colon cancer?”
a. Describe how such an experiment might be designed.
b. Could this be an experimental study? What problems might you encounter if you wanted to perform an experimental study to answer this question?
c. What treatment is under study?
Notes for Session 6, Part A
Note 2
You will investigate the distinction between an observational study and an experimental study. Several examples are presented to illustrate the difference between these two types of studies. Group discussion should focus on how the nature of experimental studies allows us to make inferences about cause and effect. In contrast, an observational study can determine if there is a difference in measurements between two groups of individual objects but cannot conclusively attribute this difference to any particular factor.
Solution: Problem A1
| a. | This is an experimental study, since the online course is deliberately imposed on one group of participants. |
| b. | The treatment here is the online course. The treatment is successful if students perform better on the exam after taking the online course. |
Solution: Problem A2
| a. | You might pick two groups of working adults, of roughly the same age and background, half of whom voluntarily take vacations during the winter and half of whom voluntarily take vacations during the summer. Then you’d give each a survey of their feelings upon returning to work and compare the results. |
| b. | This is an observational study, since choosing when to take a vacation is not imposed on anyone. |
| c. | If everyone takes his or her vacation voluntarily, then there is no treatment. If you were to change this to an experimental study and select when people go on vacation, then the treatment would be the choice of vacation time. |
Solution: Problem A3
| a. | You might select a number of people at random and tell half of them to eat a diet high in fiber, fruits, and vegetables. The other half would not receive any specific dietary instructions. Over a long period of time, measure the incidence of colon cancer in each group. |
| b. | This could be an experimental study (as described in the answer to part [a]). You might encounter difficulty enforcing the imposed dietary changes, and there may be unforeseen health risks imposed on either group as a result of to their diets. |
| c. | The treatment under study here is a diet high in fiber, fruits, and vegetables. |
