The first few have some answers, the rest are just other examples. All taken from McKay's book 'Reasons, Explanations, and Decisions'.
1. Many people claim that one of the advantages of a college education is that it increases a student's intellectual sophistication (IS) and knowledge of the world (KW). Suppose that we had a satisfactory test of these qualities (IS and KW).
(a) What would be the design of a controlled experiment to test the hypothesis that college attendance increases IS and KW?
Take some high school graduates and randomly assign them to two groups, an experimental group and a control group. Test the members of both groups for IS and KW. (If they are large enough and randomly assigned, the average scores and the distribution of scores should be similar.) Send all member of the experimental to group to college, and make sure that no members of the control group go to college (during the experiment). At the end of four years of college, re-test the members of both groups, and see if there is a significantly greater increase in IS and KW in the college group.
b) Could we run an experiment of the type you described in answering (a)? NO
c) What problems would we have? There is no practical way to require college of one randomly selected group and withhold it from another.
d) What would be the design of a observational study to test the same hypothesis? Compare the amount of increase in IS and KW in college students and non-college. Test a complete high school class and retest those you can find four or five years later. See if the people who went to college show greater increase in IS and KW
e) What problems would there be in using results from the sdtatistical study to make a convincing case for the hypothesis? Since we are looking at increase, this is no too bad. However, if the college students have a larger increase, we are still left with the question of whether those people would have had a similar increase even if they hadn’t gone to college. They were different from the college-bound students even before college --- they chose to go to college and were admitted --- and the differences associated with that may also be associated with greater increase of IS and KW, even without college.
3. [From Newsweek, February 19, 1996. This is one paragraph of a long article, "Your Child's Brain," by Sharon Begley. Gordon Shaw is a professor at UC Irvine.]
“If you’re working with little kids,” says Gordon Shaw, “you’re not going to teach them higher mathematics . . . . But they are interested in and can process music.” So Shaw and Frances Rauscher randomly selected 19 preschoolers from a class of 40 students and gave them piano or singing lessons. After eight months, the researchers found, the children “dramatically improved in spatial reasoning,” compared with children given no music lessons, as shown in their ability to work mazes, draw geometric figures and copy patterns of two-color blocks. The mechanism behind the “Mozart effect” remains murky, but Shaw suspects that when children exercise cortical neurons by listening to classical music, they are also strengthening circuits used for mathematics. Music, says the UC team, “excites the inherent brain patterns and enhances their use in complex reasoning tasks.”
(a) What did Shaw and Rauscher observe as a result of their study?
After eight months, the children studying music showed greater ability to work mazes, draw geometric figures and copy patterns of two-color blocks, compared to the control group.
(b) What theory is offered to explain these observations?
When children exercise cortical neurons by listening to classical music, they are also strengthening circuits used for mathematics. Music excites the inherent brain patterns and enhances their use in complex reasoning tasks.
(c) Are there any alternative explanations of the observations that should be considered? If so, what?
Perhaps the additional attention given to the students in music lessons helps them to work better with adults and so do better in the testing situation.
Perhaps the extra time with adults in music lessons and practice helps to develop their intellectual skills. (They could be doing any kind of concentrated study, not just music.)
Perhaps the EG developed better skills at communicating with adults, and as a result the adults were biased in their recording of the results of the final tests.
(We would like to know that the recording of results was blind, so that the people doing the recording of results did not know which groups children were in when they did the final tests. Otherwise we might expect some bias in the recording of test results.)
(d) What information could help to show the correctness of the explanation(s) mentioned in the article?
This is very small study, it would need follow-up.
Were there any drop-outs from the study? It would be surprising if there were none. But any drop-outs would seriously affect the significance of the findings. The number 19 out of 40 initially raises the suspicion that there was some reorganizing of the groups in a non-random way. (Was one child moved from the EG to the CG right at the beginning because he or she could not participate in the EG for some reason?) Good answers to these questions would give us more confidence in the study.
Did the testers know which group the children were in when they did the final tests?
Since the groups were small, there may have been differences between the groups even before the experiment. Pre-tests to ascertain scores on spatial skills before musical training would be useful in judging the extent of improvement. (We can't tell from the report whether this was done.).
6. Suppose that a manufacturer of baby foods has begun to market a new line of baby food (ClearUp Baby Foods) containing a natural product that is said to reduce diaper rash when consumed with baby food. We would like to know whether this additive is really effective.
Describe two different experiments, one a fully controlled experiment (with random assignment) and the other an observational study. For each experiment, answer these questions:
7. [Based on a story in Newsweek, October 24, 1994.]
Studies by Linda Stroh find that men from traditional families, in which the wives stay home to care for the household, earn more and get higher raises than men from two-career families. Stroh based her study on 348 male managers at twenty Fortune 500 companies, where she found that for men with working wives, salaries were significantly lower and raises were less frequent than for men with wives who did not work. These differences showed up even when age, experience and time working for the company were equal. Stroh concludes that discrimination is the cause of the salary differences that she discovered; most of the companies are run by men whose wives do not work, and those bosses reward managers who fit the same mold.
8. In a yearlong study at Stanford university, researchers attempted to determine the effects of different types of exercise on people over 65. They tested the cardiovascular fitness, ability to walk significant distances, ability to lift weights, and levels of comfort and flexibility in their daily lives. Then the population (67 women and 36 men) was divided into two groups.
Group A did exercises designed to improve endurance and strength: brisk walking, low-impact aerobics, and strength building exercises with large latex bands.
Group B did exercises designed to retain and improve flexibility: stretching and other moderate exercise.
Results for group A: better cardiovascular fitness, able to walk farther, able to lift heavier objects than those in group B. Reported a higher degree of physical pain in their daily lives (i.e., higher than before the study).
Results for group B: reported more comfort and flexibility in their daily lives (i.e., more than before the study).
(a) Based on this report, there was no true control group in this study. A true control group would be a randomly selected group that was tested at the beginning and the end of the study, but that did no significant exercise during the year of the study. What additional information would be available if there had been a true control group?
(b) Why do you think that there was no control group in this case?
(c) On the basis of these results, doctors recommended that people over 65 do all of the exercises that were done by groups A and B. Are there any reasons to be cautious about this recommendation?
9. Some people think that running regularly helps to prevent colds. In particular, they say, after you run regularly (at least three times per week, at least 20 minutes each time) for a few weeks, you will have additional cold resistance that is maintained as long as you continue such regular running.
(a1) Describe a fully controlled experiment to test the hypothesis that running helps to prevent colds. Be sure to specify how observation would be done.
(b1) What observations would show that the hypothesis is true?
(c1) What observations would show that he hypothesis is false?
(d1) Are there any serious obstacles to carrying out the experiment you have described? Are there potential obstacles to interpreting its results?
Over a long study, enforcing abstinence from running has ethical problems, since running may be beneficial to health.
The daily records may not be well kept. They may be better kept by the runners (who make it a part of a routine), so that the groups lose comparability. We could substitute clinical observation (but then we have to pay the observers, too). Observation is not blind, so there may be biased reporting.
(a2) Now describe a observational study that might be done to evaluate the hypothesis.
(b2) What observations would show that the hypothesis is true?
(c2) What observations would show that he hypothesis is false?
(d2) Are there any serious obstacles to carrying out the study you have described? Are there potential obstacles to interpreting its results? How would this compare to the controlled experiment in the value of its results?