From the issue dated December 19, 2003
http://chronicle.com/weekly/v50/i17/17b02001.htm
|
POINT OF VIEW
The Data in Your Lap: How to Interpret Naturally Occurring Experiments
By DALTON CONLEY
President Bush touts the recent news of booming third-quarter growth as evidence that his economic policy of tax cuts is working at long last. Meanwhile, Democrats are likely to claim that the recent upturn in economic growth is merely a natural result of seasonal fluctuation and other factors. The truth is that it is simply impossible to prove that tax cuts did the trick, just as it is equally unprovable that they didn't.
Some things we can be fairly certain about in life: Despite the protestations of tobacco companies, cigarettes are bad for your health. If you don't file a tax return, you're probably in trouble. Playing Russian roulette regularly will lead to an early grave. Other things are not so clear: Does wearing a bike helmet reduce your overall risk of injury? Does staying in school pay off in the long run? Does going to an Ivy League university merit the extra tuition over and above a state college's? Does military experience really help you "be all you can be"?
For so long in the social sciences, "correlation" was taken to mean "causation." Slowly -- over the past decade or so -- a revolution of sorts has occurred. No longer is it good enough to show that two phenomena happen to graph onto each other well. Spearheaded by economics, another approach -- "natural experiments" is sweeping the social and natural sciences, spreading to fields as wide-ranging as epidemiology, sociology, management, and conflict studies. Most important, it has thrown into question everything we think we have known about cause and effect in scientific inquiry.
Natural experiments are events that happen to divide the world into treatment groups and control groups, just as in formal experiments. The history of their use in research extends back to studies of the price of flaxseed in the 1920s, but natural experiments really caught on in the 1990s, when they were applied to questions of major social importance, such as the economic value of staying in school. A fierce methodological debate in economics has ensued about what makes a good natural experiment and what doesn't.
A doctor observes that men who drink red wine with dinner suffer from fewer heart attacks than those who don't. She assumes that moderate consumption of red wine reduces the risk of heart disease. A manager sees that employees who use e-mail the most are also the most efficient. He concludes that e-mail aids workers and should not be discouraged. A government official documents that poor kids from rich neighborhoods do better in school than poor kids from poor neighborhoods. He proposes housing vouchers -- assistance for low-income renters to compete in the private rental market -- in lieu of building more subsidized housing.
The doctor who tells us to drink wine, the manager who encourages connectivity at work, the official who advocates the wrecking ball for public housing -- all seem to have made a logical connection. But they might be wrong. Like President Bush, they've made the leap from correlation to causation without ruling out alternative possibilities.
After all, it may be that people who drink wine with dinner tend to be more relaxed or have better diets in other ways. (Meanwhile, the wine could actually be bad for them.) Workers who happen to be rapid e-mailers may also be faster at doing work-related tasks, and e-mail might actually be preventing them from reaching their full potential. And neighborhood quality could be irrelevant to children if it turns out that the savviest and most educated of the poor happen to live in richer areas.
So how do we find out what's really true? Ideally, in medical science, management, or politics, we conduct a formal experiment: We give one group a placebo, another the "treatment." If the treated group had substantially better outcomes than those who received the placebo, we know the treatment works. Trouble is, we can't conduct formal experiments in most areas of life. (Certainly, we don't have two economies -- one with the Bush tax cuts and one with a "placebo.")
Natural experiments, however, give us hope for sorting out what causes what in our increasingly complex information society. Take the Vietnam draft lottery. In 1969, lottery administrators pulled what looked like Ping-Pong balls out of a vat. On each ball was a birth date, which was assigned a draft number based on the order in which it was picked. Men with low numbers, who would face military call-up first, were confronted with tough choices -- expose themselves to the draft, enlist, or flee to Canada. As a natural experiment, the result of the lottery is the next-best thing to randomly assigning some people to go fight and sparing others such a fate.
So if you want to know how Vietnam affected the pocketbooks and long-term futures of U.S. veterans, don't compare those who went with those who didn't. Many people -- including Bill Clinton and George W. Bush -- generally avoided the draft, while Joe Six-Pack faced the music. Instead, compare those who got a bad Ping-Pong ball with those who got a lucky Ping-Pong ball. That is, in fact, just what two notable studies have done. In 1986, Norman Hearst, of the University of California at San Francisco, compared draft-likely and draft-unlikely birthdates in the death records of two states and found that even several years after the war had ended, those with unlucky birthdays suffered from higher mortality rates -- most notably from suicide.
Likewise, in 1991 the economist Joshua Angrist, of the Massachusetts Institute of Technology, estimated the impact of military service on later earnings by contrasting those with bad Ping-Pong balls to those with good ones. He found that white men in the "treatment" group -- i.e., the draft-eligibles -- had lower earnings that did those in the "control" group. (He observed no difference among nonwhite men.) Under a couple of assumptions -- most important, that there was no independent effect of a bad or good Ping-Pong ball (e.g., that employers did not invest more readily in those with a lucky birthdate) -- Angrist extrapolated that Vietnam cost white veterans 15 percent of their lifetime earnings. To be fair, both of those studies have limitations, but among them is not the subjects' being "self-selecting" into the treatment or control groups.
Want to know if timely medical care will reduce heart-attack fatalities? Just look at how far from emergency rooms are those who die, compared with the distance traveled by those who live. Likewise, state-to-state differences in what cancer drugs Medicaid covers provide a natural experiment for which drugs are effective when clinical trials are not possible.
While they do not provide placebos, the best natural experiments occur without the knowledge of the participants. For example, we all knew about the impact of Roe v. Wade on a woman's right to seek an abortion, yet no one in the criminal-justice system (or elsewhere) back in the early 1970s was giving much thought to its impact on crime. No one suspected that the decision might have laid the groundwork for the steepest drop in crime ever recorded (in the 1990s). The logic of the economists John J. Donohue and Steven D. Levitt is the following: Unwanted children are more likely to commit crimes in their teenage years and young adulthoods than are other children. The Supreme Court decision suddenly meant that many unwanted pregnancies were ended before birth.
Because, as a society, we were largely blind to that impact, we did not adjust other policies and behaviors to take the new reality into account. Roe v. Wade was a natural experiment in crime precisely because we did not realize that it was. (To be fair, there are valid critiques of the abortion-to-crime link; no experiment is foolproof.)
While the so-called information age has brought us more than a few supposed "paradigm shifts," one that has gone relatively unheralded is this new way of assessing causal relationships. Of course, the Catch-22 is that the more we pay attention to this way of viewing information, the fewer experiments will be lurking right under our noses. Economic policy, for one thing, certainly does not fit the mold of natural experiments, since it is so purposeful and so tied up with other factors.
Sure, the Bush tax cuts happened during a period of slow growth, and now there appears to be accelerated production. But we also experienced the lowest interest rates in decades, a third-quarter drop in the price of oil from abroad, and numerous other global and local economic changes. My favorite candidate for the economy's magic bullet is the particularly exciting baseball pennant race during the third quarter. Politicians will always twist correlations between trends or events to fit their instrumental ends, but scientists should now know better.
Dalton Conley is director of the Center for Advanced Social Science Research at New York University and author of The Pecking Order: Which Siblings Succeed and Why, to be published by Pantheon in March.
http://chronicle.com
Section: The Chronicle Review
Volume 50, Issue 17, Page B20
|
Copyright © 2003 by The Chronicle of Higher Education