The Environmental Protection Agency (EPA) released its now famous report on the health effects of environmental tobacco smoke (ETS) in December of 1992.
Titled "Respiratory Health Effects of Passive Smoking," the report was issued to much acclaim from anti-smoking advocates around the world. The conclusions were loud and unapologetic: ETS was ranked as a Class A carcinogen and exposure said to cause 3,000 deaths each year. interestingly, Helena smoke prohibitionists were claiming nearly 20 times that number, or 52,000 deaths.
The study was, and remains, a powerful weapon in the anti-smoking arsenal, cited on numerous occasions in campaigns around the world, including our own battle in Helena.
Though the study on the surface seemed to confirm once and for all everyone's "common sense" conclusions about the dangers of environmental tobacco smoke, in the years following its release, closer scrutiny revealed several serious problems.
The EPA report is what's known as a "meta-analysis" or a study of studies already in existence. Meta analyses are known to be difficult to carry out accurately--to vouch for the truthfulness of one study is difficult enough; for hundreds of studies near impossible. But they are, on the other hand, relatively inexpensive and also, perhaps more tellingly, easy to manipulate.
"Basically, meta-analysis can give you whatever numbers you want," says David Hitt, a journalist in upstate New York who has been writing about ETS science for several years. "You take a whole bunch of studies--regardless of their quality--you put them all in a blender, hit puree and scoop out whatever you need. If there are existing studies that don't mesh with your ideas, why, just leave them out!"
Hitt says this was exactly what happened in the EPA report.
The organization first located 33 studies that compared environmental tobacco smoke exposure to lung cancer rates. The EPA then selected 30 of the 33 as relevant to their needs. In the final report, the organization based their analysis on only 11 studies, three of which were later characterized in a congressional report as scientifically questionable.
The EPA apparently had good reason to exclude many of the original studies from their analysis. According to the congressional report, "From a group of 30 studies, six found a statistically significant, but small, effect; 24 found no statistically significant effect and six of the 24 found a passive smoking effect opposite to the expected relationship."
Even after excluding all but 11 studies, the EPA still had trouble coming up with the 3,000 deaths they'd promised, unfortunately, before the study was even complete.
To solve the problem, the authors were forced to make a tricky statistical move.
There's a concept in science known as the Confidence interval (CI) which, in layman's terms might be described as a study's margin of error. It is expressed as a range of values that would be considered valid--for instance .95-1.1.
The narrower the CI, the more accurate the study. The CI can be narrowed in a number of ways, including using more accurate data or drawing on a larger sample pool.
Confidence intervals are generally calculated to a 95 percent confidence level. This means the odds of the result occurring by chance are roughly five percent or less. (Of course, even this level isn't terribly reliable--"Imagine the brakes on your car failing five percent of the time," says Hitt.)
When the EPA found their analysis falling below the acceptable .95 CI, they were forced to double their margin of error.
Since most people don't know much about statistics, the EPA was able to release their study to little criticism. But as the scientific community began to more closely scrutinize the data, the shifty nature of the study soon became apparent.
In 1998, a federal judge, William Osteen, vacated the study, declaring its findings null and void and issued a 92-page decision on the shoddy way it was conducted.
"First, there is evidence in the record supporting the accusation that the EPA 'cherry picked' its data," he wrote. "Without criteria for pooling studies into a meta-analysis, the court cannot determine whether the exclusion of studies likely to disprove EPA's a priori (untested) hypothesis was coincidence or intentional."
He continued: "The record and EPA's explanation to the court make it clear that using standard methodology, the EPA could not produce statistically significant results with its selected studies. Analysis conducted with a .05 significance level and 95 percent confidence level included relation risks of 1 (no risk at all). Accordingly, these results did not confirm EPA's controversial a priori hypothesis."
And finally: "In this case, EPA publicly committed to a conclusion before research had begun--adjusted established procedure and scientific norms to validate the Agency's public conclusion, and aggressively utilized the Act's authority to disseminate findings to establish a de facto regulatory scheme intended to restrict Plaintiffs products and to influence public opinion."
Although the EPA's study has been thoroughly debunked by science and legally vacated by a federal judge, it is still regularly quoted by government agencies, advocacy groups and smoking prohibitionists as if it were legitimate. In the arguments leading up to the June 6, 2002 vote in Helena, it was drawn upon on numerous occasions.
Source: Extra,
a special supplement to The Montana Tavern Times,
Dec., 2002, published monthly by Continental Communications, 125 W. Granite St., Suite 102, Butte, MT 59701.