On November 19, 2014, the House of Representatives passed HR 4012, the Secret Science Reform Act of 2014. The bill would prohibit the US Environmental Protection Agency from regulations based on “science that is not transparent or reproducible.” Hooray! Reproducibility is the touchstone of science.  Transparency is the way to ensure that scientists who want to reproduce another scientist’s results can try to do so. No scientist anywhere would argue against reproducibility, nor should any scientist argue that research results used to make regulatory decisions (such as drug approvals, emission limits, product bans) be exempt from transparency. Unfortunately, transparency has not always been a priority.  Examples of the current reproducibility crisis in the sciences can be found here, here, here, here, and here. A requirement to use transparent, reproducible science should apply to all government agencies, not just EPA. The government need not replicate the science itself, just make sure the information needed for reproducibility is readily available. Just one of the many examples from the pavement coatings industry’s decade-long effort to obtain data from the U.S. Geological Survey illustrates the point.

A decade ago the USGS Water Science Center in Austin, Texas collaborated with the City of Austin’s Water Protection department in a small study of runoff from parking lots, including some coated with either asphalt-based or refined tar-based sealcoat. The study was reported in a published paper; additional details were included in a USGS report. The study involved chemical analysis of samples collected from parking lots and of particles washed into nearby streams by simulated rainfall.  Curiously, the study design seemingly did not include sampling stream bed sediment from any location. Yet a major conclusion advanced by the study authors was that refined tar-based sealcoat was the major source of a class of chemical compounds known as PAHs in stream bed sediments in Austin and all “urban watersheds.” There was no attempt to put a number on the PAH contribution in the science papers, but a June 23, 2005, article in the Austin American-Statesman included the following statement:

According to Wednesday's report [by Austin officials and the U.S. Geological Survey], parking lot sealants may contribute about 90 percent -- perhaps even 95 percent -- of the PAH pollution in urban watersheds.

As soon as the science paper was officially published, the City Council banned the use of refined tar-based pavement products in Austin.

It’s hard not to speculate that someone in the City was searching for a reason to ban these products.  Prior to getting the USGS involved, the City asked the Texas Commission on Environmental Quality, the Texas Department of Health, and EPA Region 6 about potential risks that might be associated with certain parking lots, covered with refined tar-based sealer, near Barton Springs. Each of the agencies told the City there was no problem (here, here, and here).  After conducting an extensive study the TDH and the U.S. Agency of Toxic Substances and Disease Registry (ATSDR) concluded:

We did not find any information to support the contention that swimming every day in Barton Springs would result in adverse health effects. Thus, we have concluded that swimming and playing in Barton Springs Pool poses no apparent public health hazard. We recommend continued public health education to address any questions that the public may have concerning the risks associated with swimming in the pool.  [All emphases are in the ATSDR report.]

The USGS study was not designed to test the findings of the other state and federal agencies; rather the intent seems to have been to provide -- dare I use the word “concoct”? -- a seemingly scientific reason for the City to ban the use of tar-based sealcoat.

The connection between sealcoat and sediment was made graphically, by showing a similarity of chemical analysis of particles washed off tar-coated parking lots with results of analysis of 20 samples identified as “urban sediment,” bolstered with some mass balance calculations. Companies in the sealcoat industry engaged the environmental consulting firm Environ to help them understand the USGS study. The USGS publications did not include information about the 20 “urban sediment” samples, so Environ asked and was given a City of Austin report containing sediment data. Reproducing the USGS findings using these data was unsuccessful. The USGS responded that the data used by Environ may not be the same as the data used to represent “urban sediment.” Not long after, City of Austin staff reported sediment sample results and were not able to find a chemical signature for PAHs from refined tar-based sealers either (this is explored in-depth here). Environ collected sediment samples before Austin’s ban went into effect and again a few years after the ban, and (again) was unable to reproduce the claimed finding, noting neither a decrease in PAHs nor any evidence that PAHs were coming from parking lot sealcoat.

Despairing of receiving the data from the USGS via a slow-walked Freedom of Information Act response, in July 2013, the Pavement Coatings Technology Council sent a letter to the Chairman of the American Chemistry Society’s Ethics Committee (with copies to USGS management) asking for assistance in getting information about the “urban sediment” samples to figure out why attempts to confirm the USGS conclusions were not succeeding.  The USGS responded directly to PCTC’s request for ACS assistance. The “urban sediment” data were, it turns out, not collected at or even near the locations used in the USGS’ parking lot study. They were collected at different times over the course of years from a creek in Austin not included in the study and from three creeks in Fort Worth, 200 miles away.  

It is tempting to speculate that the available data from creeks where the USGS studies were conducted were not used because they did not sufficiently match the PAH ratios of sealcoat-related samples. It is also tempting to speculate that chosen data were selected because their PAH ratios were sufficiently similar. But we need not speculate – the USGS provided the answer in its FOIA response. In an email dated July 19, 2004, one of the USGS scientists wrote this:

When the Williamson Creek suspended sediment data was plotted on the same graph, they tended to group with the sealed parking lots as opposed to with the unsealed (asphalt pavement or cement) parking lots. Suspended sediment data from three small urban watersheds in Fort Worth were similar.

Thus, USGS has confirmed that it was precisely because PAH ratios most closely matched tar-based sealcoat that the 20 samples (including 12 from Fort Worth) were chosen to represent Austin stream sediments. There is no information about whether there were any sealcoated parking lots in those locations. There was no explanation that the locations were all near busy highways -- vehicle exhaust and used motor oil are well-known sources of PAHs with ratios similar to sealcoat. Data from other locations -- including streams located next to parking lots that were part of the wash off study -- were available, but the ratios do not seem to have matched as well. Those data were not used. The polite way to describe this behavior is “selective use of data.” The blunt way is “cherrypicking.” Whatever words are used, it is now confirmed that the USGS manipulated the data to fit their predetermined conclusions and narrative. This is not sound science. It is the opposite of sound science. Using this USGS study as the basis for municipal ordinances, as was done In Austin (and beyond), undermines confidence in science as a regulatory tool.

This example of government scientists hiding information for years while simultaneously issuing press releases, giving interviews, testifying to county boards and state legislatures and briefing members of Congress is just one example of the “secret science” of government conducted by unelected bureaucrats that an industry comprised mostly of small family-owned businesses, such as the pavement maintenance industry, must grapple with.  

The USGS does not issue regulations, and is not part of EPA.  Nevertheless, an enforceable requirement that all federal government science must be sufficiently transparent to be reproducible if it is to be used as a basis for local, state or federal regulation would increase confidence in the use of science in regulation and perhaps help reverse the understandable feeling of many that the government is out of control and not accountable.