17 September 2012

Measuring Spin in RCT Reporting

Randomized Control Trials are the gold standard approach for conducting scientific studies. However, not every study is created equal and the interpretation of the results may not be straightforward. Someone who only has a passing understanding of how to read research, such as myself, relies heavily upon the abstract from a study, the accompanying press release and the discussion in the conclusions section.

It turns out that the information provided may have a bit of spin in it that eventually trickles down to the reporting. A group of researchers looked at some RCT press releases from Dec 2009 to March 2010 and found that there was 'spin' in the press releases. From their article:
We systematically searched for all press releases indexed in the EurekAlert! database between December 2009 and March 2010. Of the 498 press releases retrieved and screened, we included press releases for all two-arm, parallel-group RCTs (n = 70). We obtained a copy of the scientific article to which the press release related and we systematically searched for related news items using Lexis Nexis. 
“Spin,” defined as specific reporting strategies (intentional or unintentional) emphasizing the beneficial effect of the experimental treatment, was identified in 28 (40%) scientific article abstract conclusions and in 33 (47%) press releases. From bivariate and multivariable analysis assessing the journal type, funding source, sample size, type of treatment (drug or other), results of the primary outcomes (all nonstatistically significant versus other), author of the press release, and the presence of “spin” in the abstract conclusions. Findings of RCTs based on press releases were overestimated for 19 (27%) reports. News items were identified for 41 RCTs; 21 (51%) were reported with “spin,” mainly the same type of “spin” as those identified in the press release and article abstract conclusion. Findings of RCTs based on the news item was overestimated for ten (24%) reports.
I have to point out the irony that the research talks about spin in research that is present in the abstraction conclusion which I then had to rely upon in order to understand this research. In other words, I am utilizing the very thing that is found to be questionable in the study.