Thursday, December 06, 2007

File Under "Cooking the Data"

Heavens! JAMA published a study suggesting that there may be some shenanigans in the way scientific papers are cited by those with an ax to grind. The article implies that "true believers" may be, shall we say...selective in their arguments when promoting their pet theories.

Click here for more.
This is demoralizing. JAMA published a study suggesting that there may be some shenanigans in the way scientific papers are cited by those with an ax to grind. This article implies that "true believers" may be, shall we say...selective in their arguments when promoting their pet theories.

The authors looked at two observational studies published by the New England Journal of Medicine in 1993 suggesting that high vitamin E intake was associated with marked reductions in cardiac events. In 2000, an RCT was performed that showed no benefit (and possibly some deleterious effects). They then looked at how frequently the two "disproven" studies were cited later in the medical literature.

One would have expected a rapid fall-off in the citation curve after the RCT. In fact there was a fall-off but not nearly as precipitous as one might have expected. Even as late as 2005 (the last year they looked) 50% of the papers citing the original articles were still favorable towards vitamin E. Many authors were persisting in promoting vitamin A on the basis of two non-RCT's despite markedly better contradictory evidence that became available later.

This is rather damning against either the thoroughness of the citing authors (who perhaps missed the RCT in their literature searches) or against their honesty (in perhaps ignoring and not citing it). The researchers of the JAMA article did something rather slick that suggests dishonesty.

They looked at a possible correlation with how favorable towards vitamin E the citing articles were with whether or not they also cited the RCT. This is what's a bit disappointing. They found that 35% of the articles unfavorable or equivocal to vitamin E failed to cite the RCT but 83% of the pro-vitamin E failed to. Of course the erroneous conclusions could be because of sloppy research but it seems much more likely that true believers simply choose to ignore data that doesn't support their beliefs.

As my five-year-old likes to say when she doesn't get her way: "How sad".

Labels: , ,

Tuesday, November 20, 2007

Statistics and the Medical Residents Who Misunderstand Them

Dr. Helen recently blogged on a study showing that medical residents aren't too good at statistics. This of course raises the possibility that they won't be able to critically read the medical literature and that their patients will ultimately be shortchanged.

Click here for more.
Dr. Helen recently blogged on a study showing that medical residents aren't too good at statistics. This of course raises the possibility that they won't be able to critically read the medical literature and that their patients will ultimately be shortchanged.

As one who teaches residents, I have a couple of points to make about this.

First of all, the study's results seem quite plausible to me. I like to think however, that our residents at Harbor-UCLA do a little better since we run a weekly Journal Club (which I occasionally lead). It's fairly well-attended because there's usually a free drug company lunch...I know. That's another blog topic altogether.

In Journal Club, we take one article with important clinical implications and dissect it in all of its gory details. We pay particular attention to the biostatistics and methodologies involved. The purpose of such discussions is to understand the study and its potential biases. We also try to determine whether its results are valid and its conclusions generalizable to the patients we actually treat. The bottom line is that I think we make a fairly reasonable attempt to teach this stuff.

I will concede however, that my belief that our residents are not typical of those studied may be wishful thinking on my part.

But there are some larger issues here. While I agree that an understanding of basic biostatistics is essential to putting the articles comprising the medical literature in their proper perspective, many of the methodologies currently employed are extremely complex. Without a very strong background, rigorous understanding of a lot of these articles is all but impossible.

Even for those papers that don't use esoteric statistical methods (stochastic modeling, complex applications of logistic regression, nonlinear correlation methods, etc.) the amount of time necessary to digest them just isn't there most of the time; not for residents, not for most clinicians. That being the case, many doctors rely on clinical guidelines for basic decision-making. The idea is that a bunch of top experts in a particular specialty get together in Zurich for a week or so and discuss the world's literature between ski runs. They then hammer out a set of recommendations that summarize their collective knowledge.

These guidelines are generally quite readable and have the advantage of representing a consensus of these supposedly great minds.

Admittedly, such position statements have many disadvantages. There are all kinds of biases that can creep into them especially given the substantial conflicts of interest that top opinion leaders accumulate over their careers. The reality is that few other solutions are that well embraced at present and no one is going to read every significant study that comes out on his own.

My point is, don't come down too hard on the poor resident with his or her suboptimal understanding of basic statistics. I bet Sir William Osler wasn't that mathematically inclined either.

Here is a whimsical piece of irony: Dr. Donna Windish, the first author of the study mentioned above points out that many residents only read the abstract of journal articles rather than the body itself. She goes on to say that there is data to suggest that abstracts don't accurately reflect the implications of their studies. I then noticed the following in the abstract of her paper:
"Residency programs should include more effective biostatistics training in their curricula to successfully prepare residents for this important lifelong learning skill."
For this statement to be true, the authors would have to be able to cite data that establishes that
  1. Such training does in fact "successfully prepare residents for this important lifelong learning skill."

  2. That this learning skill makes them demonstrably better clinicians (or why develop it?)
I seriously doubt that either of these points has been firmly proven in the literature...which more or less justifies Dr. Windish's point that abstracts don't accurately state the implications of their studies.

Finally, if my discussion of this study seems a bit superficial, it's probably because I only read the abstract.

Labels: ,

Wednesday, November 09, 2005

A New Medical Blog

I'm adding a new blog to my blogroll today: Within Normal Limits of Reason.

Not knowing the gender attached to the name of its author, Ming-Chih Kao, I've made, in the interest of readability, the regrettably sexist assumption that he is a he. Kao is a 3rd year medical student at university of Michigan and his blog appears to have begun last month so it's quite new.

I doubt that this will be of general relevance throughout the blogosphere but for those of us involved in study design and methodology, it can be quite interesting. Kao is a newly minted Ph.D in biostatistics from Harvard and has posted some provocative thoughts on the medical literature often from a statistical perspective.

For example, this post caught my attention.

It deals with the common practice of discontinuing randomized clinical trials after statistical significance is achieved. For example, assume a study is analyzing the outcome of a disease following treatment with drug A vs. Drug B over a period of time. What is generally done is to analyze the data repeatedly as the study progresses.

If drug A shows statistically significant superiority at any time during the analysis, the study would be stopped. The idea is that the study objective has already been achieved and that to continue would subject the patients in the drug B arm of the trial to the deleterious effects of being treated with an inferior drug. This would be unethical.

Kao, however, points to an article in JAMA (Journal of the AMA) that demonstrates some pitfalls in this approach that I doubt most people would have thought of.

The argument is subtle but the gist of the concept is this: there is a statistical dispersion of expected results each time you re-analyze data prospectively (with new data points). If one simply stops the study when statistical significance is achieved without taking into account this predictable dispersion, results can be skewed to large treatment effects that would have perhaps been markedly smaller (or even been reversed) had the study been permitted to continue to completion.

Kao concludes with the apropos observation that "A watched pot never boils. Or not."

It's pretty intriguing stuff and challenges some basic assumptions many of us have about such an approach. Kao's blog is filled with such postings. I look forward to his reflections on the medical literature from the vantage point of his statistical background.

Additionally, count on me to correct the error of my ways if I discover that he is in fact a she!

Labels: ,

Monday, March 28, 2005

Requiring reporting of negative drug trial results

A Harris poll done in November of 2004 was reported in Internal Medicine News that showed, among other things, that 72% of the public would be in favor of some kind of legislation requiring that all negative drug trial results be published.

This is a very interesting question to have asked the public. My feeling is that the other 18% probably didn't understood the ramifications of such a subtle rule or it would have been 100% in favor.

A problem that has long affected the medical literature is what we call "publication bias". This is the selective publication of interesting studies, outcomes which are usually beneficial results from intervention trials. For example, if a drug is found to be effective at treating a particular condition, it is more likely to be published than if it is found to have no benefit.

Journals don't like reporting negative results because they're not sexy enough. Of course if the study demonstrates a deleterious effect of the drug, then that would be a sexy result and likely be published!

However, there is a more sinister reason for publication bias. When drug companies provide financial support for a clinical trial of one of their drugs and the results show no benefit, it may try to block the submission and therefore publication of the study. Technically, submission of the article would be at the discretion of the principle investigator (PI). But the thought of cutting off a future funding source may induce the PI to not submit.

One proposed solution to this problem would be to establish a registry for all studies regardless of the results. Outcomes and results would then have to be reported (and would be accessable by a medical literature search engine (such as www.pubmed.gov, the database maintained by the National Library of Medicine).

Obviously the drug companies would not be in favor of this and some people might argue about the intervention of big government in privately funded research. I don't believe that this is a valid argument in that there is clearly an overiding public interest in such a registry. Government is already involved in any study that uses human subjects at present. Mandating the public reporting of results of such studies seems like a very small additional burden.

Labels: ,