Wall Street Journal: "Mistakes in Scientific Studies Surge"

boardswalker

Just spotted this article in the Wall Street Journal last August.
http://online.wsj.com/article/SB10001424052702303627104576411850666582080.html

There are a number of statements in there that are concerning. Here's just two examples :-

Since 2001, while the number of papers published in research journals has risen 44%, the number retracted has leapt more than 15-fold, data compiled for The Wall Street Journal by Thomson Reuters reveal.

Retractions related to fraud showed a more than sevenfold increase between 2004 and 2009, exceeding the twofold rise in retractions related to mere error, according to an analysis published in the Journal of Medical Ethics.

El_Dangeroso

I think this is not helped by the fact that many clinicians see research as a way to further their career and attain prestige rather than mainly due to scientific curiosity.

There are good researchers out there but they seem to have a hard time getting the large grants.

Also, I have no formal training in medicine or lifesciences, I'm just an amatuer self-taught science nerd and I have read several papers in medium impact journals which had glaring errors in basic statistics. The peer-review process needs some serious improving IMO.

Biologic

Wait, so if I do believe that scientific literature is garbage, then that means the article in the Journal of Medical Ethics is rubbish, which means the literature isn't flawed, which means it is.... I need a nap.

opinion guy

El_Dangeroso wrote: »

I think this is not helped by the fact that many clinicians see research as a way to further their career and attain prestige rather than mainly due to scientific curiosity.

True but proper reviewing and editing should weed out the crap.

2Scoops

opinion guy wrote: »

True but proper reviewing and editing should weed out the crap.

And who do you think is reviewing the crap? Authors, turned reviewers, of similar crap! :pac:

It seems like there are hundreds of new journals since the 1990s and it stands to reason that there will be an increase in unworthy science. Statistical errors tend to be on the benign end of the scale, especially those so obvious that an 'amateur' can spot.

I think the article itself is more concerned with deliberate scientific fraud, an different issue altogether and far more serious. I think we can at least take heart that many instances are being caught and that the incidence is still quite low.

opinion guy

2Scoops wrote: »

And who do you think is reviewing the crap? Authors, turned reviewers, of similar crap! :pac:

It seems like there are hundreds of new journals since the 1990s and it stands to reason that there will be an increase in unworthy science. Statistical errors tend to be on the benign end of the scale, especially those so obvious that an 'amateur' can spot.

I think the article itself is more concerned with deliberate scientific fraud, an different issue altogether and far more serious. I think we can at least take heart that many instances are being caught and that the incidence is still quite low.

Hmm you are clearly less cynical than I. I think the real problem is the politics involved in the whole process. authors can suggest reviewers for their work. Reviewers are not blinded to who the authors are. I think high standard peer reviewing should be done with blinding i.e. authors should have no input into reviewer selection and reviewers should have no knowledge of who the authors are.

2Scoops

I think we'll have to fight about who is most cynical! :pac: I'm not defending the quality of peer review, I was pointing out why it fails (mass action, bad reviewers) and trying to refocus the thread on fraud rather than unimportant research.

I think we're at a point where, if you persevere for long enough, nearly any study will eventually pass peer review in lowest tier journals. Blinding reviewers etc. whatever won't change that. In fact, in most cases, if you work in the same field long enough, blinding becomes infeasible - you can identify labs based on their research aims alone and in many cases correctly guess your reviewers by their comments. As an editor, you are under no obligation to use suggested reviewers. I read once that the evidence shows, while you tend get an easier ride on the 1st review, actual acceptance rates are not different whether you use suggested reviewers or not.

Since nearly everything is indexed and accessible online these days, I think it's also the responsibility of the readers to be able to spot important science and detect scientific flaws. And what is peer review but the first readers, albeit with a background? Fraud on the other hand, often cannot be detected by readers or reviewers and that's why it's a bigger problem and why peer review can't provide the solution.

boardswalker

2Scoops wrote: »

I think we can at least take heart that many instances are being caught and that the incidence is still quite low.

What stood out for me in the article, is the length of time it takes to have errors (let's be generous) corrected.

In my reading of the various scandals,withdrawals etc it is clear that once a product or a theory gets established on the basis of faulty research, it takes years to correct the error. When conscientious professionals developed a concern, they have had to overcome systemic resistance to the possibility that there may be an error in the research.

Just look at the main case the WSJ references.

In the case of high blood pressure drugs, it took six and a half years to get a retraction.

To quote the article: "The damage was done. Doctors by then had given the drug combination to well over 100,000 patients. Instead of protecting them from kidney problems, as the study said the drug combo could do, it left them more vulnerable to potentially life-threatening side effects, later studies showed. Today, "tens of thousands" of patients are still on the dual therapy, according to research firm SDI.

When a study is retracted, "it can be hard to make its effects go away," says Sheldon Tobe, a kidney-disease specialist at the University of Toronto."

Pointing to the fact that the errors were caught is not much help to those people who suffer for many years while the incorrect information drives patient care.

El_Dangeroso

2Scoops wrote: »

Statistical errors tend to be on the benign end of the scale, especially those so obvious that an 'amateur' can spot.

Not true. Recently saw an epidemiology paper that did not give age-adjusted hazard ratios. If you crunched the numbers with that in mind, the results were in some cases opposite to the authors' conclusions!

Unless you consider epidemiology a 'benign' discipline, but considering how these huge longitudinal studies can influence public policy, I'd have to disagree there too.

2Scoops

Well, it depends on the research in question how important errors are. Was any important public policy at stake? But a self-described amateur such as yourself spotted it immediately and, presumably, those that work in the field will spot it too. That's a good thing. Also, of all the errors you have spotted, how many are that serious vs. trivial?

El_Dangeroso

2Scoops wrote: »

Well, it depends on the research in question how important errors are. Was any important public policy at stake? But a self-described amateur such as yourself spotted it immediately and, presumably, those that work in the field will spot it too. That's a good thing. Also, of all the errors you have spotted, how many are that serious vs. trivial?

It was that recent 'multi-vitamins will kill you trial' so I don't know whether that would have ever influenced public policy. But it would have a good chance of influencing health professionals. What the outcome of that would be I don't know, but it's still shoddy work that really should have undergone more rigour in it's review.

I totally concede that every paper has at least a small error, I know science isn't perfect and there are constraints of resources etc, especially in academic research. But I do think that the standard should be a little bit higher than it is, or at least we should be questioning how it can be improved. I like the blinding of reviewers idea that personally.

2Scoops

boardswalker wrote: »

What stood out for me in the article, is the length of time it takes to have errors (let's be generous) corrected.

That's because with cases of fraud, you are essentially calling someone a liar rather than efficiently correcting a scientific error. Where someone has deliberately mislead and refuses to admit guilt, you are left with legal avenues and long-term technical investigations to get the proof you need to take action. E.g. in the COOPERATE study cited, it was widely felt that it was BS, but if the Lancet retracted it on a hunch, it was open to litigation itself.

boardswalker

2Scoops wrote: »

That's because with cases of fraud, you are essentially calling someone a liar rather than efficiently correcting a scientific error. Where someone has deliberately mislead and refuses to admit guilt, you are left with legal avenues and long-term technical investigations to get the proof you need to take action. E.g. in the COOPERATE study cited, it was widely felt that it was BS, but if the Lancet retracted it on a hunch, it was open to litigation itself.

No mention of protection of patients' health in there.

Does correcting a scientific error rank higher than protecting patients?

opinion guy

I have to say though - errors are one thing, alot of dodgyness comes bout at the stage of study design I think. Lots of stuff I've read I don't think should ever have been published for these reasons. All to easy to design a study to NOT answer a question

2Scoops

boardswalker wrote: »

No mention of protection of patients' health in there.

Does correcting a scientific error rank higher than protecting patients?

Come off it, I never said anything of the sort. :rolleyes: