Article - Question marks about psychotherapy research

The Black Oil

Recent piece courtesy of BPS Digest.

The research team, led by Klaus Lieb of the University of Mainz, examined 95 systematic and meta-analytic reviews that had evaluated the efficacy of psychological therapies by looking at the weight of evidence across multiple randomised controlled trials. Such reviews are generally used to give a balanced picture of what really works, above and beyond a single trial.

The journals that publish this kind of research tend to be on the look out for financial conflicts of interest that could lead to a bias: for example, if the author of a meta-analysis of Therapy X was a license-holder for that therapy but chose not to declare that interest when reviewing it. Indeed, Lieb and his team found that four out of every five journals they investigated had explicitly asked that such conflicts be declared.

But only a third of the journals asked the same of non-financial conflicts of interest, such as a review author being trained in one specific therapy, or otherwise being a particular advocate of one over others.

~

Lieb’s team found that 34 of the reviews covered at least one and sometimes many more experimental studies authored by the reviewers themselves: not a crime in of itself, but a sign of at least a potential conflict. What’s more, Lieb’s group investigated these 34 reviews using a standard protocol to check for signs of “research allegiance”, that is whether the article betrayed a belief in the superiority of a treatment. Fifteen of the 34 reviews met these criteria, meaning that one or more of the authors had a vested interest in the conclusions.

Lieb and a co-author, Jan von der Osten-Sacken, also read through anonymised versions of all 95 reviews and labelled a conclusion as “spinned” when they felt it wasn’t consistent with the empirical results reported. Based on this, they claimed that reviews they’d identified earlier as having non-financial conflicts also tended to have biased conclusions. However, this link wasn’t statistically significant, so at best this tentative finding indicates something to be studied in the future.

Lieb’s team emphasise they are not claiming that there is rampant fraud and deliberate misrepresentation. They suspect that “researchers may simply not be seeing the necessity of declaring non-financial interests, or be clear on what this includes”. So it’s incumbent on journals to outline more explicitly what constitutes a conflict of interest, and request and reveal them with more vigilance. Researchers themselves should take to heart the foundation on which the search for truth is built: to let go of your beliefs and be willing to be wrong.

Study details

To assess spin in review conclusions, we evaluated whether the conclusion of the review as expressed in the abstract or the discussion section was inconsistent or consistent with the empirical results described in the results section of the review. If the conclusion was consistent with the empirical results, the review was considered as showing no spin. If it was inconsistent, the review was rated as showing spin. Two researchers (KL, JvdOS), who both were blind to the author names of the review as well as the journal having published the review, independently assessed review conclusions and results, and rated whether a spin in review conclusions was present or not. If no consensus was achieved, disagreements were resolved by a third person (JB). κ Statistics showed substantial inter-rater reliability (k=0.70; agreement 87%).

Full version here.

I don't follow the therapy literature myself, but I'm sure a few of you here do. :cool: Of the psych research I do read, I can't recall seeing anything about COIs being declared. First time I've seen the word 'nowadays', used in a professional study. :P