Reading Scientific papers

r3nu4l

Many people here are either students of Medicine or Biology, some fully qualified, others on the road to qualification.

I just wanted to talk about how a scientific paper should be read...or really what to look out for. It's important here on boards.ie too if you are thinking of linking to a publication to support an argument.

Journals and peer-review
Firstly, when you see a paper of interest on the net, look at what journal has published it. Determine if the journal is peer-reviewed or not. i.e. do reputable scientists review and approve the content before publication.

That's an important first step as it means that the results, while they may be controversial have at least passed before the eyes of other scientists. Peer-review is by no means fool-proof but it's a start and implies some form of credibility.

Paper title
If the paper is discussing a vaccine trial/new drug then the information will be important but you now know that a phramaceutical company have probably funded the study and therefore while results may be accurate, examine the data carefully to see if it matches the conclusions reached.

Author Addresses
Secondly, look at the address of the authors that published the information. This gives you some insight into the type of institution conducting the research. If the address is a company and they are publishing information about their own product or a competitor product then there may be some bias in data presentation.

Author disclosures
Sometimes you will not see a pharma company mentioned anywhere except in the small print. "Dr Soandso has participated in expert panels, advisory boards for PharmaGiantX". This means that the author has been paid for his/her time to advise on the company product and has also been paid to put his/her name to the paper and results presented. Most docs will avoid bias where possible but it is worth remembering that the paper is in fact funded by a company who will be promoting a product downstream of the paper.

Statistics
A minefield. Large scale clinical trials conduct huge numbers of tests on the available data and while they shouldn't ignore data that sheds bad light on their product they will often play it down. Leaving Vioxx aside, my last company worked for a large pharmaceutical corportation on a diabetes drug. Having access to the data I was always surprised and a little worried by one of the side-effects but the drug is FDA approved so I didn't think too much more about it. Last year the FDA updated advice on this drug warning that in some people this side-effect could actually lead to blindness. Reading the papers relating to that drug would never give you any indication that there could be a problem with it as the side-effects were always played down.

Funding sources
Pretty much covered already but many authors enjoy very good funding from large companies. Due to the nature of science fundng they are naturally not keen to jeapordise any funding they have. Therefore once again, this is worth bearing in mind when reading a paper.

I make it sound as though these papers are not worth the paper they are printed on but that is not the case. The vast majority of papers contain, scientifically valid and useful information. All I'm saying is that these things should all be considered when reading a paper. All doctors and scientists are trained to question, question, quesiton but yet we sometimes don't do it enough when we read an article.

If anyone else has anything to add/discuss please do

Jimoslimos

Hmmm interesting points...

Journals and peer-review
Kind of depends, scientists working in specialist areas will have a good idea which journals carry more weight and lets face it not every paper (brilliant as it may be) is going to end up in Science or Nature. I suppose you could say the strength of a journal is measured by the strength of the panel of scientists it has reviewing papers.

Paper Title
First thing that catches the eye but I'd try and read the abstract aswell to glean any information from it

Author Addresses
I suppose a well-respected institution willl add a certain amount of weight to any paper. As for companies its quite common for the majority of core research work to be done by the university etc., and the company to add its name to another paper later in the development stage (prior to launch)

Author disclosures*
A tricky one this. Most scientists I know are fiercely proud of their impartiality regardless of whether they work for large companies or not. Would the answer be banning scientists from having any commercial links with big business? In that case how would companies begin to try have their work verified by independent scientists?

Statistics
What's that saying, there's lies, dammed lies and statistics Everyone to blame here, when it seems easier to use a simple t-test some people are happier to either a) to show off their new SAS, etc., statistical analysis program or b) hide inconclusive results behind irrelevant stats.






*Jimoslimos discloses that he is a named author on a paper funded by a large multinational healthcare company

r3nu4l

Journals and Peer-review
Well the validity of peer-review is in itself a whole other area of debate Even the strength of the panel is no indicator of good science.

I work for a MedComms agency, in my last agency we were writing a paper for a major pharma company. The "author" is an incredibly respected scientist but the results and claims in the paper were so shoddy that the paper was rejected from the first four journals we sent it to.

In the end our client in the pharma company proposed that we send the manuscript to journal X because the Editorial Board members were all Key Opinion Leaders on the books of the Pharma company. :eek: Now journal X is not a bad journal at all, in its specialist field it is well respected. The editorial board are all highly thought of consultants. Lo and behold, the mansucript was submitted and accepted with minor revisions!

As for journals, I agree that a small jornal in a spcialist field can have far more impact in that field than the likes of Science and Nature. Impact factors are also another entirely different debate and one that could go on for pages and pages!

Reading abstracts
Of course the abstract is important but I'm really trying to point out areas where potential bias can easily be spotted

Author disclosures
Don't stop having involvement with Universities etc but be very aware of the association and keep it in mind. I also know how many scientists are proud of their independence and that is a good thing...but equally I sat at in expert panel meeting last July and listened to a consultant dermatologist from Guys Hospital ask our client "What would you like us to conclude? You're paying for this". He was deadly serious. He went on to say that whatever conclusion was needed they could work the discussion to come to that conclusion! :mad:

Also, I remember trying to get my hands on a certain new-anti-fungal drug many years ago. The company in question said they would supply it if we signed a contract that prohibited us from publishing anything without company consent. Reading between the lines, this says, "we control the data and we control what you publish, if we don't like your results, they won't see the light of day."

Many Scientists are put in this position and because these companies also fund equipment purchases in budgets, university labs in particular will usually agree to these contracts.

Statistics
Yes everyone is guilty but most companies have large biostatistical data departments that specialise in running thousands of variations of tests on data to shine the best light on them.

As I said already, the vast majority of papers out there are fine but the source of funding and the agenda of those writing them is important to keep in mind when deciding the validity of results published.

Jimoslimos

r3nu4l wrote:

but equally I sat at in expert panel meeting last July and listened to a consultant dermatologist from Guys Hospital ask our client "What would you like us to conclude? You're paying for this". He was deadly serious. He went on to say that whatever conclusion was needed they could work the discussion to come to that conclusion!

Wow that's scary and even scarier is that its believable:eek:

Overall when it comes to funding and conflicts of interest I'd imagine many scientists would take the view that once they lose any shred of respect amongst their peers the big company sponsoring them would drop funding like a hot potato. Therefore any work published might be seen as a compromise between what the company wants to hear and the scientists own views

InFront

I wish I had time to read scientific papers, when you're studying it's just easier to stick to the textbooks. Not to be too grade-focused, but you can get good grades without approaching these papers

r3nu4l

InFront wrote:

you can get good grades without approaching these papers

True, especially for early year undergrads, however final-year biology undergrads doing a small research project but more importantly research postgraduate biology students need to be aware of these things as they become vital at that stage.

Even at that I think that these things only become truly evident and important as your career progresses.

Jimoslimos wrote:

Overall when it comes to funding and conflicts of interest I'd imagine many scientists would take the view that once they lose any shred of respect amongst their peers the big company sponsoring them would drop funding like a hot potato. Therefore any work published might be seen as a compromise between what the company wants to hear and the scientists own views

QFT! A mjor company I worked with had to withdraw some of their product "Key messages" from a paper last year before the 'author' agreed to put his name to it. There was a lot of negotiation and in the end it was a compromise. Some people really do sell their souls but most I agree are very protective of their own reputation.

tallaght01

A good post, to which we should amend the statement that "Just because you've posted a link to a website which backs up your claims, that doesn't mean you're right" lol.

Dr Galen

this is a great topic. believe me, being able to read research in nursing is so important right now. its beiing churned out at a phenomenal rate across the profession and tbh some of it is nonsense, but taken as gospel, and some of the good stuff is being missed

r3nu4l

tallaght01 wrote:

A good post, to which we should amend the statement that "Just because you've posted a link to a website which backs up your claims, that doesn't mean you're right" lol.

That was actually part of my reason for starting this thread

It is important to know these things when you are reading paper.

nurse_baz wrote:

...and tbh some of it is nonsense, but taken as gospel...

I attended an expert panel meeting last year where the experts (two consultant paediatric dermatologists, one dermatology nurse and one nursing midwife) were very concerned at publications from a particular nurse who has published a raft of review/best practice style papers recently.

In these papers she recommends several products that have recently been shown to be at best non-efficacious, at worst harmful to babies. The problem with that is that the sheer volume of stuff she is publishing is reinforcing her opinions on these products and may influence her peers simply becuse they are published.

As Tallaght01 said, [paraphrase] just because it's published doesn't mean it's actually fully correct[/paraphrase]

InFront

I imagine that also has some (albeit indirect) implications in medical education; staff hierarchy (at least at preclinical level) and promotion potential seems to be based on how many published research papers an academic has his name on. I know this is a really problem in UCD but it's mirrored in Trinity and the RCSI too. I wonder if anyone is measuring the merit of the contributions.

r3nu4l

Well to be honest in pure academic circles promotion is based on a huge number of factors rather than just publications.

For someone running a research laboratory the following are important:

Publications - perceived prestige of journal is counted
Ability to attract funding - very important
Membership of the boards of academic bodies - Royal Irish Academy etc.
Continual turnover of PhD/MSc students - indicates good activity
Contributions to the reputation of the University - (many factors here)
Length of service - it does still count
Who you know and who your allies are - arguably the most important factor

I do know Scientists who are laughed at in their Universities because they publish anywhere they will be accepted. I've heard lecturers make jokes about other lecturers, saying things like "Oh I see Dr. X had another riveting publicaiton in Barbie Weekly last month..." So in some senses publications are important but too many in the 'wrong' journals leads to a perception of weak research.

Henry Kissinger (ex US Secretary of State) apparently once said that the reason he left academia was because he couldn't stand politics, never a truer word spoken

Jimoslimos

r3nu4l wrote:

As Tallaght01 said, [paraphrase] just because it's published doesn't mean it's actually fully correct[/paraphrase]

Never a truer word spoken;) If you have the misfortune of trying to research a topic on which there are two (or more) conflicting points of view this can be a real headache.

Of course naïve little me thinks that the main purpose of publishing a paper would be so others could reproduce experiments and verify results . Certain papers might lack key bits of information due to uncertanity regarding IP/patent rights etc.,

r3nu4l

Jimoslimos wrote:

Of course naïve little me thinks that the main purpose of publishing a paper would be so others could reproduce experiments and verify results

While that certainly is the point and philosophy of publishing I do know scientists who have said openly..."I hope no-one ever tries to repeat those experiments, they'll have no chance"

John

I'm attempting to replicate a paper tomorrow, I'll let you know how it pans out