When peer review breaks down.....

[elfdude]

http://en.wikipedia.org/wiki/Peer_review

Get back to me when you are done reading. Seriously.

Perhaps you should read it yourself. You're arguing about a specific type of peer review and its theoretical approaches in journals as though that covers the entirety of the veil of peer review when in fact every time anything is approached to be applied it is also being peer reviewed. Your attack is coming from a misinformed notion that global warming is a conspiracy and by attacking the peer review journal process you feel you can maintain your intellectual honesty.

The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability—not the validity—of a new finding.

This is what you need to comprehend. Now we also need to point out this quote is from someone discussing peer review only in published journals in the modern system. He is not discussing peer review in its totality (even the article you linked defines different types of peer review) and to try and say he's not all for accountability and checking whether or not science is correct through the independent verification and survival of scrutiny from other scientists in its totality is silly. This delineation is woefully missing in your posts and you're too intelligent to miss that. So not only are you indicating to the layman that you're arguing against the scientific process which is silly but you're also spreading your position that because of this peer review climate change is unreliable despite practical, and empirical peer review that denies this position.

I could easily get behind this thread if it wasn't simply a baseless attack on peer review in general and instead was specific about the issues in the modern peer review process. I'll agree they exist. But to deny that the peer review process is effective at weeding out positively bad science from the pile of ideas is woefully ignorant. More importantly the issue isn't in the peer review process itself but rather in the perception of it's finalness. Science must be seen as a never ending process of peer review or you're doing it wrong.

[Portuguese Rebel]

Personally i think that much of this goes around the fact that specialists in certain fields all know each other. Often a paper is reviewed by guy who, in turn, will be reviewed by him. This leads to certain situations. But in the end there will always be someone willing to test these things when money is on the table.

[Armatus]

But isn't that how it is with anything? the higher you advance the more successful you become the more connected you are?

[Jack04]

Arguably the most important aspect of the entire peer review system:

Postpublication reviews

The process of peer review does not end after a paper completes the peer review process. After being put to press, and after 'the ink is dry', the process of peer review continues as publications are read. Readers will often send letters to the editor of a journal, or correspond with the editor via an on-line journal club. In this way, all 'peers' may offer review and critique of published literature. A variation on this theme is open peer commentary; journals using this process solicit and publish non-anonymous commentaries on the "target paper" together with the paper, and with original authors' reply as a matter of course. The introduction of the "epub ahead of print" practice in many journals has made possible the simultaneous publication of unsolicited letters to the editor together with the original paper in the print issue.

http://en.wikipedia.org/wiki/Peer_re...cation_reviews

[Portuguese Rebel]

We could argue that doing what was described in the OP is a form of peer review.

[Phier]

We could argue that doing what was described in the OP is a form of peer review.

We could argue that if we wanted to be pedantic. The point of peer review is to separate the bad science from the good before it enters the public record. Its a seal of approval. Anyone could just say "this does that!" and you see it every day on websites selling herbs, or from things like homeopathy and chiropractic 'medicine'. What the disturbing thing is, is that peer review on something as important as cancer might as well not have existed. It means people are either being denied or given treatment based on bad science.

The system obviously needs to be improved, and everyone who ever worked in academia has known this a long time, but I don't think the potential magnitude of the problem was quite so apparent.

[Jack04]

Certainly, additional safeguards would be welcome. The trouble is though, that it isn't always practical. You simply cannot have every study repeated by the reviewing body, and even if the entire data set is available, it wouldn't necessarily be possible to find these faults, even if the reviewer had the time to go through every aspect. Peer review is already criticised for the sluggish pace at which it moves, any further slow-down might seriously risk slowing down the pace of development. Unless some quite spectacular levels of extra funding are found, I don't honestly see how the system can be developed.

Saying that, I do think something needs to be done regarding those particularly lucrative areas of study. Perhaps random re-tests (or not so random, if a particular group is found to have consistently reached faulty conclusions) would be practical in certain of those cases, but in the general scientific case, there simply isn't the financial clout available.

Take physics as an example. Beam time at a typical US university might cost somewhere between $5000 and $10000 an hour. The funding to run such experiments is hard enough to come by as it is. How are we supposed to find the money to repeat experiments?

[Portuguese Rebel]

What the disturbing thing is, is that peer review on something as important as cancer might as well not have existed. It means people are either being denied or given treatment based on bad science.

Wait... It is also possible that the experiments have deliberately been presented in a non functional way. Remember that every breakthrough means patents. And something that leads to a cure for a type of cancer is worth billions.

[DimeBagHo]

Dunno if this has been discussed yet: A Sharp Rise in Retractions Prompts Calls for Reform

Apart from the problems with replicating research there has also been a sharp increase the number of published papers that are retracted. The journal Nature reported that over the last decade retractions had increased 1000% while the number of published papers had increased by 44%. The positive spin is that this might be the result of wider online access to journals and thus more scrutiny.

This also seems to be a particular problem with bio-medical research. One idea for reform that strikes me as a good idea would be requiring registration of all studies in advance. That would do a lot to fix the problem of negative results not being published, and the problem of researchers trawling through datasets for correlations.

[Phier]

This also seems to be a particular problem with bio-medical research.

While there might be more there due to monetary possibilities being high, but I'd argue the problem is in all fields to some degree.

[DimeBagHo]

While there might be more there due to monetary possibilities being high, but I'd argue the problem is in all fields to some degree.

There are some common problems across all fields; particularly the pressure to publish for employment and research funding reasons, combined with the lack of interest from journals in null results. Anyone who wants to keep working needs to find something sexy in their data.

[elfdude]

It means people are either being denied or given treatment based on bad science.

First off whaaaat? There is no one given legitimate treatment based on disproven science or even very shakey science. Do you have any idea what loop de loops the medical establishment must go through to have a treatment approved? Yes there are mistakes, there always has been and always will be but as it is I would say US policy borders on the insanely restrictive with regards to applying new scientific knowledge in the medical community.

Secondly how can we be responsible for what happens in unregulated industries? While the medical community has to deal with unfair regulations, most other industries are fair to perpetuate any idea they want. You want to fix the peer review process by regulating industries which would use peer review to perpetuate false ideas? Great. Now we get to regulate religion, homeopathy, just about everything else.

Next you'll be asking for a way to regulate whether news groups say truthfully accurate and useful information. Maybe something akin to this:http://en.wikipedia.org/wiki/Fairness_Doctrine

I'm actually all for these rather liberal ideas from you.

[DimeBagHo]

First off whaaaat? There is no one given legitimate treatment based on disproven science or even very shakey science. Do you have any idea what loop de loops the medical establishment must go through to have a treatment approved? Yes there are mistakes, there always has been and always will be but as it is I would say US policy borders on the insanely restrictive with regards to applying new scientific knowledge in the medical community.

Many medical interventions are not supported by any empirical evidence at all (though there is a movement to change this called evidence based medicine). When a new drug is developed the manufacturers have to show two things: that the drug is reasonably safe and that it is effective for treating at least one medical condition. But lots of drugs are used for so called "off label" purposes, and drugs are far from the only kind of medical intervention.

[Gaidin]

Considering the insane tests a drug has to go through to be considered for use of any kind on the market, 'off label' or not, I'm not completely sure what your point is in questioning Elfdude's point on the insanely complex loops the medical community has to jump through to use new developments, medicinal, technological, or otherwise.

[Adar]

Arguably the most important aspect of the entire peer review system:
http://en.wikipedia.org/wiki/Peer_re...cation_reviews

It is important but no one is funding it, which is why I made my proposals on page one.

[Phier]

Considering the insane tests a drug has to go through to be considered for use of any kind on the market, 'off label' or not, I'm not completely sure what your point is in questioning Elfdude's point on the insanely complex loops the medical community has to jump through to use new developments, medicinal, technological, or otherwise.

If the science is bad to start with, hoops that should not be jumped through are, and hoops that should be jumped through are not.

And in this case it was a MAJORITY of the papers which were not reproducible. Let that sink in a bit.

88.7% were not reproducible. Anyone who thinks this is "ok" needs to stay out of research sciences.

[Gaidin]

If the science is bad to start with, hoops that should not be jumped through are, and hoops that should be jumped through are not.

And in this case it was a MAJORITY of the papers which were not reproducible. Let that sink in a bit.

88.7% were not reproducible. Anyone who thinks this is "ok" needs to stay out of research sciences.

Medicine is more applied science than science. Engineering, if you will. If the organic chemistry research can't stand up to review, its not going to make it to(much less through) the safety tests the medical community has to go through(an entirely different ballgame as they're not testing the bedrock science persay). As an electrical engineer I don't review a physicist's work in maxwell's equations persay. If the bedrock science doesn't work, I can't develop a product based on it. Same thing here.

Find me a question that can't be raised about the statistical evaluations they have to do and I'll show you a question that has nothing to do with statistics. But that's not the bedrock science used to develop the pill or the xray or the pacemaker. Nevermind the fact that doctors finally discovered calculus only a decade ago. You guys are talking about two different things.

[crazybanana]

I read this breakdown of the article and paper over on scienceblogs. Seems to clear a couple of things up.

http://scienceblogs.com/insolence/20..._a_problem.php