Peer review

[Denny Crane!]

I'm so damn busy with work I will not be giving a topic but a question. Unfortunately time doesn't allow for me to participate but I'd love to read your thoughts on the matter.

It is a simple question on the surface and one that will probably end in a relatively simple answer. Is peer review and consensus amongst academia fool proof?

If you wish to subdivide this question into different fields of academia that might be an idea.

[Bovril]

No, definately not. Loads of nonsense gets through peer review. Less nonsense becomes accademic concensus, but some things that accademics even in the hard sciences agreed on a while back seem absurd to people now. All of which is not to say that peer review and attempts to reach accademic concensus aren't good things, they clearly are, but they are far from infallable. More specific answers will be given, but I'd be amazed if any diverged greatly from this basic sentiment.

[Denny Crane!]

No, definately not. Loads of nonsense gets through peer review. Less nonsense becomes accademic concensus, but some things that accademics even in the hard sciences agreed on a while back seem absurd to people now. All of which is not to say that peer review and attempts to reach accademic concensus aren't good things, they clearly are, but they are far from infallable. More specific answers will be given, but I'd be amazed if any diverged greatly from this basic sentiment.

The most obvious one which is being debated in the mudpit here is that climate scientists agree, there is peer reviewed evidence and academic consensus and anyone who agrees with it is a lunatic conspiracy theorist.

That is not why this thread was created and I'd certainly hate this thread to become 'that' topic but it is a pertinent relevant example. I was actually more thinking of another field when I brought about this topic but this is more high profile.

[Tankbuster]

The most obvious one which is being debated in the mudpit here is that climate scientists agree, there is peer reviewed evidence and academic consensus and anyone who agrees with it is a lunatic conspiracy theorist.

That is not why this thread was created and I'd certainly hate this thread to become 'that' topic but it is a pertinent relevant example. I was actually more thinking of another field when I brought about this topic but this is more high profile.

The thing is this. No, peer review is not foolproof, but it's quite a powerful tool to get to truth (that's to say: we've had plenty of luck with it in the past). So having peer reviewed evidence and academic consensus on your side is a very powerful argument.

So to someone who disagrees with peer reviewed studies, I always say: "Then I hope you know a great deal about this subject." If you don't (such as many of the skeptics on any subject) then you really are a lunatic conspiracy theorist.

[Bovril]

Mmmm, the thing about debates over climate change is that, although I don't have a problem with climatologists disagreeing, and a few do, I do have a problem with non-experts trying to pass themselves off as better informed than the overwhelming majority of scientists. It seems like a grossly arrogant claim for a non-specialist to make.

But yeah, getting back to the broad question, if we agree that peer review is generally a good way of filtering research 9though sometimes poorly implemented), and non-experts should take accademic concensus seriously, and that concensus of some sort is very usefull for research, what is there is to discuss? Perhaps you would like to talk about that specific example you had in mind?

[G-Megas-Doux]

Mostly no, an educated fool is still a fool. If there are ten people and only one of them has an excellent intelligence in the form of investigation, lateral thinking, the ability to empathically see connections make the jump then research it back to prove the connection or disprove it to remove the connection from the field of research; and the others have weaker intelligence of slower wit dogmatic opinions and memorised results these people will outnumber the first and peer review just becomes might is right.

The main problem is that peer review sometimes allows for foolish ideas to come to the fore and intelligent ideas to be left behind. So it is important not to accept peer review as the final analysis but to question the results. This is much like an appeal against a court decision, peer review is equally capable of miscarriage of justice, then it is used by others as a giant club to beat someone over the head and discussions become well nine experts say this and one expert says this, numbers don’t always mean correct.

[Bovril]

Whilst the above is sometimes true, there are often 'dissident' journals one can publish in, or just journals with low standards. And the proceedings of some conferances are not reviewed at all. A bigger problem is that certain eminent people get their work published no matter what, even if its quite lazy. This is an especially big problem when prominent accademics stick their name onto someone else's research to help them get it published, and to raise their own numbers of published papers.

[Denny Crane!]

Hmm it is interesting. Here is a dilemma then. If someone comes out with a drug, and there are skeptics about that drug; some of which who may well fall into that lunatic conspiracy theorist bag, and a doctor advises you to take it what level of research would you undertake to check that drug out and potentially look for alternatives. Given that most people are idiots when it comes to medicine, yet tragedies have happened like thalidomide and in less severe cases the detrimental traits attributed to anti imflammatory drugs for the long term control of pain.

Similarly with climate change there is another dilemma. If peer reviewed academic science has its flaws which of course I was aware of but fancied a conversation on the topic but is as TB says a very powerful tool at what level can it be used as a bludgeon against us in representative democracy. We are being told to make vast changes to our lives on the basis of this. My willingness to learn about the topic is limited as is my interest, however, being forced to pay for something or make changes has the following problems. One is that it is such a vastly complicated topic that in certain places they are far from understanding how to interpret changes like climates around the poles according to one source familiar with the subject so while the consensus may be drawn that something is happening the models are flawed. With this in mind governments are going to start, or have already, appropriating resources from us to combat this. In such a complicated topic with such far reaching ramifications do we actually think a government who can barely run a highway system can institute change, and of what benefit when compared to the lack of action in countries that dwarf the emissions. The leap from academic consensus to the necessity of legislative action seems like a rather large gap to bridge when the science is questionable and the action relative to that highly suspect.

[G-Megas-Doux]

In such a complicated topic with such far reaching ramifications do we actually think a government who can barely run a highway system can institute change, and of what benefit when compared to the lack of action in countries that dwarf the emissions. The leap from academic consensus to the necessity of legislative action seems like a rather large gap to bridge when the science is questionable and the action relative to that highly suspect.

Currently I view the government as a destitute bankrupt with a gambling addiction and a penchant to investing in crackpot ideas. Would you lend money to this guy?

The main problem you have hit upon is that in almost all instances people are idiots, they are even heading in the wrong direction of what they think they know about. Who will let it be said of them that they don’t know their own topic and as such they will back their own first opinions with greater zeal to the detriment to all. It is an inevitable social flaw in all areas of society. Peer review is just another tool that works but can be misused. Remember a bad workman blames his tools. Society is very much an industry cowboy.

[Bovril]

Hmm it is interesting. Here is a dilemma then. If someone comes out with a drug, and there are skeptics about that drug; some of which who may well fall into that lunatic conspiracy theorist bag, and a doctor advises you to take it what level of research would you undertake to check that drug out and potentially look for alternatives. Given that most people are idiots when it comes to medicine, yet tragedies have happened like thalidomide and in less severe cases the detrimental traits attributed to anti imflammatory drugs for the long term control of pain.

In medical research there is a whole new problem, that of time. It doesn't desperately matter how long it takes for archaeologists to argue over the dating of the various layers of an excavation. No one is going to die as a result. But with drugs, there often are people who will live or die depending on whether they get it, or will experience alot of pain, impairment or have their life spans reduced by not getting it. So this needs to be balanced against the need for longitudinal studies which, to be absolutely comprehensive, could take a lifetime, litterally. Most totally new drugs go through several stages of trialing. First on animals in various doses. Then on humans in tiny doses, then in therepeutic doses, then on a larger scale, and then are liscenced. Other drugs are given new liscences for other uses. For example, an anti-epileptic may be used as a mood stabliser or a painkiller used as a blood thiner. This also requires stages of trialing. But despite all this people can get it wrong. With thalidomide, for example, the number of people effected compared to the number of women who took the drug was tiny, and could not have been expected to be picked up at trial. Also, many drugs have potentially severe side effects, but very pronouned benefits. For example, there are some anti-depressants which increase the risk of diabetes. But their efficacy when other classes of anti-depressant have failed reduces incidence of mortality and raises quality of life on average.
In the end, its about judgement calls, and the call is going to be different for different types of drug. Drugs to reduce miscarriage by increasing levels of estrogen caused female babies to devellop male characteristics (estrogen is converted into androgens during key devellopmental stages). This drug was shown by independent research to increase risk of miscarriage by all independent studies. So clearly its liscencing on the say so of its develloper was scandalous. This genuine scandal is somewhat comparible to that of thalidomide, but thalidomide really did help mothers during pregnancy, and the negative effects really were hard to pick up on.
Sorry for going on about specifics, but I think this issue is basically reducible to an agrogate of specifics. There is no general rule other than perhaps to make sure research is independent.

Similarly with climate change there is another dilemma. If peer reviewed academic science has its flaws which of course I was aware of but fancied a conversation on the topic but is as TB says a very powerful tool at what level can it be used as a bludgeon against us in representative democracy. We are being told to make vast changes to our lives on the basis of this. My willingness to learn about the topic is limited as is my interest, however, being forced to pay for something or make changes has the following problems. One is that it is such a vastly complicated topic that in certain places they are far from understanding how to interpret changes like climates around the poles according to one source familiar with the subject so while the consensus may be drawn that something is happening the models are flawed. With this in mind governments are going to start, or have already, appropriating resources from us to combat this. In such a complicated topic with such far reaching ramifications do we actually think a government who can barely run a highway system can institute change, and of what benefit when compared to the lack of action in countries that dwarf the emissions. The leap from academic consensus to the necessity of legislative action seems like a rather large gap to bridge when the science is questionable and the action relative to that highly suspect.

There is a tiny amount of debate amongst people we should pay attention to over whether humans have a massive impact on climate change. And I do mean tiny. There is more debate over how great the problem is. But governments never have all the information when they make policy changes, large or small. Governments go to war on far, far shodier evidence than the evidence for climate change. If we had taken the money spent by the UK or US on the Iraq war and spent it on alternative energy we would have David King (former scientific adviser to the UK government and eminant earth science accademic) have created nigh on carbon nuetral energy grids. Besides, shifting away from fossil fuels is something we are going to have to do anyway, so why not make those investments now?

[Denny Crane!]

In medical research there is a whole new problem, that of time. It doesn't desperately matter how long it takes for archaeologists to argue over the dating of the various layers of an excavation. No one is going to die as a result. But with drugs, there often are people who will live or die depending on whether they get it, or will experience alot of pain, impairment or have their life spans reduced by not getting it. So this needs to be balanced against the need for longitudinal studies which, to be absolutely comprehensive, could take a lifetime, litterally. Most totally new drugs go through several stages of trialing. First on animals in various doses. Then on humans in tiny doses, then in therepeutic doses, then on a larger scale, and then are liscenced. Other drugs are given new liscences for other uses. For example, an anti-epileptic may be used as a mood stabliser or a painkiller used as a blood thiner. This also requires stages of trialing. But despite all this people can get it wrong. With thalidomide, for example, the number of people effected compared to the number of women who took the drug was tiny, and could not have been expected to be picked up at trial. Also, many drugs have potentially severe side effects, but very pronouned benefits. For example, there are some anti-depressants which increase the risk of diabetes. But their efficacy when other classes of anti-depressant have failed reduces incidence of mortality and raises quality of life on average.

In the end, its about judgement calls, and the call is going to be different for different types of drug. Drugs to reduce miscarriage by increasing levels of estrogen caused female babies to devellop male characteristics (estrogen is converted into androgens during key devellopmental stages). This drug was shown by independent research to increase risk of miscarriage by all independent studies. So clearly its liscencing on the say so of its develloper was scandalous. This genuine scandal is somewhat comparible to that of thalidomide, but thalidomide really did help mothers during pregnancy, and the negative effects really were hard to pick up on.
Sorry for going on about specifics, but I think this issue is basically reducible to an agrogate of specifics. There is no general rule other than perhaps to make sure research is independent.

Yes it is entirely dependant on specifics I agree. I am thinking specifically of a business I was involved in with an unlicensed herb that basically reduced arthritic pain massively, far more than any painkiller and massively slowed necrosis of the joints. This along with other treatments like it meant that I was exposed to a world where antibiotics, painkillers that are literal killers and other overuses of treatments are the norm and no one especially not the qualified vets who administered them questioned them, in fact due to profit based motives they were known to push them based on lies. This all came to a head in my mind and motivated this topic when I came across a human case, the manufacturer whose age induced arthritis forced him on medication that did a number on his liver. The herbal treatment was an effective replacement. In this respect people don't investigate what they give their animals as they are blinded by ignorance and put to much faith in institutions. Similarly we are willing to take such risks with our own health and nutrition because there is an assumption of faith.

I don't mean to attack academia or peer review, but that dishonesty or ambivelence is a culture that should be countered if possible and it might enhance the current situation.

There is a tiny amount of debate amongst people we should pay attention to over whether humans have a massive impact on climate change. And I do mean tiny. There is more debate over how great the problem is. But governments never have all the information when they make policy changes, large or small.

So they shouldn't make them.

Governments go to war on far, far shodier evidence than the evidence for climate change.

Well it is actually being proven right now in the UK that they will go to war and make up the information to justify it, so yes the blood of 500 000 iraqis is on their hands and it should have implications on how we trust them to use evidence from scientists.

Furthermore they recently sacked a government scientific adviser almost provoking a full scale revolt in the UK amongst the scientific community because evidence went against populist drug policy.

If we had taken the money spent by the UK or US on the Iraq war and spent it on alternative energy we would have David King (former scientific adviser to the UK government and eminant earth science accademic) have created nigh on carbon nuetral energy grids. Besides, shifting away from fossil fuels is something we are going to have to do anyway, so why not make those investments now?

Because the government is pretty much incapable. David King for all his intelligence was either whimsical or just idiotic to make that statement. There is no way the government could or would do such a project. Besides which there are technologies developing that could well render any such investment a waste. The progress of biofuels in a decade is analgous to that of computers in the 80's. Imagine what will happen in another ten years. No one thinks corn fuel could work, but that is the distant past.

[Bovril]

Yes it is entirely dependant on specifics I agree. I am thinking specifically of a business I was involved in with an unlicensed herb that basically reduced arthritic pain massively, far more than any painkiller and massively slowed necrosis of the joints. This along with other treatments like it meant that I was exposed to a world where antibiotics, painkillers that are literal killers and other overuses of treatments are the norm and no one especially not the qualified vets who administered them questioned them, in fact due to profit based motives they were known to push them based on lies. This all came to a head in my mind and motivated this topic when I came across a human case, the manufacturer whose age induced arthritis forced him on medication that did a number on his liver. The herbal treatment was an effective replacement. In this respect people don't investigate what they give their animals as they are blinded by ignorance and put to much faith in institutions. Similarly we are willing to take such risks with our own health and nutrition because there is an assumption of faith.

I don't mean to attack academia or peer review, but that dishonesty or ambivelence is a culture that should be countered if possible and it might enhance the current situation.

There are far too many cases like the one you describe. Unfortunately governments cut costs when assesing the safety of drugs. International bodies are often bad things, but I think an international body on drug safety research could really help to cut costs through economies of scale and then governments could decide individually. Equally, doctors who get kick backs from companies, companies who offer the first few prescriptions for free or at cut down prices and state funded doctors who try to cost cut at the expense of patients are all causing suffering and, sometimes, death.

So they shouldn't make them.

Being something like an anarchist agree with you in the abstract. However, the sort of risk aversion you are proposing would cripple all institutions including governments.

Well it is actually being proven right now in the UK that they will go to war and make up the information to justify it, so yes the blood of 500 000 iraqis is on their hands and it should have implications on how we trust them to use evidence from scientists.

Of course I agree that the Iraq war was an abhorant abuse of evidence. The point I'm making is that governments by their nature make decision based on less than perfect information (just like all institutions or individuals) so negative effects are bound to result sometimes. But this is better than doing nothing.
(the Iraw example was probably a bad one, since I don't think many policy makers actually believed Iraq posed any threat to anyone except Iraqis. None the less, poor decision based on imperfect evidence are made every day in all big institutions).

Furthermore they recently sacked a government scientific adviser almost provoking a full scale revolt in the UK amongst the scientific community because evidence went against populist drug policy.

Absolutley. The fact the government is willingfully ignoring scientific advice on this issue is despiciable. Equally, their ignoring scientific advice on climate change is also despicable.

Because the government is pretty much incapable. David King for all his intelligence was either whimsical or just idiotic to make that statement. There is no way the government could or would do such a project. Besides which there are technologies developing that could well render any such investment a waste. The progress of biofuels in a decade is analgous to that of computers in the 80's. Imagine what will happen in another ten years. No one thinks corn fuel could work, but that is the distant past.

King may have been exagerating (though he does specialise in policy, including costs, related to alternative energy), but the point still stands. Governments spend huge amounts of money on projects far more dubious than reducing CO2 output.

[Phier]

Peer review is just a a review of your work by your peers.

Being those peers are human, mistakes will be made.

Still its the BEST tool we have to eliminate 'bad' science as a rule.

Peer review should only base its 'judgments' on if the methods used and conclusions are sound based on whatever evidence was given. Ideally what those conclusions are wouldn't play a part in the decision as long as the paper was scientifically valid. This is where we get a bit fuzzy on the climate side as some people are more worried about the message than the science.

Because of the politics that can be involved in some subjects, and we are not only talking climate, it does fail a bit at times. The other side of this though, is that when legitimately criticized papers get slammed in peer review, the authors, especially if they are on the fringe or just dont' know what they are doing, will claim a conspiracy against their 'work'.

[Morkonan]

Peer review is just a a review of your work by your peers.

Being those peers are human, mistakes will be made.

Still its the BEST tool we have to eliminate 'bad' science as a rule.

Peer review should only base its 'judgments' on if the methods used and conclusions are sound based on whatever evidence was given. Ideally what those conclusions are wouldn't play a part in the decision as long as the paper was scientifically valid.

I'm quoting the above as a virtual duplicate of the answer I would have given.

...Because of the politics that can be involved in some subjects, and we are not only talking climate, it does fail a bit at times. The other side of this though, is that when legitimately criticized papers get slammed in peer review, the authors, especially if they are on the fringe or just dont' know what they are doing, will claim a conspiracy against their 'work'.

Ideally, peer review would be as you stated in the earlier portion of your post - An unbiased examination of the paper for scientific merit by peers within the subject discipline. However, what seems to happen in some cases is that particularly new areas of scientific investigation following radical lines of questioning don't have many peers familiar with the work itself. So, what scientific merit that may be there can sometimes be overlooked until the rest of associated fields catch up. Fortunately, because of the nature of strictly regimented scientific investigation, only a very few notable achievements get overlooked as most still fall within the realm of being able to pass credible peer review no matter how radical they may seem: Res ipsa loquitur

But, I will also say there is a definite "conspiracy" in place within Science. But, it's an open conspiracy against "Not Science." Some psuedo-science definitely falls within the category of Not Science and is easily dispensed with. Unfortunately, psuedoscientists are quick to popularize their claims and gain support from those who would not be considered appropriate peers to begin with. Thus, teh interwebz "woo" was born...

[Oswald von Wolkenstein]

*

[Denny Crane!]

Currently I view the government as a destitute bankrupt with a gambling addiction and a penchant to investing in crackpot ideas. Would you lend money to this guy?

The main problem you have hit upon is that in almost all instances people are idiots, they are even heading in the wrong direction of what they think they know about. Who will let it be said of them that they don’t know their own topic and as such they will back their own first opinions with greater zeal to the detriment to all. It is an inevitable social flaw in all areas of society. Peer review is just another tool that works but can be misused. Remember a bad workman blames his tools. Society is very much an industry cowboy.

This is the trouble I suspect. It doesn't matter if bad ideas are propagated academically as sooner are later it will surely be sorted out.

It is when people start extracting vast amounts of resources and using them based on bad ideas, or when people act on those ideas and hurt each other that it becomes a problem.

[G-Megas-Doux]

This is the trouble I suspect. It doesn't matter if bad ideas are propagated academically as sooner are later it will surely be sorted out.

It is when people start extracting vast amounts of resources and using them based on bad ideas, or when people act on those ideas and hurt each other that it becomes a problem.

Can anyone think of earlier examples of peer review than theology (including ancient times)? I'm not kidding I cant and look what happened to Galileo.

[Avendiel]

Peer review among academia in the humanities is an entirely subjective, unscientific, and largely nepotistic process by which those who chant the idiocy du jour get rewarded with a publication.

wow I'm so glad I got out of the humanities

[Ummon]

Peer review is like democracy: the worst system unless we consider all others that have been invented.

[Phier]

Denny to quote Tim Minchin, you know what they call alternative medicine thats been proved to work? Medicine.

I'm not quite sure what you are getting at. Was it that the peer review didn't support this herb or was it that people rejected it because its properties were not scientifically proven and subjected to peer review?