Wikipedia talk:Identifying reliable sources (medicine)

Proposal re retrospective studies and general epidemiological data-mining

Following User:Zad68's suggestion above and addressing User:AuburnMagnolia's request, I suggest that we add the following to the definition of "Primary source": "Publications that present the results of retrospective studies or the results of epidemiological data-mining, in which a hypothesis is investigated by searching publications or databases, are primary sources; these publications are different from secondary and tertiary sources that present the scientific consensus." I have run into the issue of people pushing studies like the one discussed above as secondary, and it could be useful to have explicit language on it. Jytdog (talk) 17:52, 8 February 2014 (UTC)

Thanks, Jytdog, I think this is a great idea. AuburnMagnolia (talk) 13:00, 13 February 2014 (UTC)

External links in refs

Hi, now I think that where possible when we reference open-access articles like those in PMC we should write down the url of said article in PDF format for easy access. I've tried this on the Huntington's disease page, but for some reason it was reverted. Thoughts?

Example:

Baughman, RP (March 2011). "A concise review of pulmonary sarcoidosis" (PDF). American journal of respiratory and critical care medicine. 183 (5): 573–81. doi:10.1164/rccm.201006-0865CI. PMC 3081278. PMID 21037016. {{cite journal}}: Unknown parameter |coauthors= ignored (|author= suggested) (help)

Fuse809 (talk) 13:22, 19 February 2014 (UTC)

Toxicity and MEDRS

I'm sure this issue has been covered before, but I'm having trouble finding it. My question is a general one, which I will illustrate using some examples from the Glyphosate article. Specifically these questions revolve around issues of how to take into account that much/most toxicity data that is available, and that is used by regulators, seems to fall outside the requirements of MEDRS as I have understood it.

"The EPA considers glyphosate to be noncarcinogenic and relatively low in dermal and oral acute toxicity.[19]" Are regulatory reviews by national environmental and medical authorities normally secondary sources? Tertiary?

"A study published in 2000 found that Roundup interfered with an enzyme involved in testosterone production in mouse cell culture.[80] A study by the Seralini lab published in 2005 found that glyphosate interferes with aromatase, an estrogen biosynthesis enzyme, in cultures of human placental cells and that the Roundup formulation of glyphosate had stronger such activity.[81]" Again, these seem to be excluded by MEDRS, but any such result observed as part of pharmaceutical development program would absolutely have to be reported to FDA, and would be considered a major black mark in the evaluation process. So again I find myself leaning toward deleting a statement as not sufficiently proven for Wikipedia that regulatory authorities would take very seriously. Another interesting question that arises here is that these studies were done at unrealistically high concentrations - how to note that sort of thing without being guilty of injecting original research?

"A study on rats published in 2010 found that administering Roundup Transorb orally to prepubescent rats once a day for 30 days, reduced testosterone production and affected testicle morphology, but did not affect levels of estradiol and corticosterone.[83]" Same issue. Seems to be excluded by MEDRS, but one does not perform such experiments on people and most of what is known about the toxicity of many substances comes from animal models. And again in this particular case the study is very poor quality because the doses used were massive and vastly out of proportion to any likely human exposure. I don't like killing it based on the fact that it is an animal study, but its original research if I mention that the doses used in the study are above the LD50 of table salt. How to proceed?

"Various other studies suggest that glyphosate may be mutagenic.[87]" - Just illustrating once again that genotoxicity studies are an important part of EPA and FDA decision making, but as in vitro results, seem to be excluded by MEDRS.

No example of this in the glyphosate article, but noting here that the FDA also makes decisions to put warnings on drug labels and even to withdraw drugs based on aggregate case reports to AERS. But case reports are excluded by MEDRS. Some adverse events are important enough to result in regulatory action even if too rare to be seen in clinical trials. I have mixed feelings about this case report rule, and wonder if there might be some exception for aggregated case reports as covered in a review.

Any clarification and thoughts would be appreciated. I'm really struggling to understand this issue. Formerly 98 (talk) 13:07, 27 February 2014 (UTC)

I have not given the information you are presenting much review, and have not looked at it at all in context, but generally, the idea is to exclude primary sources and cite secondary sources. MEDRS sources typically will cite and be derived from non-MEDRS sources. For issues of toxicology, what happens is that someone reports small-scale research which is not MEDRS compliant. No conclusions can be drawn from this research alone. However, when an organization like the EPA or FDA cites those research findings, then because of that their credibility increases. That may or may not make the government paper a MEDRS source, but one cannot be certain without examining the document because there are a range of quality among government sources. If the government gives an opinion or has a conclusion then that is worth mentioning only by virtue of it being from a major stakeholder and that has nothing to do with MEDRS at all, although often major organizations will back their statements with research which would also confirm it applicable under MEDRS. Is that a start toward helping? Blue Rasberry (talk) 15:20, 27 February 2014 (UTC)

Yeah, I think that's good. It just seems troubling to me that in a lot of ways the MEDRS rules seem designed mostly to address issues of efficacy (where most subjects are expected to show an effect of exposure) and not so much toxicity (where a serious adverse effect in 0.1% of exposed persons is significant, but difficult to detect via the usual RCT gold standard). Because of the lower frequency of events in toxicology and because of ethical considerations, toxicology knowledge is much more likely than efficacy knowledge to be based on lower quality evidence such as in vitro studies, animal studies, and case reports. Furthermore (and possibly because it is lower quality data), my experience is that reviews of tox data are fewer and farther between than efficacy reviews in most cases.

So I find myself worrying that MEDRS, if systematically enforced (its often not), would systematically bias Wikipedia toward an unrealistically high benefit/risk ratio of drugs, ag chemicals, etc. So maybe this is a philosphical concern more than an interpretation one. Would you delete any or all of the specific items above if only primary references could be found? Formerly 98 (talk) 15:53, 27 February 2014 (UTC)

I totally support adding language to MEDRS specific to toxicity as per my previous bellyaching on this. I was invited to create an essay for consideration which I haven't done yet. my bad. i very much appreciate Formerly bringing this to everyone's attention, as it remains a problem. Jytdog (talk) 16:28, 27 February 2014 (UTC)

We want sources that take into account all the primary sources and give them due weight. This is often a regulatory agency or a toxicology textbook. Both of these would be good sources. We do not want to have people selecting their favorite primary sources as than they could write anything they want as there are nearly always sources that go both ways. Doc James (talk · contribs · email) (if I write on your page reply on mine) 22:46, 27 February 2014 (UTC)

Reviews by from regulatory agencies are still reviews. Reviews of case studies are still reviews.

Some regulatory documents are primary, but you may WP:USEPRIMARY. It'd be silly to demand a systematic review on whether the FDA approved a drug or not: you can perfectly well cite a primary source, such as the FDA's announcement of their decision for this non-biomedical fact. WhatamIdoing (talk) 01:01, 6 March 2014 (UTC)

"Primary sources should generally not be used for medical content"

I propose we add the following line to WP:MEDRS to clarify things for people. Doc James (talk · contribs · email) (if I write on your page reply on mine) 19:27, 27 February 2014 (UTC)

SUPPORT::I think it would cause a few problems (including exacerbating the ones in the section above to some extent), but would solve more than it caused.

I am a little new here to be weighing in on policy, but my impressions as a somewhat new guy and former industry guy are as follows:

• I see a lot of Admins and senior editors telling people that a secondary source is required for medical information, but when I read the MEDRS text as it currently stands, it only seems to recommend secondary sources, not require them. It would be good for the Admins and the guidance to be very clearly saying the same thing. At a minimum, pick one or the other but speak with one voice.

• Requiring secondary references as the source of all medical information would be a very powerful way of keeping a cap on those who want to use Wikipedia to WP:RGW or to "enlighten" the mis-informed mainstream experts. It might aggravate some of the issues I refer to in the section above, but I think it would be an overall improvement.

• On the other hand, as I noted above, I'm worried that the more or less blanket prohibition on results from animal studies and in vitro testing is not reflective of how toxicology is practiced in industry or in the regulatory world. I won't repeat what I wrote above, but I'd like to see the restrictions on toxicology results loosened in a few specific situations. The most important of these would be carcinogenicity and reproductive toxicology testing. Most of what we know about the carcinogenicity and repro tox of most substances is derived from animal studies and in vitro genotox studies. I'd like to see these data allowed provided they are taken from secondary references. Since case reports are frequently the basis for regulatory action in the case of rare but severe side effects, I think there should be some point at which they are allowed in, though the threshold should be high. Perhaps it is arbitrary, but the threshold I would suggest is that if a nationally competent regulatory authority takes action on the basis of AE case reports, the agency memo documenting that regulatory action and the reasoning behind it should be considered a MEDRS compliant source.

I've suggested to Jytdog that we contact the Society for Toxicology to find an expert interested in helping us set guidelines for toxicology reliable sources. I'd be interested to hear what people think of that. Formerly 98 (talk) 21:10, 27 February 2014 (UTC)

Positions of national bodies is deemed to be a high quality source. So are major medical textbooks. If these comment on the animal studies and invitro testing than so should we. (keeping in mind due weight of course) Doc James (talk · contribs · email) (if I write on your page reply on mine) 22:41, 27 February 2014 (UTC)

It would be nice to get this language clarified in the MEDRS document, as it currently seems to read "generally omit in vitro and animal data" or at least "don't imply that it is relevant in peoople". Overall the language appears non-clearcut, which may simply represent the fact that it is a compromise document with many fathers/mothers. For drugs, this information will usually be available from the FDA approval summary, available through the drugs@fda website. Formerly 98 (talk) 01:30, 28 February 2014 (UTC)

Yes agree that bit needs fixing too. If this data is emphasized in high quality secondary sources we should have it too. Much tox research is based on the precautionary principle and extrapolation from animals as the trials is people would be unethical. Doc James (talk · contribs · email) (if I write on your page reply on mine) 02:39, 28 February 2014 (UTC)

I think that the guideline is already sufficiently clear, in its Definitions, Basic advice, Use up-to-date evidence and Choosing sources sections, that primary sources generally should not be used for medical content, though it does note exceptions. I don't think that Jmh649 (Doc James)'s proposal is going to have the type of benefit that Formerly 98 is suggesting, but we can certainly try it. Flyer22 (talk) 21:56, 27 February 2014 (UTC)

SUPPORT. a lot. Jytdog (talk) 21:57, 27 February 2014 (UTC)

Jytdog, I think the reason some of us see a disconnect is that of the 4500 words of the MEDRS article, nearly 20% (and mainly in the first third of the guideline) describe when and how it is OK to use primary references. At a minimum its confusing. Maybe its just 'undue weight". : > ) Formerly 98 (talk) 22:26, 27 February 2014 (UTC)

i think you are right about that. there have been loads of loads of discussions on this Talk page about primary sources. and it will be hard to get consensus to ban them outright. Your comment about WEIGHT is awesome; I agree it should be toned down a lot. But there is no explicit sentence like the one that DocJames proposed and it would be great to get it in. It can be a separate proposal to reduce the weight on primary sources... maybe it would make sense to have a new section called "on primary sources" and consolidate and rationalize it. btw, with respect to what you wrote above, about tox... especially for tox we want only secondary sources. otherwise we open the door for every stupid paper where some basic researcher dumps a bunch of chemical right on cells and kills and them and goes "look look X is toxic!!" argh. so i deadly oppose (!) opening the door to primary sources for tox. i want it closed more tightly. the regulatory action itself becomes the source we cite, btw! Jytdog (talk) 22:36, 27 February 2014 (UTC)

Oppose - This was closed too fast. What about primary sources on human trials? Oppose as is. - Sidelight12 ^Talk 04:44, 5 March 2014 (UTC) *rationale 05:03, 5 March 2014 (UTC)

Yes primary source on human trials should not be used. Many primary human trials go unpublished as the results are negative. The positive ones are published. A proper review article will take into account both the published and unpublished results. Doc James (talk · contribs · email) (if I write on your page reply on mine) 04:51, 5 March 2014 (UTC)

Also the category of human trials can span an enormous range of trial design and execution quality and therefore result quality and statistical power. Zad68 05:09, 5 March 2014 (UTC)

This is a drastic proposed change, I think we should get more opinions. Certainly, abstract human primary studies, especially where there is open-ended interpretation, can be prohibited. I'm uncertain if everything falls under that category. - Sidelight12 ^Talk 05:57, 5 March 2014 (UTC)

It is not a drastic change. This is how this guideline is currently interpreted. This is simply clarifying how most people interpret the guideline right now. Doc James (talk · contribs · email) (if I write on your page reply on mine) 06:10, 5 March 2014 (UTC)

Hey User:Sidelight12 - the spirit of MEDRS (and wikipedia in general) is that we provide content that is reliable. As MEDRS already says, in the health arena publications get released and the press goes gaga quite often; we need to rely on secondary sources to tell us how big of a deal the work and conclusions in the primary source are - we don't decide ourselves nor do we follow popular media. We take the long view guided by reviews. So, so important and fundamental. I generally find that when people are pushing content based on primary sources into articles, they are grinding an ax and/or treating wikipedia like it is "science news" instead of an encyclopedia; neither are good. Jytdog (talk) 15:01, 5 March 2014 (UTC)

It is not drastic, it's a plain-language restatement of the current consensus regarding the application of WP:MEDRS. Sidelight removed the sentence, the sentence should be restored to the guideline. Zad68 15:13, 5 March 2014 (UTC)

Support this and even stronger wording if support can be developed for it. Zad68 05:07, 5 March 2014 (UTC)

Support as this is basically codifying what we already do with regards to use of primary sources Yobol (talk) 15:40, 5 March 2014 (UTC)

It has been open a week with substantial support. Thus added the text in question again. Doc James (talk · contribs · email) (if I write on your page reply on mine) 18:40, 5 March 2014 (UTC)

• Three more supports here: [1] Doc James (talk · contribs · email) (if I write on your page reply on mine) 23:24, 5 March 2014 (UTC)

• note: It's been less than a week, and quantification of support needs to be expressed here. Take into consideration RexxS's [2] and Formerly 98's comments, and human tests that aren't based on abstract measurements. Until these are addressed, I oppose it. - Sidelight12 ^Talk 04:52, 6 March 2014 (UTC)

• Support. This looks like a straightforward clarification. Axl ¤ [Talk] 00:27, 6 March 2014 (UTC)
• Support. This is a reformulation of what WP:RS already says. It has very broad support from current medical editors, and is an absolutely vital defense in the battle against poor quality hyped-up content. JFW | T@lk 12:58, 7 March 2014 (UTC)

• Support. But I think we should italicise, "generally". Like, for instance, a number of natural supplements are not sufficiently well-studied for secondary sources to even exist and hence I think primary sources should be preferred over no sources in these cases. Fuse809 (talk) 13:43, 7 March 2014 (UTC)

sorry but i don't agree with what you write here; if there are no reliable sources as defined in MEDRS, Wikipedia should say nothing. Primary sources are not reliable for health content - that is the whole point of this. Jytdog (talk) 13:47, 7 March 2014 (UTC)

I would say "less likely to be reliable" rather than "not reliable". There are certainly many primary sources that are reliable. The converse implication, "secondary sources are reliable", does not hold up to scrutiny either. Actually my main concern about primary sources is misuse to support minority/fringe viewpoints, especially by POV/COI pushers. Axl ¤ [Talk] 16:11, 7 March 2014 (UTC)

Question about a journal

Hey all, I am looking to get some wider feedback on a journal. The journal is Acupuncture In Medicine. Briefly, AiM is owned by the British Medical Acupuncture Society and published by BMJ; it is MEDLINE-indexed, and peer-reviewed. Its stated aim is "to promote the scientific understanding of acupuncture and related treatments by publishing scientific investigations of their effectiveness and modes of action as well as articles on their use in health services and clinical practice....The Western understanding of neurophysiology and anatomy is used to interpret the effects of acupuncture." Arguments are being made on the German acupuncture trials Talk page (here and here) that the journal itself is an unreliable source under Wikipedia:FRINGE#Independent_sources and, most relevant to this Talk page, under Wikipedia:MEDRS#Use_independent_sources. The claim is that all articles related to acupuncture from this journal should be considered unreliable sources, because the journal is not independent of the field of acupuncture. All the other usual issues would apply (what is the specific content being supported, and is the specific article used as a source a review, etc). It is the wider question I am coming with, however, as it is a conversation stopper. I can see how reasonable arguments can be made on either side of the question, but my sense is that the claim is invalid. Input is welcome! Input grounded as carefully on policies and guidelines as possible (not wikilawyered, just grounded) is very welcome. Jytdog (talk) 13:25, 5 March 2014 (UTC)

The editorial board is clearly made up of only acupuncture believers, and they are promoting an editorial bent which is clearly promotional of acupuncture in a very seedy way. This list is particularly damning. They are preferentially recruiting acupuncturists and pseudoscience/altmed-supporters of various sorts. Notice how, for example, Edzard Ernst is prominent in his absence on that list as are Bed Goldacre, David Colquhoun, and Steven Novella. The journal is systematically excluding WP:FRIND. jps (talk) 13:44, 5 March 2014 (UTC)

I'm not so concerned about the absence of Ben Goldacre on the editorial board (he plays pretty fast and loose with the facts himself at times), but on first pass roughly 9 of the first 10 clinical trials I found published in this journal gave positive results. This is a dramatically higher rate than is found in the mainstream literature. I don't know how exactly we exclude this journal however given that it is Medline indexed. Formerly 98 (talk) 14:12, 5 March 2014 (UTC)

Curiosity: with which facts has Ben Goldacre played fast and loose? jps (talk) 18:22, 5 March 2014 (UTC)

A journal owned by acupuncturists whose editorial control is by acupuncturists cannot possibly be what WP:FRINGE and WP:MEDRS has in mind regarding independent sourcing for an objective look at acupuncture. I would suggest that articles from this journal are reliable for what acupuncturists think but are probably not reliable as an independent, objective source to see if it is effective. As always, there is an "out" here; if the journal article in question has been positively cited in the medical literature, this would go a long way to establishing it as reliable for MEDRS purposes, regardless of where it is published. If it has not garnered much support in the medical literature, I would avoid its use and use only those sources clearly independent of the topic. Without knowing the background, if the only journal articles stating that acupuncture work are coming from acupuncture owned and edited journals, that is a massive red flag.

I would think an analogous situation would be the journal Homeopathy (See WP:PARITY), which although MEDLINE indexed, is specifically disallowed by WP:FRINGE, for what I assume is the exact same reasons. Yobol (talk) 15:37, 5 March 2014 (UTC)

You can't condemn the entire journal out of hand merely because it has an editorial position. This is on par with saying that (your choice: Fox News or The New York Times) should always be rejected as a source. Shall we reject all specialist journals, unless we can prove that the editorial board has regular fights? I bet you'll find that the editorial board of NEJM has only people holding MDs. Anyone can see that this is obviously "biased" against DOs and nurses, and promotes MDs. Do you think you'll find many neurologists on the editorial board of JAMA Psychiatry? I don't. How about endocrinologists, to make sure that thyroid disorders that mimic psychiatric conditions are given fair attention? I'm guessing they've got zero. Shall we toss them out, too, since we've just proven that their boards are "biased" and "non-independent"?

When we talk about the need for an WP:Independent source, we are talking about whether that the author or publisher is likely to directly benefit from it. Publishing information about how you are the best _____ in the world or that people with this condition should pay you for treatment is not independent. Publishing what you did in a clinical trial, with no reason to believe that you'll benefit from this, is sufficiently independent.

Or, to put it another way, whatever the game, whatever the rules, the rules are the same for both sides. If it's "independent" for a surgery journal with a bunch of surgeons on its board to let a surgeon publish what happened to his surgery patients, then it's "independent" for an acupuncture journal with a bunch of acupuncturists on its board to let an acupuncturist publish what happened to his acupuncture patients. SPOV was rejected. We don't apply different rules to subjects that editors decide are "scientific" or "not".

Additionally, nearly any source can be reliable for some statements, per the FAQ (which you'll find towards the top of WT:V). WhatamIdoing (talk) 01:39, 6 March 2014 (UTC)

The publisher of this journal is in the business of promoting acupuncture for essentially any and all ailments. Acupuncture, I would remind you, is a non-protected term. Anyone can call themselves an acupuncturist. What surgery journal are you referencing which excludes articles that show evidence that surgery for certain conditions may not be advisable? Most surgery journals allow for non-surgeons to review clinical studies, for example. Do you see the problem? jps (talk) 03:30, 6 March 2014 (UTC)

The publisher of this journal is in the business of promoting acupuncture for essentially any and all ailments. Citation needed? Original research?

The publisher actually says that their intent is "to promote the scientific understanding of acupuncture", which is importantly different from saying that they're "promoting acupuncture".

Anyone can call themselves an acupuncturist.

If we assume that this "anyone" doesn't mind spending a year in jail, then I guess you're right. (Guess what? That's the same jail term given to people who pretend to be physicians in California.) It may not be a protected term everywhere, but it certainly is protected in some places. WhatamIdoing (talk) 22:47, 7 March 2014 (UTC)

Also, I checked and many of the members of the editorial board of JAMA Psychiatry are not psychiatrists. One is, in fact, a neuroanatomist. Others are statisticians. In fact, the board looks exactly like what one would hope a board would look like for a reputable journal. It's a good study in contrasts. jps (talk) 05:56, 6 March 2014 (UTC)

thanks, WAID. this is pretty close to how i see it. this is such an interesting question to me. under your (our) reasoning, however, any group of fringe-y practitioners can set up a journal and even have critical reviews in it that are perfectly MEDRS compliant. right? would you please address that? am very interested in your thoughts on the hard part :) Jytdog (talk) 02:23, 6 March 2014 (UTC)

It's difficult. Wikipedia allows sources to be biased without impinging on their reliability. What we have to do is consider whether it's WP:DUE, considering all the sources that exist in the world. It's not good to over-represent the word of people in the field against people outside of the field—or the other way around. But you can't completely exclude either side or toss out all of the sources that are from one "side". WhatamIdoing (talk) 22:47, 7 March 2014 (UTC)

@WAID: Not sure if you're replying to me or not, but I did say specifically say what it would be reliable for, in my opinion. Regardless, let's take this a step back and use a more apt analogy. Let's say a group of orthopedic surgeons say they came up with a "revolutionary" new toe surgery that cures low back pain. The technique is so lucrative that they teach a select group of other surgeons who promote this across the country. There is no medical/physiological reason for such a surgery to work, and studies conducted by outside groups find negative results. This group then sets up a journal specifically dedicated to this surgical technique, the editorial board is comprised of surgeons who perform this technique and surprisingly, the results in this journal are markedly more positive than the studies done by, and published in, journals not affiliated with this technique. I would suggest to you that this journal has a clear conflict of interest with regards to the technique, and is not an independent source to objectively discuss the effectiveness of this technique, though it is reliable for what proponents of the technique believe. Is it your suggestion that we should take such a journal at face value? When you have a fringe group, publishing a fringe journal specifically about the fringe topic they are advocating, they are probably not independent of the topic, no matter what degree they hold. If Wikipedia's rules say that we need to treat this particular journal as equally valid as other journals, then there is something systemically wrong with Wikipedia's rules. Yobol (talk) 02:49, 6 March 2014 (UTC)

User:Yobol, I love what you wrote there - that faces that prong of the dilemma headon. OK now I am going to put the hard part of the question to you. It seems entirely plausible (to me at least) that there are scientists who (for whatever reason) want to test acupuncture interventions to see if they work and who do so rigorously and publish good science, and others who might write rigorous reviews. Where do they publish that, but for journals that at least aspire (as per the mission statement) to be solidly scientific, like the one I posted about? There are hardcore skeptics among us who will cut the legs out from an under an effort to use such a source - calling the journal FRINGE stops the conversation; and Wikipedia loses what may be good sources that way (and good sources seem to be hard to find in acupuncture). This is why I am on the fence and brought the question. Maybe this is a case where the fifth pillar comes in, but I am very uncomfortable doing that in alt med articles that are fringey and where we need to reason closely based on PAG. do you see what i mean? thx Jytdog (talk) 03:04, 6 March 2014 (UTC)

Let me be clear, I do not want suggest that there is any intentional wrong-doing on any part of any of these journals. I suspect that the vast, vast majority of these journals, editors, authors have nothing but the best of intentions to do good science; however we all know how easy it is to have unintentional biases, and how many different ways biases can creep into any work, despite the best of intentions. I also do understand your point; we all want to follow WP rules, be as fair to all sources as possible. However, I think WP rules already give us guidance here; we use independent sources (both in fringe and in medical topics), and I would not consider these are not independent sources. There is no need to invoke WP:IAR here. There are enough studies about acupuncture in non-acupuncture journals that we do not need to use the dubious ones. Yobol (talk) 03:25, 6 March 2014 (UTC)

Yobol nails it. The problem is that they just exclude independent review. That's inimical to our rules about independent sources being necessary. jps (talk) 03:27, 6 March 2014 (UTC)

Our rules about independent sources do not demand that the sources not be in the field. We accept specialized string-theory physics journals for writing about string theory; we accept specialized vaccination journals for writing about vaccination; we accept specialized acupuncture journals for writing about acupuncture. WhatamIdoing (talk) 22:47, 7 March 2014 (UTC)

I'm not sure why we're having this discussion. WP's rules are quite clear: whether a journal can be trusted as reliable mainly depends on whether it is peer reviewed; cf. WP:FRINGE#Peer-reviewed sources:

One important barometer for determining the notability and level of acceptance of fringe ideas related to science, history or other academic pursuits is the presence or absence of peer-reviewed research on the subject. While a lack of peer-reviewed sources does not automatically mean that the subject should be excluded from Wikipedia, there must be adequate reliable sources to allow the subject to be covered in sufficient detail without engaging in original research. Care should be taken with journals that exist mainly to promote a particular viewpoint. Journals that are not peer reviewed by the wider academic community should not be considered reliable, except to show the views of the groups represented by those journals.[5] Peer review is an important feature of reliable sources that discuss scientific, historical or other academic ideas, but it is not the same as acceptance by the scientific community. It is important that original hypotheses that have gone through peer review do not get presented in Wikipedia as representing scientific consensus or fact. Articles about fringe theories sourced solely from a single primary source (even when it is peer reviewed) may be excluded from Wikipedia on notability grounds.

. --Mallexikon (talk) 03:58, 6 March 2014 (UTC)

Peer review is a necessary, but not sufficient criteria for most MEDRS. In this case, you cannot take one part of FRINGE or MEDRS out of context to the rest (i.e. the need for independent sources). Peer review is not a magic pixie dust you can sprinkle on a source and say "reliable!". Not all peer-review is the same, and not all journals/editorial boards are the same. Yobol (talk) 04:02, 6 March 2014 (UTC)

You're right that peer review is not a sufficient criteria for MEDRS (sources also would have to be secondary at up-to-date), however, by WP's rules, "Peer reviewed medical journals are a natural choice as a source for up-to-date medical information in Wikipedia articles" (WP:MEDRS#Biomedical journals). That means that WP recognizes peer review as the litmus test concerning the question whether a journal is independent/reliable or not. Of course there are some fringe journals that fake peer review (cf. WP:PARITY: "Note that fringe journals exist, some of which claim peer review. Only a very few of these actually have any meaningful peer review outside of promoters of the fringe theories, and should generally be considered unreliable. Examples of unreliable journals include, but are not limited to: The Creation Research Society Quarterly, Homeopathy, and the Journal of Frontier Science (which uses blog comments[6] as its supposed peer review)"). But I don't see any evidence that Acupuncture in Medicine constitutes this kind of fraud. Let's keep in mind that it is MEDLINE-indexed and published by BMJ. Their reviewer list comprises more than 100 people, and the editorial board clearly does not mainly include acupuncturists. I think it is regrettable that they have 2 people from the "Royal London Homeopathic Hospital" (regrettable that this kind of hospital exists at all) on their editorial board, and I would agree with jps that it is regrettable that Ernst is not on their reviewer list. But that is not evidence enough to label their peer-review a fraud and throw this source out. --Mallexikon (talk) 05:29, 6 March 2014 (UTC)

I have gone through the entire editorial board and reviewer list and see all alt-med professionals or acupuncturists. Can you point to one who isn't? jps (talk) 06:08, 6 March 2014 (UTC)

The reviewer list comprises more than 100 people... You checked them all? --Mallexikon (talk) 07:35, 6 March 2014 (UTC)

It's not that hard to copypaste them into a google search. Try it out. You'll get the hang. I could be mistaken, I suppose, since I didn't run it very quickly, so if you find one who isn't do let me know. jps (talk) 11:32, 6 March 2014 (UTC)

If I recall correctly, the last time we discussed acupuncture (on this page) you called Andrew Vickers and all the others authoring this meta-analysis acupuncturists, so it's difficult to take you, ScienceApologist, at your word. It's really, really sad that I have to say that I question your ability to present information faithfully; you're sharp enough that this means I question your basic integrity. On the overall point, I lean towards allowing Wikipedia to present both sides (especially in, say, the acunpuncture article) but basically agree with Yobol when it comes to the overall evaluation: "There are enough studies about acupuncture in non-acupuncture journals that we do not need to use the dubious ones". If someone feels strongly that there is a niche gap that a specialist reference fills, perhaps a a nuanced perspective of a hypothesized theory, a specialist journal might be appropriate particularly if it can be balanced appropriately. Yes, in my view even in homeopathy the journal Homeopathy likely should be referenced to allow the article to present the proponents' strongest case as such. In my opinion, allowing the fringe view a little breathing room is a fundamental and incontrovertible requirement for NPOV, as much distaste as it leaves for those with hardcore passion. II | (t - c) 07:53, 6 March 2014 (UTC)

Homeopathy is specifically called out as unreliable at WP:FRINGE. You're asking us to ignore our own guideline. jps (talk) 11:32, 6 March 2014 (UTC)

No, we're asking you to pay careful attention to the policy that says "The appropriateness of any source depends on the context". There are statements for which even a garbage journal could be perfectly reliable, or even authoritative, source. WhatamIdoing (talk) 23:02, 7 March 2014 (UTC)

Thanks for everybody's comments - some are more careful and reasoned and generalizable than others. Things are getting wander-y, so I want to pull up. My goal here is to try to understand the basis, grounded in policy and guidelines, for excluding a journal that has many of the trappings of a potential publisher of legit MEDRS-compliant sources, except its subject matter. WAID's post above says "you cannot exclude it." Which is very useful for preventing editors from arbitrarily excluding a journal because they don't like its subject matter, but doesn't address the hard part of the question - that there is FRINGE out there. Yobol's answer calls on Wikipedia editors to examine the reviewers and editorial board of a journal to see if those lists include a wide range of expertise. This has good support in a guideline, as User:Mallexikon quoted above from WP:FRINGE, namely "Journals that are not peer reviewed by the wider academic community..." This seems well generalizable to me, and would be as useful for screening out a journal by a "cabal" of orthopedic surgeons promoting some funky procedure, as for excluding a journal "by acupuncturists, for acupuncturists." Applying this will lead to questions/debates of what "wider" means, of course, and indeed that discussion has broken out above. But interpretation is always debatable. OK, so re-posing the question: is that it then? Or are there relevant bits of policies or guidelines, other than reviewing editorial board/peer-reviewers, that can help rationally exclude or include a journal on a fringey topic as a publisher of MEDRS sources? Thanks again! Jytdog (talk) 13:26, 6 March 2014 (UTC)

Two more things to consider: publication record and citations. If journal articles are not cited outside of the journal itself or outside of the insular community, that's a redflag. Additionally, if the journal has a questionable publication record (for example, only publishing positive studies or only publishing negative studies on competing ideas) that should be considered as well. jps (talk) 01:16, 7 March 2014 (UTC)

are those criteria somewhere in policy or guideline? Jytdog (talk) 01:56, 7 March 2014 (UTC)

I think some editorial judgement is required in the use of sources such as the AIM journal. For references within acupuncture articles and sections, I'd say it is probably an acceptable source ("acupuncture is thought by its proponents to be beneficial for condition X") but you wouldn't want it as a source for general content in the way we would cite Cochrane or non-CAM journals. It is not dissimilar to citing the Bible - it is a good source if what you're saying is that the Bible says one thing or another, but you wouldn't use it as the definitive source on a current-day issue. I'm sure there will be discussions about the scope of AIM as a source, but I wouldn't rule it out on principle provided there is consensus for each individual use. JFW | T@lk 14:30, 7 March 2014 (UTC)

You'll want to check for the usual features of reliable journals:

• Does it have a reputation for fact-checking and accuracy: Do they issue corrections? Do they withdraw bad papers?
• Is it published by a reputable publishing house, rather than by the author(s)? Association with a reputable group is good. Do other sources name it as an example of a bad or captive journal? Do other sources cite it (as much as might be reasonable—niche subjects don't get cited much in general journals)?
• Is it a third-party or independent source, with no significant financial or other conflict of interest? Do they publish only papers written by their own reviewers (or employees)? Will the authors or journal's owners make money off of this (more than the typical academic or medical paper)? Is this part of somebody's get-rich-quick scheme?
• Does it have a professional structure in place for deciding whether to publish something, such as editorial oversight or peer review processes? Do their "instructions to authors" indicate a proper, preferably anonymous, peer review process? WhatamIdoing (talk) 23:02, 7 March 2014 (UTC)

Thank you, particularly to WAID, Yobol, and JFW for taking time to write careful responses. Not a simple question and I appreciate the time and thought. I don't want to drag the battles on the GERAC page out here; you have given us plenty to work with over there. Thanks again.Jytdog (talk) 00:36, 8 March 2014 (UTC)

General comment: I'm a bit late to this party due to a short break, but will post here what I posted at Talk:GERAC: The "independent sources" part of WP:FRINGE has been construed more narrowly than in academia. See Independent sources & refs therein. Dependence isn't a function of a journal's topic in the real world; it actually has to do with external relationships (e.g., a reviewer being the parent of an author). No important conflicts -- according to prevailing standards in academia -- are declared by its editorial board (most/all of whom practice acu, not surprising given the subject matter, and have additional qualifications as well). The journal appears to fulfill WP's other tests for sources, such as peer review and indexing; furthermore, studies published in it have been used by the Cochrane Collaboration and in other accepted MEDRS's, and MD's and PhD's publish in it (e.g. Edzard Ernst). "Dependence" seems to have been misinterpreted by editors unfamiliar with what it actually means. It doesn't have to do with a journal's topic area.

That said, User:Yobol makes a good analogy, but given that Cochrane evaluates papers from this journal on their merits -- and AFAIK they do publish findings of no efficacy, since that's what clinical trials show for most conditions -- it seems to be that we should as well, rather than dismiss it as fringe, or necesarily and inherently biased, outright. IOW, Yobol's hypothetical example is fair, but this real-world case isn't considered conflicted or inherently biased in academia, including among those who write the best of the best MEDRS's (Cochrane). Middle 8 (leave me alone • talk to me • COI?) 17:45, 15 March 2014 (UTC)

Whoa, just caught the thing above on peer review. Their review board doesn't include people outside of alt-med research/practice? If so, obviously it's not reliable. (This is true in any field but especially alt-med; alt-meds shouldn't stay alt, but rather undergo scrutiny and then join real-med or go away. You can't do that by being selective about who reviews the research. I'm disappointed in AIM if they lack the balls & integrity to do it right.) --Middle 8 (leave me alone • talk to me • COI?) 07:47, 17 March 2014 (UTC)

1. There is no policy basis for that statement. We don't get to completely reject FOX News' political coverage even if their editorial board is all Republicans. We don't get to completely reject an academic journal even if all of their reviewers are in the same general branch of medicine. See WP:BIASED: reliable sources are allowed to be biased.
2. You seem to have been misled by the description above. There's a dubious claim that the board is "made up of only acupuncture believers", but there's no real basis for that claim, and certainly no basis for assuming that there are no people outside of the alt med field. This biostatistician, for example, doesn't appear to be an alt med practitioner of any kind, and if you look through his publications, the most prominent subject is surgery for prostate cancer—more than 100 papers on surgery. Now perhaps that's the only one; I just picked a name from an institution I recognized and I haven't checked all the others. But it's clear that the editorial review board is not merely altmed practitioners. WhatamIdoing (talk) 20:35, 28 March 2014 (UTC)

Journal of the American Psychiatric Nurses Association

I am wondering why the Journal of the American Psychiatric Nurses Association would not be an acceptable source when used to discuss psychiatric care of cancer patients. Thoughts? Gandydancer (talk) 15:40, 28 March 2014 (UTC)

I am wondering why you think that treatment of psychiatric conditions = treatment of cancer. Thoughts? Yobol (talk) 19:35, 28 March 2014 (UTC)

Cancer patients and cancer survivors commonly have psychiatric conditions as a result to diagnosis and treatment. It might range from PTSD to physical brain damage. Mixed anxiety and depression is common. It would not be "a treatment of cancer", but it would certainly be "a treatment that is necessary because of cancer". WhatamIdoing (talk) 20:16, 28 March 2014 (UTC)

Of course, numerous chronic medical conditions are associated with depression and other psychiatric conditions, not just cancer. Treatment of the associated psychiatric conditions is still != treatment of that condition. This seems to be going astray of the intended use of this talk page, though. Should've known better than to respond to a loaded question. Yobol (talk) 19:12, 29 March 2014 (UTC)

Thanks Waid. You (and Doc James) are one of the few editors around these parts that do not treat "outsiders" with contempt and I appreciate it. I believe that this language: Other indications that a journal article may not be reliable are its publication in a journal that is not indexed in the bibliographic database MEDLINE,[16] or its content being outside the journal's normal scope (for instance, an article on the treatment of cancer in a psychiatric nursing journal) needs to be changed. Surly you guys can find a better example, and I will bite my tongue rather than say more. Gandydancer (talk) 21:35, 29 March 2014 (UTC)

The stifling impact of secondary sources on the flow of information.

I have to admit that there is particular behavior that is annoying me. I understand MEDRS considers secondary sources as a golden standard. However, most members of MEDRS use the "ideal" standard as an excuse to remove primary studies at their own discretion, especially if they are the primary editor of an article. In most instances there is little attempt at qualifying the information that is currently available. In fact, I've noticed several instances of individuals using outdated literature/systemic reviews as a way of reaching their wanted conclusion. To name names, Ashburn used a literature review on lipoic acid's impact on dementia to state that there is not good evidence of it's efficiency. However, when one takes time to read the cited article it simply stated that there were no randomized/controlled trails using lipoic to treat dementia and that it was not recommended currently. Ashburn then reverted multiple attempts at stating this fact simply saying that he was making it "plainer". This isn't me holding a vendetta against Ashburn specifically; I have noticed this behavior from many people within the MEDRS community.

This sort of behavior is bad enough, however, it is even worse when one realizes that conceptions such as statistical significance and clinically beneficence are arbitrarily defined social conventions. Reading many articles as they are currently written guarantees that readers will reach unfounded conclusions.

I have few ideas that could remedy this situation. 1. Extensively qualifying primary and animal model studies instead of reflexively deleting them. There is no need to ignore all evidence that is not ideal. This is especially true with newer and behavioral treatments. 2. Instead of using terms like "weak", "no good" or "good" evidence actually state the numerical benefit derived by a particular treatment compared to active or passive placebo. This is probably the best way to avoid systemic biases. 3. Recognize that the current used hierarchy of scientific journals is again a social construct with it's own particular problems. It's not reasonable to pick and choose which article to use based on the impact factor of the journal that published it nor altmertics. 4. Understand that there are other Wikipedians with differing editing styles. And that there are few rules that allow carte blanche authority within this faction.

TL;DR: Don't let the perfect be the enemy of the good.

I read and edit Wikipedia to discover, learn, share new and interesting information. I am also by nature an inclusionist and eventualist. I find it dismaying to be beholden to the arbitrary whims of councils and bureaucracy. I would much rather allow the free flow of information than win internet points and attain a "good" article status. I sincerely hope we can find a way to bridge this impasse. Khimaris (talk) 19:33, 31 March 2014 (UTC)

Hmm, you've been mass removing secondary sources from Lipoic acid‎ with misleading edit summaries[3] and and now repeatedly adding bigged-up language ("emerging evidence") and re-interpreted conclusions to omit their negativity. I don't know how that helps the flow of information; rather it helps the flow of misinformation. Alexbrn ^{talk|contribs|COI} 20:54, 31 March 2014 (UTC)

I'm perfectly fine with all results as long as they are verifiable and substantially qualified. Your edits in that particular article had a negative bias( NEGATIVITY IS A BIAS ) that needed to be addressed. In one source that was cited you prevented the addition of information that there were no known trials of Lipoic acid and tried to use that to say that there was "no good evidence". In another there was a cited review article that simply restated the existence of a trail of lipoic acid on metabolic disorders. Except it stated a negative review whilst the original article did not. This is what I mean about lot letting the perfect be the enemy of the good and not being bogged down by arbitrary definitions. It's simply lazy to use reviews in the manner that you have been using them. Some amount of detective work is required.

And now of course you've sent me a warning. As if any sort of addition I could make that deviates from what you've written would ever not be reversed by you or anyone else in your faction. This sort of behavior is the greatest continuing dick move being pulled by the Wiki powers that be. Khimaris (talk) 21:26, 31 March 2014 (UTC)

Khimaris What you are proposing is deeply antithetical to Wikipedia in at least two ways and displays an ignorance of biomedical science.

1. (Wikipedia reason 1) It requires a ton of original research which has no place at Wikipedia.
2. (biomedical science reason) Much of the primary biomedical literature is preliminary and turns out to be unreplicable - much of it is noise, not information.
3. (Wikipedia reason 2) Wikipedia has nothing to do with "the free flow of information." Our mission is to provide reliable and verifiable information to the public, which is why both WP:RS and WP:MEDRS emphasize using secondary sources (which also guards against the introduction of original research in the selection and weighting of primary sources). The importance of secondary sources is even greater in the biomedical field, as per #2; secondary sources are essential for determining what actually is information and is not noise.

You need to understand these things if you are going to have a happy and productive time at Wikipedia. If you don't agree with these things, you are going to have change WP:OR and WP:RS which are each policies; I don't know what you would do about #2 as that is just reality. Jytdog (talk) 21:37, 31 March 2014 (UTC)

No, Jytdog it is antithetical to YOUR personal view of Wikipedia and the existing culture of some within the MEDRS community.

(Wikipedia reason 1) It doesn't require original research to state the numeral benefit derived from a treatment nor to state the age of a study. Neither does avoiding arbitrarily derived terms. Am I the only person here who finds it unacceptable to use an old article stating no currently existing trials to state that a particular treatment is not effective whilst there are exiting (newer) trials on pubmed? That should give you at least an ounce of pause. You say that Wikipedia is for the dissemination of reliable (sourced and independent) information but in many instances it is very lacking.

(biomedical science reason) Your point? Wikipedia is not an academic product nor a place to seek personal medical information. Given time wrongs will be righted. That is the basic premise behind the scientific method. This is why we know that vaccines don't cause autism ffs.

(Wikipedia reason 2) Every edit that I have made that was eventually removed by a member of MEDRS met every gold standard. I'm calling bullshit on that one. It doesn't matter how "reliable" and "verifiable" the information posted is. If it is disagreeable to an extensive editor of an article it gets removed and other members of MEDRS fall in line with agreement.

Basically you've just told me to shut up, fall in line and nod my head to you and the other members. Somehow all the rules, regulations and culture put in place by the current and previous members of MEDRS are infallible and unquestionable. How dare some new comer question anything that is currently in place. And people wonder why the active membership of wikipedia is in decline.Khimaris (talk) 22:09, 31 March 2014 (UTC)