User:Anthonyhcole/Talking points/Background

History of peer review

"Most often, authors date the advent of the thing we’re talking about when we refer to editorial peer review today – the assessment of manuscripts by more than one qualified reader, usually not including the editor of a journal or press – to the 1752 Royal Society of London’s creation of a "Committee on Papers" to oversee the review and selection of texts for publication in its nearly century-old journal, Philosophical Transactions. A number of authors complicate this history by pointing to the existence of at least one earlier instance of formalized peer review in a scientific journal: the Royal Society of Edinburgh seems to have had such a system in place as early as 1731." --http://mcpress.media-commons.org/plannedobsolescence/one/the-future-of-peer-review/

"systems of peer review, internal and external to journals, were put in place by editors during the eighteenth century in order to assist editors in the selection of manuscripts for publication. It was appreciated from the start that the peer review process could not authenticate or endorse because the editors and reviewers could not be at the scene of any crime… the journals from the beginning threw the ultimate responsibility for the integrity of the article squarely upon the author" -- Drummond Rennie Editorial Peer Review 2

Peer review today

House of Commons Science and Technology Committee (HCSTC) - Peer Review in Scientific Publications - Eighth Report of Session 2010-2012 [1]

• "There are three main types of peer review in use. They are: 'single-blind review', 'double-blind review' and 'open review'. The Royal Society explained that:

  By far the commonest system in use is 'single blind' peer review in which the author's name and institution is known to the reviewer, but the reviewer's name is not provided to the author.

  A number of journals instead choose to operate a 'double blind' peer review system which is fully anonymised (i.e., the author(s) are unaware of the identity of the reviewer(s) and vice versa).

  Recently, there have been some experiments with a third type, 'open' peer review, in which the authors' and reviewers' names are revealed to each other. Open peer review can be reasonably described as an experimental system at this stage and is far from common.

"... [Single blind review] is the commonest system in scientific journals. In the social sciences, peer review 'is almost invariably a double blind process'. Some journals, such as the BMJ, choose to use open peer review." HCSTC p. 11

• Fiona Godlee: "Pre-publication peer review is only one aspect of the peer review process, which begins with grant-funding peer review, ethics committees, the pre-publication process, the editing process and then the peer review that goes on after publication. Then there is correction and, in some rare cases, retraction. All of these systems constitute peer review." HCSTC p. Ev 21

• Fiona Godlee: "The unaccounted cost is the peer reviewers' time. One of the questions is how we make that more of a professional activity for which they get academic credit rather than something that gets no credit." HCSTC p. Ev 22

• Dr Sugden (international managing editor of Science magazine at Cambridge): "Often you will get two or three referees' reports on a paper, but those referees may not agree with each other. It is the editor's job, if they consider the paper worth pursuing, to then make a recommendation as to which of those referees' revisions they should follow, and which they should not, and maybe do some extra ones too." HCSTC p. Ev 24

• Roger Williams (MP): "In this initial sorting out of submitted papers, what are the benefits and disadvantages of editorial boards against staff editors?"

  Dr Godlee: "Cost."

  Dr Sugden: "We don't pay our editorial boards. Most of our submissions will go to one or more members of the board in the first week they arrive. Then the staff editors will make their decision based partly on that advice."

  Roger Williams: "Do you all have editorial boards as well as staff editors?"

  Dr Campbell (editor-in-chief of Nature and the Nature Publishing Group): "Nature and the journals do not have editorial boards. ..." HCSTC p. Ev 26

Fiona Godlee 2000 "The Ethics of Peer Review" in Anne Hudson Jones and Faith McLellan (eds) Ethical Issues in Biomedical Publication [2]

"According to the International Committee of Medical Journal Editors (ICMJE), a peer-reviewed journal is 'one that has submitted most of its published articles for review by experts who are not part of the editorial staff.' Even this definition leaves scope for wide variation. Smaller journals with part-time editors tend to rely on their external advisers to decide the fate of a manuscript. If two reviewers disagree, a third may be asked to adjudicate. Editors of large journals, on the other hand, use reviewers for advice only and may have additional systems such as a weekly panel meeting of editors and external expert advisers, to help them decide on publication. [...]" Godlee 2000 p. 60

A few journals pay their reviewers; most do not. Whatever the type of system, the peer-review process involves the combined input of journal editors and external reviewers. In addition, it involves not only the decision to accept or reject an article for one particular journal but also decisions on how accepted articles should be revised before publication." Godlee 2000 p. 60

"The ICMJE (1997, 44) suggests four main causes of conflict of interest: financial relationships, personal relationships, academic competition and intellectual passion. [...] As the ICMJE states, "Public trust in the peer review process and the credibility of published articles depend in part on how well conflict of interest is handled.' Godlee 2000 pp. 68-70

"To heighten awareness and encourage full disclosure, many journals now ask reviewers to sign a statement, at the time they are asked to review each paper, either saying that they have no conflicts of interest or detailing any of which they are aware. Reviewers can then choose to disqualify themselves from the review of a particular paper or, having informed the editors, act on their advice. [...] Reviewers may choose not to declare conflicts or may be unaware, especially in the case of intellectual passion, that any such conflict exists. It is therefore up to the editor to select reviewers who will be as far as possible free from such influences." Godlee 2000 p. 70

"More open peer-review processes may encourage peer-reviewers to declare conflicts, in the knowledge that the authors and possibly the readers will be told who they are and may know of existing conflicts. Godlee 2000 p. 70

"Several journals, including the New England Journal of Medicine and the BMJ, have made it a rule that editors should not receive payments from or own stock in medically related companies." [...] 'The cost of peer review varies across journals. Larger journals have full-time editors who may spend more than half their time reading papers, selecting reviewers, assessing reviews, asking authors to revise papers, and assessing revisions, all of which constitutes peer review." Godlee 2000 p. 71

"Belshaw (1982) suggested that sending a paper to multiple reviewers (as many as 15) would reduce the effect of each reviewer's individual biases." Godlee 2000 p. 74

"Traditionally, peer reviewers' comments have remained anonymous. Journals have justified this practice on various grounds. The main arguments are that reviewers would be unwilling to sign their reviews, making it difficult for editors to recruit good reviewers; that there would be more animosity and resentment within science; that acceptance rates would increase; and that reviewers might tone down their comments, especially if the reviewer were junior and the author eminent and politically powerful. But anonymity is increasingly out of favor in biomedical publishing - editorials, for example, are now rarely anonymous - and society in general is less tolerant of what it sees as power without responsibility." Godlee 2000 p. 79

"Journals should have systems to reward good reviewers and to prevent some reviewers from receiving too many manuscripts. They should also encourage training in critical appraisal for editors and reviewers. This training, along with the provision of clear guidelines and checklists, provides a framework for assessing the quality of manuscripts and should reduce the arbitrary element in peer review." Godlee 2000 p. 79

Extent of use of Wikipedia's medical content

Wikipedia is the leading single source of healthcare information for patients and healthcare professionals.

"Wikipedia used to be a guilty secret for many of us. As doctors and academics we may have used it but wouldn’t tend to admit this in public. Things have changed. Most people I ask now freely admit to using Wikipedia, often going to it first as a starting point, even for clinical questions."

— Fiona Godlee (2014) BMJ

Various commentators in health education have noted that Wikipedia is popular among medical students. Students frequently cite search engines such as Google and Web 2.0 information sharing sites such as Wikipedia as the first places they look when seeking information for an assignment. A study of a group of veterinary students found that the majority sought and found medical information on Wikipedia. Wikipedia's health information has been described as "transforming how our next doctors learn medicine"

"Because of the errors it contains, the place for Wikipedia in medical education is non-existent," said one of my lecturers during a discussion on teaching. [...] In my experience, though, the website is widely used by students and healthcare professionals.

— Teun Teunis (2013) Student BMJ

Whether we’d like to admit it or not, Wikipedia has become a central pillar of medical — and honestly most — education. It’s used everywhere. In lecture, at home, during discussion groups, and even when we’re presenting patients. As physicians-in-training, we use it to look up everything from prescription medications to complicated words. And it provides instant answers in the simplest of forms, through a medium that just makes sense to our generation.

Last week, I informally surveyed my second-year medical school class; among 100 respondents, 89 indicated that Wikipedia was the first resource they consulted when faced with confusing concepts or terms. Seventy-nine said they used it “all the time” and, perhaps most striking, 45 said they thought they would use it for clinical information as physicians.

— Nathaniel P Morris (2013) Boston Globe

Wikipedia cited in health science journals

Wikipedia has been inappropriately cited as an authoritative source in many health science journals.

Improvement endeavours

• In 2010, Magnus Manske produced a widget for adding a tab to the top of articles which, when clicked, displayed a link to the relevant web page of the organisation responsible for an expert review of the article, a permanent link to the reviewed version of the article, a link to the diff between the reviewed and current versions, and the name of the reviewer/s.
• In 2011, Erik Moeller created a meta page, m:Expert review.
• Wikipedia editor Daniel Meitchen has championed a project at PLoS Computational Biology called "Topic Pages", where a mature Wikipedia article on computational biology may be peer reviewed (by named reviewers) and the version that passes review is published in the online journal. Seven articles have been through the process: One in 2012, five in 2013 and one in 2014.

• Topic Pages: PLoS Computational Biology Meets Wikipedia
• Table of Contents: PLOS Computational Biology: Topic Pages

• A team of editors worked Dengue fever up to a high standard and then submitted it to the now defunct Canadian journal Open Medicine for review. The article was anonymously reviewed on the article's talk page, and published in 2014.[3].
• In 2014, Wikipedia editor John Byrne led the CRUK project, providing rigorous anonymous review for several of our cancer articles by experts associated with (or selected by) CRUK
• A team of biomedical editors has started Wikiversity Journal of Medicine. Their peer review process can be found here.

My thoughts on a useful expert review process

Because Wikipedia's model allows for anonymous authorship, any Wikipedia expert review process needs to involve named reviewers. The peer review process relies on accountability. In the usual model, authors are named and any falsehoods, omissions or biases found in their published work will impact their reputation. In a review model that allows for anonymous authorship that accountability shifts to the reviewer/s.

Reviewers will need to be adequately compensated for shouldering the accountability burden. Compensation may be of two kinds: (i) payment or (ii) academic (and popular) credit. Probably both payment and credit will play a role; but we may be able to persuade the best minds to review our work for a lesser payment (or maybe even for free) if we can position that role in the eyes of the public and the academy as one of high prestige - that is, if an invitation to review a Wikipedia article is a credible and very public acknowledgement that one has arrived at the peak of one's field. We may achieve the latter if we

1. begin at the top, only inviting the very cream of a field to review a topic in that field
2. launch it with a huge song and dance - beginning in the academic press (Nature, Science, BMJ, etc.) and following up immediately with a campaign emanating from the WMF, including speeches to targetted audiences and television appearances from Jimmy and/or Lila, press releases, and interviews - again, with Lila and/or Jimmy - repeatedly emphasising the point that we'll only be inviting the best in the relevant fields to review our work. The people we're appealing to will be at the top of their profession, and the invitation to help us needs to come from the top of this organisation. Google and other partners/players should be briefed ahead of time so they are ready with (hopefully positive) independent commentary.

Wikipedia scholar

We should create a new project to host locked (uneditable) versions of CC-BY-SA encyclopedia articles and other media that have been rigorously peer reviewed. The pages should be elegant and lean (none of the clutter associated with wiki pages), with little demand on bandwith and super-fast to load. No fruit; just text, tables, thumbs and simple links.

These expert-reviewed articles should prominently link to

1. the current editable version of the article and
2. an elegant, simple, readable diff between the expert-reviewed version and the current editable version, so readers can easily see how the article/topic has evolved since the last review. This will require some work on WMF's part: producing a diff without all the wiki markup. To see the readability problem with our current diffs, check out this diff between the peer reviewed version and current version of our dengue fever article.

Funding

Due to perceived conflict of interest, the best source for funding would be independent of the WMF. We should start by approaching Wellcome Trust, because of their existing close relationship with WMUK and their previous support for several initiatives aimed at improving our medical offering.

Wellcome trust has an £18 billion endowment and its 2014 grant funding and direct charitable expenditure was £674 million.[4]

The objects of the trust include supporting "the study and understanding of any of the biosciences."[5]