Peer Review Highly Sensitive To Poor Refereeing, Claim Researchers

Just a small number of bad referees can significantly undermine the ability of the peer-review system to select the best scientific papers. That is according to a pair of complex systems researchers in Austria who have modelled an academic publishing system and showed that human foibles can have a dramatic effect on the quality of published science.

[snip]

While the concept of peer review is widely considered the most appropriate system for regulating scientific publications, it is not without its critics. Some feel that the system's reliance on impartiality and the lack of remuneration for referees mean that in practice the process is not as open as it should be. This may be particularly apparent when referees are asked to review more controversial ideas that could damage their own standing within the community if they give their approval.

Questioning referee competence

Stefan Thurner and Rudolf Hanel at the Medical University of Vienna set out to make an assessment of how the peer-review system might respond to incompetent refereeing. [snip]

The researchers created a model of a generic specialist field where referees, selected at random, can fall into one of five categories. There are the "correct" who accept the good papers and reject the bad. There are the "altruists" and the "misanthropists", who accept or reject all papers respectively. Then there are the "rational", who reject papers that might draw attention away from their own work. And finally, there are the "random" who are not qualified to judge the quality of a paper because of incompetence or lack of time.

[snip]

Within this model community, the quality of scientists is assumed to follow a Gaussian distribution where each scientist produces one new paper every two time-units, the quality reflecting an author's ability. At every step in the model, each new paper is passed to two referees chosen at random from the community, with self-review excluded, with a reviewer being allowed to either accept or reject the paper. The paper is published if both reviewers approve the paper, and rejected if they both do not like it. If the reviewers are divided, the paper gets accepted with a probability of 0.5.

Big impact on quality

After running the model with 1000 scientists over 500 time-steps, Thurner and Hanel find that even a small presence of rational or random referees can significantly reduce the quality of published papers. When just 10% of referees do not behave "correctly" the quality of accepted papers drops by one standard deviation. If the fractions of rational, random and correct referees are about 1/3 each, the quality selection aspect of peer review practically vanished altogether.

"Our message is clear: if it can not be guaranteed that the fraction of rational and random referees is confined to a very small number, the peer-review system will not perform much better than by accepting papers by throwing (an unbiased!) coin," explain the researchers.

[snip]

Don't forget the editors

But Tim Smith, senior publisher for New Journal of Physics at IOP Publishing, which also publishes physics world.com, feels that the study overlooks the role of journal editors. "[snip]. In relation to this study however, one shouldn't ignore the role played by journal editors and Boards in accounting for potential conflicts of interest, and preserving the integrity of the referee selection and decision-making processes," he says.

Michèle Lamont a sociologist at Harvard University who analyses peer review in her 2009 book, How Professors Think: Inside the Curious World of Academic Judgment, feels that we expect too much from peer review. [snip]

When asked by physicsworld.com to offer an alternative to the current peer-review system, Thurner argues that science would benefit from the creation of a "market for scientific work". He envisages a situation where journal editors and their "scouts" search preprint servers for the most innovative papers before approaching authors with an offer of publication. [snip]

[snip]

Related

"Peer Review And Journal Models" / Paolo Dall'Aglio

[http://arxiv.org/abs/physics/0608307]

About the author

James Dacey is a reporter for physicsworld.com

[http://physicsworld.com/cws/article/news/43691]

!!! Thanks To Antonella De Robbio For The HeadsUp !!!

The Idea of Order: Transforming Research Collections for 21st Century Scholarship

The Idea of Order explores the transition from an analog to a digital environment for knowledge access, preservation, and reconstitution, and the implications of this transition for managing research collections. The volume comprises three reports. The first, "Can a New Research Library be All-Digital?" by Lisa Spiro and Geneva Henry, explores the degree to which a new research library can eschew print. The second, "On the Cost of Keeping a Book," by Paul Courant and Matthew "Buzzy" Nielsen, argues that from the perspective of long-term storage, digital surrogates offer a considerable cost savings over print-based libraries. The final report, "Ghostlier Demarcations," examines how well large text databases being created by Google Books and other mass-digitization efforts meet the needs of scholars, and the larger implications of these projects for research, teaching, and publishing.

The reports are introduced by Charles Henry; the volume includes a conclusion by Roger Schonfeld and an epilogue by Charles Henry.

June 2010 / 123 pp. / $25 / ISBN 978-1-932326-35-2 / CLIR Reports 147

Source And Full Text Available At

[http://www.clir.org/pubs/abstract/pub147abst.html]

NISO, IU Receive Mellon Grant to Advance Tools for Quantifying Scholarly Impact From Large-scale Usage Data

BLOOMINGTON, Ind., and BALTIMORE, Md. -- A $349,000 grant from the Andrew W. Mellon Foundation to Indiana University Bloomington will fund research to develop a sustainable initiative to create metrics for assessing scholarly impact from large-scale usage data.
IU Bloomington School of Informatics and Computing associate professor Johan Bollen and the National Information Standards Organization (NISO) will share the Mellon Foundation grant designed to build upon the MEtrics from Scholarly Usage of Resources (MESUR) project that Bollen began in 2006 with earlier support from the foundation. Bollen is also a member of the IU School of Informatics and Computing's Center for Complex Networks and Systems Research (CNetS) and the IU Cognitive Science Program faculty.

The new funding for "Developing a Generalized and Sustainable Framework for a Public, Open, Scholarly Assessment Service Based on Aggregated Large-scale Usage Data," will support the evolution of the MESUR project to a community-supported, sustainable scholarly assessment framework. MESUR has already created a database of more than 1 billion usage events with related bibliographic, citation and usage data for scholarly content.

[snip]

The project will focus on four areas in developing the sustainability model -- financial sustainability, legal frameworks for protecting data privacy, technical infrastructure and data exchange, and scholarly impact -- and then integrate the four areas to provide the MESUR project with a framework upon which to build a sustainable structure for deriving valid metrics for assessing scholarly impact based on usage data. Simultaneously, MESUR's ongoing operations will be continued with the grant funding and expanded to ingest additional data and update its present set of scholarly impact indicators.

[snip]

Data from more than 110,000 journals, newspapers and magazines, along with publisher-provided usage reports covering more than 2,000 institutions, is being ingested and normalized in MESUR's databases, resulting in large-scale, longitudinal maps of the scholarly community and a survey of more than 40 different metrics of scholarly impact.

Sources
 
[http://www.librarytechnology.org/ltg-displaytext.pl?RC=15037]
 
[http://newsinfo.iu.edu/news/page/normal/15040.html]

NYTimes > Scholars Test Web Alternative to Peer Review

[snip] Now some humanities scholars have begun to challenge the monopoly that peer review has on admission to career-making journals and, as a consequence, to the charmed circle of tenured academe. They argue that in an era of digital media there is a better way to assess the quality of work. Instead of relying on a few experts selected by leading publications, they advocate using the Internet to expose scholarly thinking to the swift collective judgment of a much broader interested audience.

[snip]

That transformation was behind the recent decision by the prestigious 60-year-old Shakespeare Quarterly to embark on an uncharacteristic experiment in the forthcoming fall issue — one that will make it, Ms. Rowe says, the first traditional humanities journal to open its reviewing to the World Wide Web.

Mixing traditional and new methods, the journal posted online four essays not yet accepted for publication, and a core group of experts — what Ms. Rowe called “our crowd sourcing” — were invited to post their signed comments on the Web site MediaCommons, a scholarly digital network. Others could add their thoughts as well, after registering with their own names. In the end 41 people made more than 350 comments, many of which elicited responses from the authors. The revised essays were then reviewed by the quarterly’s editors, who made the final decision to include them in the printed journal, due out Sept. 17.

The Shakespeare Quarterly trial, along with a handful of other trailblazing digital experiments, goes to the very nature of the scholarly enterprise. [snip]

[snip].

Each type of review has benefits and drawbacks.

[snip]

Clubby exclusiveness, sloppy editing and fraud have all marred peer review on occasion. Anonymity can help prevent personal bias, but it can also make reviewers less accountable; exclusiveness can help ensure quality control but can also narrow the range of feedback and participants. Open review more closely resembles Wikipedia behind the scenes, where anyone with an interest can post a comment. This open-door policy has made Wikipedia, on balance, a crucial reference resource.

Ms. Rowe said the goal is not necessarily to replace peer review but to use other, more open methods as well.

[snip]

The most daunting obstacle to opening up the process is that peer-review publishing is the path to a job and tenure, and no would-be professor wants to be the academic canary in the coal mine.

The first question that Alan Galey, a junior faculty member at the University of Toronto, asked when deciding to participate in The Shakespeare Quarterly’s experiment was whether his essay would ultimately count toward tenure. “I went straight to the dean with it,” Mr. Galey said. (It would.)

Although initially cautious, Mr. Galey said he is now “entirely won over by the open peer review model.” The comments were more extensive and more insightful, he said, than he otherwise would have received on his essay, which discusses Shakespeare in the context of information theory.

Advocates of more open reviewing, like Mr. Cohen at George Mason argue that other important scholarly values besides quality control — for example, generating discussion, improving works in progress and sharing information rapidly — are given short shrift under the current system.

“There is an ethical imperative to share information,” said Mr. Cohen, who regularly posts his work online, where he said thousands read it. Engaging people in different disciplines and from outside academia has made his scholarship better, he said.

To Mr. Cohen, the most pressing intellectual issue in the next decade is this tension between the insular, specialized world of expert scholarship and the open and free-wheeling exchange of information on the Web. “And academia,” he said, “is caught in the middle.”

A version of this article appeared in print on August 24, 2010, on page A1 of the New York edition.

[http://www.nytimes.com/2010/08/24/arts/24peer.html/]

Letters To The Editor > 08-30-10

[http://www.nytimes.com/2010/08/30/opinion/l30review.html]

See Also CHE > Leading Humanities Journal Debuts 'Open' Peer Review, and Likes It

[http://chronicle.com/article/Leading-Humanities-Journal/123696/]

BTW: I Profiled Several Innovative Alternative Initiatives Several Years Ago > For Example

"Alternative Peer Review:

Quality Management for 21st Century Scholarship"

[http://www.public.iastate.edu/~gerrymck/APR-1.ppt]

> >> It's A Large PPT (200+ Slides) >>>

>>> But IMHO ... Well Worth The Experience [:-)] <<<

See Also (My) Other Works Listed / Linked At

The Scientist > August 2010 > Peer Review Rejected

[http://scholarship20.blogspot.com/2010/08/scientist-august-2010-peer-review.html]

NPR > An Un-'Common' Take On Copyright Law

It's safe to say that most Americans don't spend much time thinking about intellectual property law. But in Common As Air, Lewis Hyde explains why these laws profoundly affect our culture -- and how they are based on assumptions that are artificial, illogical and outdated.

[snip]

.... [I]ntellectual property laws affect our culture profoundly, in ways that go beyond college students being taken to court for downloading songs. Some people believe that not only are current copyright laws too stringent, but that the assumptions the current laws are based on are artificial, illogical and outdated.

Among them is Lewis Hyde, a professor of art and politics who has studied these issues for years. In his new book Common As Air, Hyde says he's suspicious of the concept of "intellectual property" to begin with, calling it "historically strange." Hyde backs it up with an impressive amount of research; he spends a significant amount of time reflecting on the Founding Fathers, who came up with America's initial copyright laws.

Hyde is a contrarian, but he's not a scorched-earth opponent of all copyright laws. He does believe the national paradigm for intellectual property issues should be changed, though, at one point offering several examples of the absurd situations the current laws have created. [snip]Hyde advocates for a return to a "cultural commons" and quotes, approvingly, Thomas Jefferson, who believed that "ideas should freely spread from one to another over the globe, for the moral and mutual instruction of man."

[more]

Source / Excerpt Available At

[http://www.npr.org/templates/story/story.php?storyId=129299939]

 

>>> Slide Show >>>

[http://www.salon.com/books/feature/2010/08/23/common_as_air_lewis_hyde/slideshow.html]

Teacher Magazine / AP > Net-Age Students Have Different View of Plagiarism

Published: August 16, 2010 > EASTON, Pa. (AP) — Last year, students in an Easton Area High School entrepreneurship class were assigned to write business plans. While reviewing them, teachers quickly realized one student had copied entire portions of his from a plan posted on the Internet.

[snip]

"There is a blurred vision by many in this digital age because there's just so much information they have access to," Koch said. "It's very difficult for them to filter what is mine and what is yours. It's all out there for you to utilize."

Although local educators said they haven't seen any rise in plagiarism cases lately, many found students from a generation raised on the Internet have a different perspective on what constitutes plagiarism.

[snip]

"Some students seem to think that whatever is out there is free to take," said Ed Lotto, director of first-year writing at Lehigh University. "They seem to think it's common knowledge and they can cite common knowledge without citation."

Plagiarism is a cultural, nationwide phenomenon. The problem is not limited to the Lehigh Valley.

[snip]

Lotto said the problem may manifest, in part, from the way the Internet has changed young people's perspective about digital media.

[snip]

"It's the culture they grew up in," Lotto said. "And that would affect their perception of ownership of things in the world."

[snip]

Part of the problem, he said, is students learn at a young age to use the Internet as a research tool, but are not necessarily instructed early on what requires a citation and how to cite it.

"I think it's a problem in the curriculum for younger grades," Ziegler said. "There has to be a way to better transition those kids from middle school reports to high school reports."

Other local educators don't necessarily see a connection between students raised in the digital age and plagiarism.

Hannah Stewart-Gambino, dean of Lafayette College, said students are instructed as thoroughly as ever about what constitutes plagiarism and most know the difference between what is right and wrong.

"Even as humans were scratching rudimentary writing into tree bark, the inclination to claim work that is not one's own has been human nature," Stewart-Gambino said. "I'm not convinced technology or the Internet has altered human nature or that temptation substantially, frankly."

Editorial Comment > Are You Serious ???

But she does believe students have difficulty differentiating between legitimate sources on the Internet and nonacademic sources.

[snip]

Just as music companies and film studios have had to change their online sales models in response to illegal downloading, Lotto said some educators have suggested schools will have to change their views of what constitutes plagiarism due to the rise of the Internet.

"I've heard it argued a change in perspective of plagiarism is coming, that we shouldn't be too hard-nosed about it," he said. "I'm an older guy, so I don't really buy it so much, but I do hear it sometimes from younger teachers and graduate students."

Lotto added, "It's going to change. I don't know how or in what way, but you can see it."

Information from: The Express-Times [http://www.lehighvalleylive.com]

Copyright 2010 Associated Press. All rights reserved. This material may not be published, broadcast, rewritten, or redistributed.

Source  > Available To Subscribers / Registered Guests At

[http://www.edweek.org/tm/articles/2010/08/16/netstudentsplagiarism_ap.html]

The P-Word II > The Ontology of Plagiarism: Part Two > Get A Clue

Colleagues/

In response to comments on his recent NY Times Opinionator Piece > Plagiarism Is Not A Big Moral Deal

[http://scholarship20.blogspot.com/2010/08/p-word-stanley-fish-and-i-agree.html] ,

Stanley Fish has authored "The Ontology of Plagiarism: Part Two "

[http://opinionator.blogs.nytimes.com/2010/08/16/the-ontology-of-plagiarism-part-two/].

IMHO > Most Still Do Not (Really) Recognize/Understand The Issue(s) > Please (Do) View / Read My Perspective On The Issue >>>

A few years ago I gave a keynote at the 3rd International Plagiarism Conference / 23 - 25 June 2008 / City Campus East, Northumbria University / Newcastle-upon-tyne, UK /

"Disruptive Scholarship: An Idea Whose Time Has Come: (Re)Use / (Re)Mix / (Re)New"

Abstract

Hadrian's Wall is a stone and turf fortification built by the Roman Empire across the width of modern-day England. ... [It was] 117 kilometres long,

... [I]ts width and height [were] dependent on the construction materials [that] ... were available nearby.

... [T]he wall in the east follow[ed] the outcrop of a hard, resistant igneous diabase rock escarpment ... Local limestone was used in the construction, except for ... section[s] in the west ... where turf was used instead ... .

The Broad Wall was initially built with a clay-bonded rubble core and mortared dressed rubble facing stones, but this seems to have made it vulnerable to collapse, and repair with a mortared core was sometimes necessary.... [I]n time ... [Hadrian's] Wall was abandoned and fell into ruin. Over the centuries and even into the twentieth century a large proportion of the stone was reused in other local buildings.

Throughout history, humans have (re)used local resources to create not only buildings and fortifications, but monuments, roads, and a wide variety of other structures. For countless generations, artists, composers, and writers have freely incorporated elements from local and distant cultures to create new visual, musical, and textual forms.

In The Web 2.0 World, the open (re)combination of multiple media has become commonplace in many venues, practices that Lawrence Lessig [snip], founder of Creative Commons [snip]and others, would characterize as emblematic of a 'Remix ' or 'Read/Write' culture. Indeed, from his point of view, “the health, progress, and wealth creation of a culture is fundamentally tied to this participatory remix process” [snip]

In the recently-released Horizon Report 2008 - a joint publication of the New Media Consortium (NMC) and the EDUCAUSE Learning Initiative (ELI), six emerging information technologies and practices that are expected to significantly impact educational organizations are profiled: Grassroots Video, Collaborative Webs, Mobile Broadband, Data Mashups, Collaborative Intelligence, and Social Operating Systems.

In this presentation, we will review the Read/Write Traditions of the Arts, Humanities, and Sciences; analyze key Past / Present / Future Participatory Technologies; and explore the potential of Web 2.0 for creating/fostering Disruptive Learning / Scholarship / Teaching in the 21st century.

The Director's Cut of the (150+ Slides) PPT is available from my _Scholarship 2.0_ blog at

[ http://bit.ly/9riXmc ]

I hope The Title and Abstract indicate That I Have A Different View Of The P-Word [:-)]

/Gerry