Monday, May 28, 2012

BioMed Central Blog > Assessing Research Impact at the Article Level

Posted by Ciaran O'Neill / Friday May 25, 2012

The impact of academic research has long been measured using citations, often with the Journal Impact Factor being used to assess individual publications within it. However, the Impact Factor is a journal level - not an article level - metric and, as academic publishing and the surrounding discussion moves increasingly onto the web, novel opportunities to track and assess the impact of individual scientific publications have emerged.

These web-based approaches are starting to offer an article-level perspective of the way research is disseminated, discussed and integrated across the web. The hope is that a broader set of metrics to complement citations will eventually give a more comprehensive view of article impact, ... . is one of a growing number of web-based tools taking a novel approach to the assessment of scholarly impact – it aggregates the mentions on twitter and social media sites, and coverage in online reference managers, mainstream news sources and blogs to present an overview of the interest a published article is receiving online. BioMed Central has today added the 'donut' to the about page of published articles – the donut will display for articles receiving coverage which has been tracked by, along with an article score ... .

The donut visualization shown on the 'about this article' page aims convey information about the type of attention the article has received ... .

This summary supplements our existing article-level measures of impact – article accesses and citations are displayed on all 'about this article' pages, ... . [snip].

As more indicators of article performance, visibility and impact emerge, the hope is that authors, readers and funding institutions will be able to assess research impact in a way which is more informed than relying on Impact Factors alone. [snip]. We plan to keep adding to this range of metrics and indicators, as they continue to expose a fuller image of research impact.

Source and Fulltext Available At 


Saturday, May 19, 2012

Latest Developments in PLoS Article-Level Metrics

By Richard Cave
Posted: May 14, 2012

 PLoS continues to expand and refine Article-Level Metrics (ALM). This suite of performance measures (including usage statistics, citations, trackbacks from blogs, bookmarks, social media coverage and user comments and ratings) are available on every PLoS article so that authors and the scientific community can assess the impact of the research. We are also broadening our outreach activities to spread the word on ALM to more researchers, technical experts, other publishers, funders, and institutions.

A key part of the current effort is to convene scholarly metrics thought leaders to help spearhead the widespread adoption of ALM. By engaging leading authorities in metrics, and bringing them together in a working group, PLoS can better coordinate the development of ALM. The following experts serve on the ALM Technical Working Group in an advisory role to help steer the direction of PLoS ALM implementation:

  • Pedro Beltrao, University of California San Francisco
  • Phil Bourne, University of California Santa Cruz
  • Bjoern Brembs, Freie Universit├Ąt Berlin
  • Martin Fenner, PLoS
  • Duncan Hull, European Bioinformatics Institute
  • Cameron Neylon, Science and Technology Facilities Council Oxford
  • Heather Piwowar, NESCent, Duke University
  • Jason Priem, University of North Carolina at Chapel Hill
  • Dario Taraborelli, Wikimedia Foundation
  • Jevin West, University of Washington
  • Johan Bollen, Indiana University


Source and Fulltext Available

PLoS > Article-Level Metrics

PLoS Article-Level Metrics (ALM): A Research Impact Footprint

Article-Level Metrics consist of a transparent suite of established measures that offer a view into the overall performance and reach of a research article;
  • Videos and Presentations
  • Upcoming Events
  • Latest News

Source and Links Available At 

New Data Sources Added to the PLoS Article-Level Metrics Program > Feb 2012

Cameron Neylon > Article-level Metrics > 2010

Article-Level Metrics at PLoS - What Are They, and Why Should You Care > Nov 2009

Wednesday, May 16, 2012

Journal of Librarianship and Scholarly Communication

The Journal of Librarianship and Scholarly Communication seeks to share useful innovations, both in thought and in practice, with the aim of encouraging scholarly exchange and the subsequent benefits that are borne of scrutiny, experimentation and debate. As modes of scholarly communication, the technologies and economics of publishing and the roles of libraries evolve, it is our hope that the work shared in the journal will inform practices that strengthen librarianship and that increase access to the "common Stock of Knowledge."

Current Issue: Volume 1, Issue 1 (2012)
Video Credit: Defining Scholarly Communication, created/owned by the contributors and produced by Kathryn Pope and Vin Aliberto, Center for Digital Research and Scholarship, Columbia University Libraries/Information Services. CC-BY.


What is in a Name? Introducing the Journal of Librarianship and Scholarly Communication
Isaac Gilman and Marisa Ramirez

In Memoriam: Deborah Barreau Gary Marchionini


The Movement to Change Scholarly Communication Has Come a Long Way – How Far Might It Go? / Joyce Ogburn

Coming in the Back Door: Leveraging Open Textbooks To Promote Scholarly Communications on Campus / Steven J. Bell

Point & Counterpoint: Is CC BY the Best Open Access License? / Klaus Graf and Sanford Thatcher

Research Articles

The Anatomy of a Data Citation: Discovery, Reuse, and Credit / Hailey Mooney and Mark P. Newton

The Accessibility Quotient: A New Measure of Open Access / Mathew A. Willmott, Katharine H. Dunn, and Ellen Finnie Duranceau

Does Tenure Matter? Factors Influencing Faculty Contributions to Institutional Repositories /
Anne M. Casey

Practice Article

Innovation Fair Abstracts, SPARC 2012 Open Access Meeting / Abstract Authors'

Theory Article

Open Access Publishing Practices in a Complex Environment: Conditions, Barriers, and Bases of Power / Thomas L. Reinsfelder

Brief Reviews of Books and Products

Developing Open Access Journals: A Practical Guide by David J. Solomon
Caitlin Bakker

Starting, Strengthening, and Managing Institutional Repositories / Jonathan A. Nabe
Therese F. Triumph

Embedding Repositories: A Guide and Self-Assessment Tool by JISC / Michele I. Wyngard

Source and Links Available At 

Tuesday, May 15, 2012

How Twitter Will Revolutionise Academic Research and Teaching

Monday 12 September 2011

Social media is becoming increasingly important in teaching and research work but tutors must remember, it's a conversation not a lecture, says Ernesto Priego

In chapter two of Christian Vandendorpe's From Papyrus to Hypertext titled: In the beginning was the ear, Vandendorpe says it took millenia "for literature to free itself from primary orality, albeit not completely". In the beginning all reading was done out loud, and it was not until the 12th century that books were created for silent reading. Orthographic signs and the separation between words had appeared around the 7th century, but did not become common until the 9th century amongst the learned communities of monks. Walter J Ong, in his classic study of writing and orality, Orality and Literacy defines as the "technologizing of the word" the process of developing a new relationship between language and thought.

Something similar is happening today in academia. Just like Augustine marveled, in the year 400, at the sight of Ambrose reading in silence, many members of academia marvel (or react with rejection) at the rapid changes in the production and dissemination of scholarly work and interaction between academics and those "outside" academic institutions. Thousands of scholars and higher education institutions are participating in social media (such as Twitter), as an important aspect of their research and teaching work.

There is still considerable resistance to embracing social media tools for educational purposes, but if you are reading this article you are probably willing to consider their positive effects. [snip].

Mobile phones and tablets enable the user in producing content, and to publish and disseminate it online. The microblogging platform Twitter is purposefully designed to exchange information and to facilitate reciprocal communication and attribution, [snip].

This is the key question where "the ear", our ability to listen and to place ourselves in a particular context at a particular time, comes to the fore. In traditional scholarly communications, academics produced a document ... . [snip]. Today, more and more academics are creating content and posting it online. [snip]. Therefore, the 21st century scholar has the tools not only to publish and disseminate, but also to facilitate the development of specialised audiences, and therefore of what is called "impact": people read, and in turn write about your work, which is in turn read by others.


"It's a conversation, not a lecture," is a well-known trope that is useful to remember in the scholarly web. This does not mean we should spend every waking hour chatting to strangers on social networks; it means that social media is not a uni-directional broadcasting tool. Those who "follow" us online are likely to be our students, colleagues, employers. They are not a passive audience.


Source and Fulltext Available At 

Sunday, May 13, 2012

Research Blogs and the Discussion of Scholarly Information

Shema H, Bar-Ilan J, Thelwall M (2012) Research Blogs and the Discussion of Scholarly Information. PLoS ONE 7(5): e35869. doi:10.1371/journal.pone.0035869


The research blog has become a popular mechanism for the quick discussion of scholarly information. However, unlike peer-reviewed journals, the characteristics of this form of scientific discourse are not well understood, for example in terms of the spread of blogger levels of education, gender and institutional affiliations. In this paper we fill this gap by analyzing a sample of blog posts discussing science via an aggregator called (RB). aggregates posts based on peer-reviewed research and allows bloggers to cite their sources in a scholarly manner. We studied the bloggers, blog posts and referenced journals of bloggers who posted at least 20 items. We found that RB bloggers show a preference for papers from high-impact journals and blog mostly about research in the life and behavioral sciences. The most frequently referenced journal sources in the sample were: Science, Nature, PNAS and PLoS One. Most of the bloggers in our sample had active Twitter accounts connected with their blogs, and at least 90% of these accounts connect to at least one other RB-related Twitter account. The average RB blogger in our sample is male, either a graduate student or has been awarded a PhD and blogs under his own name.

Source and Fulltext Availabae At


Sunday, May 6, 2012

Science and Truth: We’re All in It Together

Published: May 5, 2012

THE greatest bird news of our lifetime occurred at the height of the George W. Bush administration. In April 2005, amid a pageant of flags and cabinet ministers in Washington, John Fitzpatrick, the director of the Cornell Lab of Ornithology, announced that an ivory-billed woodpecker had been spotted for the first time in more than half a century in an Arkansas swamp.

President Bush pledged millions for habitat restoration. This and hundreds of other papers heralded the news. Public radio did one of those field reports in which you can hear the reporter’s canoe purling through swamp waters.

The news was exciting because the evidence of this new truth was overwhelming. There was an empirical article in the journal Science, an online video of the bird, audio clips reminiscent of its famous tinhorn squeak and seven sightings of the bird by credentialed experts.

Moreover, the ivory-bill is charismatic megafauna, regally beautiful and a natural mascot for fund-raising: a magnificent blast of snow in its trailing feathers, a jaunty red cap for a crown and a Harry Potteresque bit of white lightning down its neck. For it to appear after so many years was mythological, a message of forgiveness: maybe our environmental sins weren’t so bad. Not since the dove returned to Noah’s Ark has a bird’s appearance been so fraught.

Right away, though, there was controversy. Several academics, among them Richard Prum and Mark Robbins, questioned the evidence but held their criticisms when privately shown more and better data.

Then something new happened. Outsiders and other disbelievers kept on coming. A painter of birds, David Sibley (joined by several academics outside Cornell), dissected the video frame by frame and saw a common pileated woodpecker. Uh-oh. Then an amateur birder, Tom Nelson, began to gather the Internet commenters on his own blog. For the next several years, was a watering hole where weekend bird enthusiasts, field guides and others produced reams of counter-evidence and arguments, and so completely dismantled each piece of ivory-bill evidence that few outside the thin-lipped professionals at Cornell still believed in the bird.

Almost any article worth reading these days generates some version of this long tail of commentary. Depending on whether they are moderated, these comments can range from blistering flameouts to smart factual corrections to full-on challenges to the very heart of an article’s argument.


Should this part of every contemporary article be curated and edited, almost like the piece itself? Should it have a name? Should it be formally linked to the original article or summarized at the top? By now, readers understand that the definitive “copy” of any article is no longer the one on paper but the online copy, precisely because it’s the version that’s been read and mauled and annotated by readers. [snip].

We call the fallout to any article the “comments,” but since they are often filled with solid arguments, smart corrections and new facts, the thing needs a nobler name. Maybe “gloss.” [snip]


Sure, there is still the authority that comes of being a scientist publishing a peer-reviewed paper, or a journalist who’s reported a story in depth, but both such publications are going to be crowd-reviewed, crowd-corrected and, in many cases, crowd-improved. (And sometimes, crowd-overturned.) Granted, it does require curating this discussion, since yahoos and obscenity mavens tend to congregate in comment sections.

Yet any good article that has provoked a real discussion typically comes with a small box of post-publication notes. And, since many magazines are naming the editor of the article as well as the author, the outing of the editor can come with a new duty: writing the bottom note that reviews the emendations to the article and perhaps, most importantly, summarizes the thrust of the discussion. If the writer gains the glory of the writing, the editor can win the credit for chaperoning the best and most provocative pieces.

Some scientists are already experimenting with variations of this idea within the stately world of peer review. New ways to encourage wider collaboration before an article is published — through sites like ResearchGate — are attempts to bring the modern world of crowd-improvement to empirical research.

Already, among scientists, there is pushback, fear that incorporating critiques outside of professional peer review will open the floodgates to cranks. Not necessarily. The popular rejection last year of the discovery of a microbe that can live on arsenic was mercifully swift precisely because it was executed by online outsiders. Not acknowledging that crowd-checking and amateur commentary have created a different world poses its own dangers.

Take the case of the ivory-bill. The article in Science has never been retracted. Cornell still stands by its video. [snip]

Some may fear that recognizing the commentary of every article will turn every subject into an endless postmodern discussion. But actually, the opposite is true. Recognizing the gloss allows us to pause in the seemingly unending back and forth of contemporary free speech and free inquiry ... .


Source and Fulltext Available At 


CfP > Peer Reviewing and Research Evaluation

Tuesday, May 1, 2012

Refereeing Academic Articles in the Information Age


The new technology (such as ScholarOne) used for submitting papers to academic journals ...  increases the possibilities for gathering, analysing and presenting summary data on stages in the refereeing process. Such data can be used to clarify the roles played by editors and publishers as well as referees—roles less widely discussed in the previous literature. I conclude, after a review of related issues, that refereeing should be “open” in this information age—i.e. correspondence between editors, referees and authors should be open and available, and not private. Some of the issues involved in achieving this are outlined and discussed.

Practitioners' Notes
  • What is already known about this topic
The importance and the value of refereeing articles submitted to journals are widely debated and are contentious issues.
  • What this paper adds
This paper discusses these concerns in the context of electronic submission processes in general and the British Journal of Educational Technology in particular.
  • Implications for practice and/or policy
Electronic submissions allow for the collection of much more data (both public and private) on authors, editors and referees. These data should not be hidden but used to inform research and practice. In particular, open review (where the names of the authors and the referees are known to all concerned in the refereeing process) is possible and achievable.


Conclusions: a personal view

In this article I have discussed some of the current practices used by editors, authors and referees when using electronic submission and publishing systems. Some of these practices are more open than others, but I believe, in this information age of the WikiLeaks and Twitter, that little—if anything—should be hidden from different contributors to the total system. Thus, I feel that it is System 5 that we should be aiming for when it comes to refereeing. What little evidence there is (as opposed to opinion) suggests that, with open refereeing, there will be some improvement in the quality of the reports received and an increase in the number of reviewers recommending publication but that there will be a decrease in the number of reviewers willing to review. (This evidence can be found in the studies of Bingham et al, 1998; Smith, 1999; van Rooyen, Godlee, Evans, Black & Smith, 1999; Walsh, Rooney, Appleby & Wilkinson, 2000). However, no one to my knowledge has studied the effects of open peer reviewing and, because there is so little empirical evidence on these issues, it would be remiss of me to advocate strongly any or one particular approach. It would be good, nonetheless, to obtain more evidence on these issues in the future.

Note: Source and Fulltext Available To Subscribers

Hartley, J. (2012), Refereeing academic articles in the information age. British Journal of Educational Technology, 43: 520–528. doi: 10.1111/j.1467-8535.2011.01211.x

Related Blog Post Available At 

Web-based Journal Manuscript Management and Peer-Review Software and Systems > PPT

McKiernan, Gerry. “Web-based Journal Manuscript Management and Peer-Review Software and Systems.” Presentation given at the Workshop on Peer Review in the Age of Open Archives, International School for Advanced Studies, Trieste, Italy, May 24, 2003.

Self-archived At: 


See Also                  

McKiernan, Gerry. “eProfile: Web-based Journal Manuscript Management and Peer-Review Systems and Services.” Library Hi Tech News 19, no.7 (August 2002): 31-43.  Self-archived at


Web-based Journal Manuscript Management and Peer-Review Systems and Services

McKiernan, Gerry. “eProfile: Web-based Journal Manuscript Management and Peer-Review Systems and Services.” Library Hi Tech News 19, no.7 (August 2002): 31-43.

Self-archived At


See Also

McKiernan, Gerry. “Web-based Journal Manuscript Management and Peer-Review Software and Systems.” Presentation given at the Workshop on Peer Review in the Age of Open Archives, International School for Advanced Studies, Trieste, Italy, May 24, 2003. Self-archived at


The Virtues of Blogging as Scholarly Activity

April 29, 2012 / Martin Weller

I have been an active blogger since 2006, and I often say that becoming one was the best decision I have ever made in my academic life.

In terms of intellectual fulfillment, creativity, networking, impact, productivity, and overall benefit to my scholarly life, blogging wins hands down. ... [B]logging tops the list because of its room for experimentation and potential to connect to timely intelligent debate. [snip].

My academic identity—I'm a professor of educational technology at the Open University in the United Kingdom—is strongly allied with my blog. [snip].


This trend is evident in academic practice. Previously if I wanted to convey an idea or a research finding, my choices were limited to a conference paper or journal article or, if I could work it up, a book. These choices still remain, but in addition I can create a video, podcast, blog post, slidecast, and more. It may be that a combination of these is ideal—a blog post gets immediate reaction and can then be worked into a conference presentation, shared through SlideShare, or turned into a paper that is submitted to a journal. In each case the blog or social network becomes a key route for sharing and disseminating the findings. One recent study suggests that use of Twitter, for instance, can both boost and predict citations of journal articles.


In my book The Digital Scholar: How Technology Is Transforming Scholarly Practice (Bloomsbury USA, 2011; free online), I argue that if you look across all scholarly activities, the use of new technology has the potential to change practice. [snip]

Scholars no longer need a television series to engage the public. The academic publishing system is being thrown into question as we look at open-access approaches and different means of conducting peer review. And so on. There is hardly an area of scholarly practice that remains unaffected.

An example I like to cite is that of my colleague Tony Hirst, who blogs at OUseful.Info. He likes to play with open data, visualization tools, and mashups. On any one day an idea may occur to him, such as, '"I wonder how the people who tweet a particular link are connected? And who are they connected to?" In a short space of time, he will have experimented with data and tools to provide an answer, and blogged his results. None of this requires research funds or a peer-review filter, and it takes place over a much shorter time period than traditional research. Yet it would be difficult to argue that the blog does not constitute widely accepted definitions of scholarly research.

So I would argue that the answer to the first question above, as to whether new approaches such as blogging constitute scholarly activity, is an emphatic yes. Which leads us to a more problematic question: How should we recognize it?

This is where the issue of increased complexity really begins to cause friction. When it comes to tenure and promotion, the three factors generally considered are teaching, service, and research. The first two are easily understood by tenure committees. The last, and often most heavily weighted one for scholars expected to do research, is that of scholarship. Here tenure committees have increasingly come to rely upon journal-impact factors to act as a proxy for research quality. In short, we know what a good publication record looks like. But these criteria begin to creak and groan when we apply them to blogs and other online media. Simple metrics are subject to gaming, and because of the removal of the peer-review filter, may be meaningless anyway. [snip].

It's a difficult problem, but one that many institutions are beginning to come to terms with. Combining the rich data available online that can reveal a scholar's impact with forms of peer assessment gives an indication of reputation. Universities know this is a game they need to play—that having a good online reputation is more important in recruiting students than a glossy prospectus. And groups that sponsor research are after good online impact as well as presentations at conferences and journal papers.

Institutional reputation is largely created through the faculty's online identity, and many institutions are now making it a priority to develop, recognize, and encourage practices such as blogging.


For instance, I've found that since becoming a blogger, I publish fewer journal articles, so it has had a "negative" impact on that aspect of my academic life. However, it has led to so many other unpredictable benefits—such as the establishment of a global peer network that helps me stay up to date with my topic, increased research collaboration, and more invitations to give talks—that it's been worth the trade-off. [snip]

Source and Fulltext Available At