... Open Access Will Open New Ways to Measure Scientific Output

The Next Revolution in Science ... 


Posted on April 19, 2012 by Tom Olijhoek

Open Access will not only change the way that science is done, it will also change the way that science is judged. The way that scientific output is measured today centers around citations. Essentially, on an author level this means the number of publications and citations of an author’s articles (author-level metrics). On a journal level, it means the average number of citations that articles published in that journal have received in a given time period (journal-level metrics).

For author-level metrics the Author citation Index has now been replaced by the H-Index that was introduced in 2005 by JE Hirsch. Here the criterion is the number of articles [n] that have received ≥ n citations at a fixed date. [snip]

[snip]

The H-Index, Google Scholar metrics and the JIF are all rather good indicators of scientific quality. However, in measuring real-world impact they are seriously flawed. Think for a moment of how impact is felt for whatever random topic you can think of. Everyone of us  will consider the publication itself, but probably also downloads, pageviews, blogs, comments, Twitter, different kinds of media and social network activity (Google+, Facebook), among other things. In other words, all activities that can be measured by “talking” through social media and other online activities can be used to give a more realistic impression of the real impact of a given research article. Since talking about articles depends on actually being able to read the articles, this is where open access comes into play.  The use of the proposed kind of article-level metrics only makes sense when many people are being able to discuss the actual content of published articles, which in turn is only possible when articles are open access. The optimal conditions for using altmetrics would be when articles would all be published as open access, but even with the current growth of open access published papers the method is already starting to make sense.

A number of article-level metrics services are currently in the start-up phase. A company called Altmetric is a small London-based start-up focused on making article level metrics easy. They do this by watching social media sites, newspapers and magazines for any mentions of scholarly articles. The result is an “altmetric” score which is a quantitative measure of the quality and quantity of attention that a scholarly article has received.

[snip] Finally, Total-Impact also makes extensive use of the analysis of social media and other online statistics, to provide a tool to measure total impact of a given collection of scientific articles, datasets and other collections. Their focus on collections represents still another approach to the problem of evaluating scientific output.

The previous overview is probably far from complete, so please feel free to add other possibilities in your comments to this post. However, I do think that the description above is an accurate reflection of the fact that the field of bibliometrics is moving fast and that Open Access will provide the key to the development and implementation of better ways to evaluate scientific output. Compared with the current practices, all of which are based on citations only, the inclusion of altmetrics plus online usage statistics and post-publication peer-review in an open access world will represent a true revolution in the way that science is perceived by all, scientists included.

Source and Fulltext Available At 

[http://bit.ly/J7VjK1]

1st International Workshop on Mining Scientific Publications

1. INTRODUCTION

Digital libraries that store scientific publications are becoming increasingly central to the research process. They are not only used for traditional tasks, such as finding and storing research outputs, but also as a source for discovering new research trends or evaluating research excellence. With the current growth of scientific publications deposited in digital libraries, it is no longer sufficient to provide only access to content. To aid research it is especially important to improve the process of how research is being done.

The recent development in natural language processing, information retrieval and the semantic web make it possible to transform the way we work with scientific publications. However, in order to be able to improve these technologies and carry out experiments, researchers need to be able to easily access and use large databases of scientific publications.

This workshop aims to bring together people from different backgrounds who:
(a) are interested in analysing and mining databases of scientific publications,
(b) develop systems that enable such analysis and mining of scientific databases or
(c) who develop novel technologies that improve the way research is being done.


2. TOPICS

The topics of the workshop will be organised around the following three themes:

• Infrastructures, systems, datasets or APIs that enable analysis of large volumes of scientific publications.
• Semantic enrichment of scientific publications by means of text-mining, crowdsourcing or other methods.
• Analysis of large databases of scientific publications to identify research trends, high impact, cross-fertilisation between disciplines, research excellence etc.

Topics of interest relevant to theme 1 include, but are not limited to:

Systems, services, datasets or APIs for accessing scientific publications and/or research data. The existence of datasets, services, systems and APIs (in particular those that are open) providing access to large volumes of scientific publications and their metadata is an essential prerequisite for being able to research and develop new technologies that can transform the way people do research. We invite papers presenting new systems, services, APIs or datasets that enable people to access databases of scientific publications and carry out their analysis. Papers addressing Open Access are of a special interest. We also invite papers that discuss issues and current challenges in design of these systems or address the issues of accessing and managing scientific publications and/or research datasets.

Topics of interest relevant to theme 2 include, but are not limited to:

• Novel information extraction and text-mining approaches to semantic enrichment of publications. This might range from mining publication structure, such as title, abstract, authors, citation information etc. to more challenging tasks, such as extracting names of applied methods, research questions (or scientific gaps), identifying parts of the scholarly discourse structure etc.
• Automatic categorization and clustering of scientific publications. Methods that can automatically categorize publications according to an established subject-based classification/taxonomy (such as Library of Congress classification, UNESCO thesaurus, DOAJ subject classification, Library of Congress Subject Headings) are of particular interest. Other approaches might involve automatic clustering or classification of research publications according to various criteria.
• New methods and models for connecting and interlinking scientific publications. Scientific publications in digital libraries are not isolated islands. Connecting publications using explicitly defined citations is very restrictive and has many disadvantages. We are interested in innovative technologies that can automatically connect and interlink publications or parts of publications, according to various criteria, such as semantic similarity, contradiction, argument support or other relationship types.
• Models for semantically representing and annotating publications. This topic is related to aspects of semantically modeling publications and scholarly discourse. Models that are practical with respect to the state-of-the-art in Natural Language Processing (NLP) technologies are of special interest.
• Semantically enriching/annotating publications by crowdsourcing. Crowdsourcing can be used in innovative ways to annotate publications with richer metadata or to approve/disapprove annotations created using text-mining or other approaches. We welcome papers that address the following questions: (a) what incentives should be provided to motivate users in contributing metadata, (b) how to apply crowdsourcing in the specialized domains of scientific publications, (c) what tasks in the domain of organising scientific publications is crowdsourcing suitable for and where it might fail, (d) other relevant crowdsourcing topics relevant to the domain of scientific publications.

Topics of interest relevant to theme 3 include, but are not limited to:

• New methods, models and innovative approaches for measuring impact of publications. The most widely used metrics for measuring impact are based on citations. However, counting citations does not take into account the publication content and the qualitative nature of the citation. In addition, there is a delay between the publication and the measurable impact in citations. We in particular encourage papers addressing new ways of evaluating publications’ impact beyond standard citation measures.
• New methods for measuring performance of researchers. Methods for assessing impact of a publication can often be extended to methods that can assess the impact of individual researchers. However, there are also other criteria for measuring impact in addition to publications, such as the development and publication of research data, economical and market impact, that should also be taken into account. We welcome papers addressing these aspects.
• New methods for measuring impact of research groups. The same as for impact of individual researchers holds for research communities.
• Methods for identifying research trends and cross-fertilization between research disciplines. Identifying research trends should allow discovering newly emerging disciplines or it should help to explain why certain fields are attracting the attention of a wider research community. Such monitoring is important for research funders and governments in order to be able to quickly respond to new developments. We invite papers discussing new methods for identifying trends and cross-fertilization between research disciplines using methods ranging from social network analysis and text- and data-mining to innovative visualization approaches.
• Application of mining from scientific databases. New methods and models developed for mining from scientific publications can be applied in many different scenarios, such as improving access to scientific publications, providing exploratory search in digital collections, identifying experts. We encourage papers describing innovative approaches that use scientific publications and data to solve real-world problems.

[snip]

5. IMPORTANT DATES

April 30, 2012 - Submission deadline
May 12, 2012 - Notification of acceptance
May 25, 2012 - Camera-ready
June 14, 2012 - Workshop

[snip]

Source and Fulltext Available At 

[http://core-project.kmi.open.ac.uk/jcdl2012/]

Google Starts Ranking Journals

 On April 1 2012,  Google announced a new feature to its Scholar service ...  called Google Scholar Metrics. The service follows the same principle that has made Google's web search engine so successful - when you are unsure what a user is looking for, give them a list of options ranked by a metric of popularity. In this instance, the users are academics ready to submit their next breakthrough but are uncertain which journal to choose. The solution Scholar Metrics offers is a database summarizing the sway of the distributors of scholarship "to help authors as they consider where to publish their new research".

Here's how it works. Google creates a list of all the articles a journal has published in a specified period of time. The citations to each article are counted in order to determine the publication's h-index, which is the largest number "h" such that each of the set of "h" articles were cited "h" or more times. As an example of how the h-index is calculated, consider a publication that has had six total articles having 2, 18, 11, 3, 22, and 9 citations, respectively. This gives the journal an h-index of four. Articles meeting the h-index criterion constitute the h-core. In the example, the core is the articles with 18, 11, 22 and 9 citations. Within the h-core, the median of the citation counts is used to assess the typical influence among the most highly cited set and is reported as the h-median.

[more]

Source and Links Available At

[http://bit.ly/HxHipX]

Investigation of Peer Review Processes For Digital Humanities Monographs

April Loebick /  April 10 2012 / 1:57 pm

The University Press of North Georgia (UPNG) recently won a National Endowment for the Humanities (NEH) Digital Start-Up grant for $24,923. This NEH Level I funding supports the press’s exploring peer review processes for publishing born-digital book length scholarly monographs in the Humanities in order to encourage their support, acceptance, and use in academia.



Project participants from the North Georgia College & State University community are BJ Robinson, UPNG Director and NEH Grant Project Director; April Loebick, UPNG Managing Editor; Markus Hitz, Professor of Computer Sciences; Chris Jespersen, Dean of the School of Arts & Letters; and Denise Young, Executive Director of Institutional Effectiveness. Other participants include the Directors of the University Press of Akron, the University Press of Florida, and Wayne State University Press. Advisory Board project members are Kathleen Fitzpatrick, Director of Scholarly Communication, Modern Language Association; Kathy Kiloh, Director of Athabasca University Press; Saskia deVries, Director of Amsterdam University Press; and members of the Open Textbook Consortium.

NEH Digital Start-Up grants are designed to encourage innovations in the digital humanities. Level I funding supports brainstorming sessions, data gathering, and initial planning. With this consortia of peers and publishing groups, UPNG will develop and pilot a model for peer review and eventual electronic publishing of single-author, digital monographs ; this model will involve sharing resources among small university presses to ensure economic viability and to help alleviate the pressures facing academic publishing.

[more]

Source and Fulltext Available At

[http://upnorthgeorgia.org/?p=804]

PLoS Computational Biology Goes Wiki

Posted on March 29, 2012 by Daniel Mietchen

Today saw an important step forward towards a wikification of scholarly workflows: PLoS Computational Biolog published an article that did not only follow the journal’s own author guidelines but also those for writing articles on the English Wikipedia, where a copy of the journal article has been pasted into [[Circular Permutation in Proteins]], where it shall live on in the hands of the wiki community.

The article is the first in a new manuscript track – Topic pages – that adds a dynamic component to articles published in the journal, as explained in the accompanying editorial:

This month, we have published our first Topic Page on “Circular Permutations in Proteins” by Spencer Bliven and Andreas Prlić [6] as part of our Education section. Topic Pages are the version of record of a page to be posted to (the English version of) Wikipedia. In other words, PLoS Computational Biology publishes a version that is static, includes author attributions, and is indexed in PubMed. In addition, we intend to make the reviews and reviewer identities of Topic Pages available to our readership. Our hope is that the Wikipedia pages subsequently become living documents that will be updated and enhanced by the Wikipedia community, assuming they are in keeping with Wikipedia’s guidelines and policies, either by individuals, or, perhaps as is already happening in medicine and molecular and cell biology, by something more organized, or with a more formal review structure. We also hope this will lead to improved scholarship in a changing medium of learning, in this case made possible by the Creative Commons Attribution License that we use.

[more]

Source and Access to the Fulltext and Associated Links Are Available Via

[http://bit.ly/HprjLq]

BTW: In 2005, I gave an invited presentation at the TICER intenational conference at Tilburg University, The Netherlands, titled

"Wikis: Disruptive Technologies for Dynamic Possibilities”

The presentation reviews the general nature and structure of select wikis, the features and functions of popular wiki software engines, and describes the content and use of wikis by select businesses, colleges and universities, and libraries.The presentation also speculated about the wiki as an environment, framework, and venue for Disruptive Scholarship, my proposed model for alternative scholarly authorship, review, and publishing.

A Link to the PPT is Available at

[http://disruptivescholarship.blogspot.com/]

More recently, I gave a presentation at the _Science in the 21st Century_ conference in Waterloo, Canada, titled

"The Wiki: An Environment For Scholarly Conversation and Publishing"

 A "wiki is a ... collaborative space ... because of its total freedom, ease of access, and use, [and] simple and uniform navigational conventions ... ." "[It] ... is also a way to organize and cross-link knowledge ..." Ward Cunningham, Father of The Wiki (Leuf and Cunningham, 2001, 16). Most wikis provide the user with a set of navigation or utility tools such as the ability to create and edit a page, view recently changed pages, and rollback to previous page versions. In addition, many wikis include a discussion forum for proposed page changes.

Among its many perceived benefits are its potential for facilitating a more creative environment and expanding knowledgebase, and a significant ability to harness the power of diverse point-of-views in creating collaborative works.

In this presentation, we will speculate on the Wiki as a digital environment that not only supports current scholarly practices, but more importantly, offers a framework for their enhancement and transformation.

Access to the PPT as well as associated A/V is available via

[http://scholarship20.blogspot.com/2008/09/wiki-environment-for-scholarly.html]

Altmetrics in the Wild: Using Social Media to Explore Scholarly Impact

Jason Priem,  Heather A. Piwowar, Bradley M. Hemminger

Abstract

In growing numbers, scholars are integrating social media tools like blogs, Twitter, and Mendeley into their professional communications. The online, public nature of these tools exposes and reifies scholarly processes once hidden and ephemeral. Metrics based on this activities could inform broader, faster measures of impact, complementing traditional citation metrics. This study explores the properties of these social media-based metrics or "altmetrics", sampling 24,331 articles published by the Public Library of Science.

We find that that different indicators vary greatly in activity. Around 5% of sampled articles are cited in Wikipedia, while close to 80% have been included in at least one Mendeley library. There is, however, an encouraging diversity; a quarter of articles have nonzero data from five or more different sources. Correlation and factor analysis suggest citation and altmetrics indicators track related but distinct impacts, with neither able to describe the complete picture of scholarly use alone. There are moderate correlations between Mendeley and Web of Science citation, but many altmetric indicators seem to measure impact mostly orthogonal to citation. Articles cluster in ways that suggest five different impact "flavors", capturing impacts of different types on different audiences; for instance, some articles may be heavily read and saved by scholars but seldom cited. Together, these findings encourage more research into altmetrics as complements to traditional citation measures.

Source and Fulltext Available Via

[http://arxiv.org/abs/1203.4745]

See Also

Altmetrics in the Wild: Towards Creating a Live CV

[http://bit.ly/GMntvD]

Webcast > Article-Level Metrics | Peter Binfield | April 12, 2012

Building New Measures for Impact: Article Level Metrics

Free SPARC online event / Thursday, April 12, 2012 / 12:00 - 1:00PM EST

>>> Registration is Free, but Required <<<

In March 2009, the Public Library of Science (PLoS) became the first publisher to track transparent and comprehensive information about the usage and reach  of published articles - rather than journals - so that the academic community has another avenue to help assess their value. These measures are called "Article-Level Metrics (ALMs)," and currently include:

• Article Usage Statistics - HTML pageviews, and PDF and XML downloads;
• Citations - From Web of Science, PubMed Central, Scopus, and Crossref:
• Social Bookmarks - currently from CiteULike and Connotea;
• Comments - left by article readers
• Notes - also from readers
• Blog posts - aggregated from a variety of sources

A primary aim of Article Level Metrics is to provide the academic community with new ways to evaluate individual articles directly on their own merits, rather than on the reputation of the journal in which they happen to be published. As a result, Article Level Metrics  hold the promise of helping new ways for measuring and evaluating research quality and impact - to evolve.

Source and link To Registration Available Via

[http://sparc.arl.org/civicrm/event/info?reset=1&id=105]

Presentation Now Available (04-12-12)

[http://www.arl.org/sparc/bm~doc/binfield-alm.pdf]

Webcast Now Available )03-23-13)

[http://www.arl.org/sparc/media/Binfield_Webcast_Article_Level_Metrics.shtml]