Establishing a Research Agenda for Scholarly Communication. 5

5. Value and Value Metrics in Scholarly Communications

Determining and measuring the effectiveness of and the value that is derived from scholarly communication is challenging and often subjective. John Houghton’s work on scholarly communication in Australia seeks to measure the economic and social returns to public sector investment in research and development and how those might rise with open access to published research findings. Analyses of citations in the published literature to the published literature lead to metrics such as the h-index, Eigenfactor, and the heavily-used impact factor. Extant measures may suffer from being tightly coupled to traditional processes while also inhibiting the application of other measures of value.

In the new digital environment, activities other than traditional or formal publication should be valued in the reward structure for scholarship. To this end, the Modern Language Association provides an example of examining current standards and emerging trends in publication requirements for tenure and promotion.

Illustrative Challenges
Citation analysis relies on a 50 year old assumption that the number of citations represents value, but in today’s environment this assumption is limiting. Other metrics could reflect the scholarly significance of new discoveries as they are developed and communicated. Effective metrics must be based on resources and practices that truly advance scholarly research. For example, it could be argued that journal articles have become totems to accrue and count for tenure and promotion but are not unique in their ability to advance scholarship and may be losing some effectiveness for this purpose.

"Open notebook science" and “open data” are examples of new research and communication practices that might be advancing scholarly research as much or more than what is possible through scholarly publication. The relationship between the reward system and indicators of the progress of knowledge is more tenuous. What resources and practices truly advance scholarly research? Even where robust indicators of the progress of knowledge exist, their relationship with the current reward system may be tenuous. How can the value and impact of communication practices be assessed and documented? How could these assessments be assimilated into the reward system?

Libraries should adopt a stronger role that more directly advances scholarly research beyond satisfying tenure and promotion practices. A starting point is work reported by King and Harley regarding formal vs. informal scholarly communications. Given that scholars are finding new ways to register, seek comment, refine, evaluate, and certify their work, how can those processes be tracked, recorded, measured, and reported as part of the value-chain in scholarship?

Should informal communications be captured and preserved by libraries, and if so, how? A useful analogy is to consider that presentations, preprints and letters and other informal communications are the conversations of science, while publications are the minutes. In some disciplines, journals are becoming less important to scholars than their professional meetings and informal networks where their accomplishments are recognized. How can librarians better characterize and, measure the contributions of these informal communications, and thereby make wise decisions about organized access to them?

Libraries also need to determine the value of their own services as contributions to the communication of scholarship. Which services, such as institutional repositories, should be evaluated, and what tools and measures exist for this purpose?

Research Possibilities
• Identify and evaluate the range of metrics currently used to measure the value and impact of scholarly publishing. There is citation analysis and its derivatives; but there are other measures being developed, including those that combine usage/readership and citation, such as those by the MESUR project at Los Alamos National Laboratory, and the UKSG Serials Group. A literature review to collect these efforts would provide a central reference point.
• Explore additional measurements that incorporate new kinds of indicators of value and covering a broader range of communication activities. New measures should address increased research efficiency and productivity, variations between disciplines, advancement of the process of research.

Other metrics may:
° characterize the value of Open Data and Open Notebook Science (disseminating source data, research methods, and negative experimental or clinical results) to advancing research and knowledge.
° correspond to technology transfer or other uses of new knowledge beyond generating further research, for instance, number of views, number of patents.
° show how informal communications are advancing the process of research

• Explore the relative value, importance, and significance of traditional journal and book publication compared to newer, informal forms of scholarly communication for a sample of representative scholars. This could build on studies by the CIC and Estabrook that indicated that in the humanities there is some acceptance of digital publications and new forms, while the scholarly monograph was still the standard for promotion and tenure.

[http://acrl.ala.org/scresearchagenda/index.php?title=Theme_5:_Value_and_Value_Metrics_of_Scholarly_Communications]

Establishing a Research Agenda for Scholarly Communication: A Call for Community Engagement

Establishing a Research Agenda for Scholarly Communication: A Call for Community Engagement

By ACRL Scholarly Communications Committee

Executive Summary:
The system of scholarly communication – which allows research results and scholarship to be registered, evaluated for quality, disseminated, and preserved – is rapidly evolving. Academic libraries and their parent institutions are adopting strategies, making plans, and taking action to respond to the changing environment and to influence its development. Believing that meaningful research can inform and assist the entire academic community in influencing and managing this evolution, the Association of College and Research Libraries (ACRL) convened an invitational meeting on July 21, 2007, to collectively brainstorm the evidence needed to inform strategic planning for scholarly communication programs.

Influencing a system as complex and dynamic as scholarly communication requires broad and deep understanding. The issues for investigation that emerged at the meeting range from cyberinfrastructure to changing academic organizational models to public policy. This report thematically summarizes and synthesizes the meeting’s rich discussion, framing eight essential research challenges and opportunities. We invite those engaged in creating, supporting, and distributing scholarship to comment and extend the issues and possible research initiatives. Without substantive comment from librarians and their partners, the goal of outlining a community research agenda cannot be considered complete.

Introduction

Themes and Research Opportunities:
Participants identified eight themes characterizing the changes transforming scholarly communications. Developing a deeper understanding of these challenges through research can enable academic stakeholders to influence and construct scholarly communication systems that optimally support the academic enterprise and the communities it serves.

Theme 1: The Impact and Implications of Cyberinfrastructure

Theme 2: Changing Organizational Models

Theme 3: How Scholars Work

Theme 4: Authorship and Scholarly Publishing

Theme 5: Value and Value Metrics of Scholarly Communications

Theme 6: Adoption of Successful Innovations

Theme 7: Preservation of Critical Materials

Theme 8: Public Policy and Legal Matters

Conclusion and Invitation

Appendix A: Attendee List

Appendix B: ACRL Scholarly Communications Committee Roster 2007-2008

Bibliography

Coverage and Commentary

[http://acrl.ala.org/scresearchagenda/index.php?title=Main_Page]

PDF Version Podcast Interview Press Release

Released November 2007

Show Me The Data! Show Me The Data! Show Me The Data!

Show Me The Data
Mike Rossner, Heather Van Epps, and Emma Hill
The Journal of Cell Biology, Vol. 179, No. 6, 1091-1092

The integrity of data, and transparency about their acquisition, are vital to science. The impact factor data that are gathered and sold by Thomson Scientific (formerly the Institute of Scientific Information, or ISI) have a strong influence on the scientific community, affecting decisions on where to publish, whom to promote or hire, the success of grant applications, and even salary bonuses. Yet, members of the community seem to have little understanding of how impact factors are determined, and, to our knowledge, no one has independently audited the underlying data to validate their reliability.

Calculations and negotiations
The impact factor for a journal in a particular year is declared to be a measure of the average number of times a paper published in the previous two years was cited during the year in question. For example, the 2006 impact factor is the average number of times a paper published in 2004 or 2005 was cited in 2006. There are, however, some quirks about impact factor calculations that have been pointed out by others, but which we think are worth reiterating here:

[snip]

3. Some publishers negotiate with Thomson Scientific to change these designations in their favor . The specifics of these negotiations are not available to the public, but one can't help but wonder what has occurred when a journal experiences a sudden jump in impact factor. For example, Current Biology had an impact factor of 7.00 in 2002 and 11.91 in 2003. [snip]

4. Citations to retracted articles are counted in the impact factor calculation. [snip]

5. Because the impact factor calculation is a mean, it can be badly skewed by a "blockbuster" paper. [snip]

When we asked Thomson Scientific if they would consider providing a median calculation in addition to the mean they already publish, they replied, "It's an interesting suggestion...The median...would typically be much lower than the mean. There are other statistical measures to describe the nature of the citation frequency distribution skewness, but the median is probably not the right choice." Perhaps so, but it can't hurt to provide the community with measures other than the mean, which, by Thomson Scientific's own admission, is a poor reflection of the average number of citations gleaned by most papers.

6. There are ways of playing the impact factor game, known very well by all journal editors, but played by only some of them. For example, review articles typically garner many citations, as do genome or other "data-heavy" articles [snip]. When asked if they would be willing to provide a calculation for primary research papers only, Thomson Scientific did not respond.

Integrity
As journal editors, data integrity means that data presented to the public accurately reflect what was actually observed. [snip]

Thomson Scientific makes its data for individual journals available for purchase. With the aim of dissecting the data to determine which topics were being highly cited and which were not, we decided to buy the data for our three journals (The Journal of Experimental Medicine, The Journal of Cell Biology, and The Journal of General Physiology) and for some of our direct competitor journals. Our intention was not to question the integrity of their data.

When we examined the data in the Thomson Scientific database, two things quickly became evident: first, there were numerous incorrect article-type designations. Many articles that we consider "front matter" were included in the denominator. This was true for all the journals we examined. Second, the numbers did not add up. The total number of citations for each journal was substantially fewer than the number published on the Thomson Scientific, Journal Citation Reports (JCR) website ... . The difference in citation numbers was as high as 19% for a given journal, and the impact factor rankings of several journals were affected when the calculation was done using the purchased data [snip]

Your database or mine?
When queried about the discrepancy, Thomson Scientific explained that they have two separate databases—one for their "Research Group" and one used for the published impact factors (the JCR). [snip]

When we requested the database used to calculate the published impact factors (i.e., including the erroneous records), Thomson Scientific sent us a second database. But these data still did not match the published impact factor data. This database appeared to have been assembled in an ad hoc manner to create a facsimile of the published data that might appease us. It did not.

Opaque data
It became clear that Thomson Scientific could not or (for some as yet unexplained reason) would not sell us the data used to calculate their published impact factor. If an author is unable to produce original data to verify a figure in one of our papers, we revoke the acceptance of the paper. We hope this account will convince some scientists and funding organizations to revoke their acceptance of impact factors as an accurate representation of the quality—or impact—of a paper published in a given journal.

Just as scientists would not accept the findings in a scientific paper without seeing the primary data, so should they not rely on Thomson Scientific's impact factor, which is based on hidden data. As more publication and citation data become available to the public through services like PubMed, PubMed Central, and Google Scholar®, we hope that people will begin to develop their own metrics for assessing scientific quality rather than rely on an ill-defined and manifestly unscientific number.

[snip]

[ http://www.jcb.org/cgi/content/full/179/6/1091]

The Insight Journal: An Open Access, Dynamic Publishing Environment

The Insight Journal is an Open Access on-line publication covering the domain of medical image processing. The unique characteristics of the Insight Journal include:

• Open-access to articles, data, code, and reviews : All submissions are available for free. The Insight Journal advances the idea that the results of scientific research should be made available to the public. Open access enables others to more fully understand the research and to more easily build upon the research. The field of medical image analysis will more rapidly advance.


• Open peer-review that invites discussion between reviewers and authors: The review process is made public and performed online. Every reader is a reviewer and may post comments. Reviewers' comments are posted with the submissions. Authors may respond to reviewers' comments. Authors may submit revisions to the papers in response to reviewers' comments. [snip]


• Emphasis on reproducible science via automated code compilation and testing: Code submitted to the IJ is verified by an automatic system. Authors are encouraged to post their papers along with the source code, the input data, and the expected output data needed to replicate the results presented in the paper. [snip]


• Support for continuous revision of articles, code, and reviews: The IJ is a dynamic publication environment. All submissions can be continually reviewed, tested, revised, and extended. [snip]


• The IJ is volunteer supported: We need your help in the form of submissions, reviews, and ideas! The IJ is a novel concept in the field of medical image analysis, and we need your time, intelligence, and insight to make it work.

[http://www.insight-journal.org/index.htm]

Economics: A Public Peer Reviewed e-Journal

Economics is a new type of academic journal in economics. By involving a large research community in an innovative public peer review process, Economics aims to provide fast access to top-quality papers. Modern communication technologies are used to find for every research issue the best virtual team out of a network of highly motivated researchers from all over the world. Thus, publishing is seen as a cooperative enterprise between authors, editors, referees, and readers. Economics offers open access to all readers and takes the form of an e-journal, i.e. submission, evaluation, and publication are electronic. economics embodies the following principles:

• Open Access: Following the principle that knowledge is a public good, all readers have open access to reading and downloading papers. The simple and free access ensures maximum readership and high citation records for published papers.
• Open Assessment: The traditional peer review process is substantially supplemented by a public peer review process in which the community of active researchers from all over the world has a hand in the evaluation process. Due to interactive peer review and public discussion, Eonomics provides fast and efficient quality assurance. Within a two stage publication process, much of the research evaluation takes place after rather than before an article is published.[snip]
• Submitted papers that have been identified as sufficiently promising for a referee process are made available on the journal’s homepage within three weeks. Thus, the time for new ideas to find their way into the scientific community is substantially reduced.
• Add-on Services: To foster scientific exchange, Economics embeds forums on special themes, where authors and readers can communicate and possibly conduct joint research. In addition, interested readers can take advantage of alert services announcing new papers in their fields. As far as possible, Economics also provides hyperlinks to the referenced literature.

Style and Contents: Economics aims to cover all the main areas of economics. Inevitably, articles in different areas of economics are addressed at different audiences. Many of the articles submitted to the journal are standard technical pieces, addressed to a purely academic audience. Others concern economic policy and thus are addressed both to economists and policy makers with some economic background. Yet others are surveys and overviews, often interdisciplinary, addressed to a non-technical audience. To attract this variety of contributions, Economics will contain the following areas, in addition to the standard contributions for a purely academic audience:

• Policy Papers
  Economics Policy Papers are concerned with the economic analysis of current policy issues. The analysis is rigorous, from a theoretical and empirical perspective, but the articles are written in non-technical language appropriate for a broad spectrum of economic decision makers and participants economic policy discussion. The rigorous analysis is contained in the appendix of each article.
• Surveys and Overviews
  Surveys and Overviews aim to integrate the analysis and lessons from various fields of economics with the aim of providing new insights, that are not accessible from any particular sub-discipline of economics. The contributions may include survey and review articles, provided that the broad perspective is maintained. They are addressed to a general audience interested in economic issues.
• New Frontier Papers
  New Frontier Papers contain articles that make potentially fundamental contributions to economic thinking. The papers are meant to change the way we conceptualize economic phenomena or transform the way the profession thinks about important economic issues. Contributions will be reviewed by renowned, top-flight economists, including Nobel prize winners.

[http://www.economics-ejournal.org/about-economics/aims-and-scope]

Review Process
Economics aims to allow the research evaluation process to be market-driven. The traditional peer review process is substantially supplemented by a "public" peer review process in which the community of active researchers from all over the world has a hand in the evaluation process. This is realized within a two-stage publication process.

First Stage: Publication of the submitted version as Economics Discussion Paper

Access to public peer review (less than three weeks)

The author submits a paper, if possible with hyperlinks to the referenced literature. [snip].

The managing editors check formal and professional standards and pass the paper to an associate editor in the relevant research field ... [snip].

The associate editor decides on whether to accept the paper for the further peer review process by answering two questions: (i) Whether the paper shows promise of making a significant contribution and (ii) whether it meets basic scientific standards. [snip]

If these questions are answered in the affirmative, the paper is published on the platform of the Economics Discussion Papers.

Open assessment of the submitted paper (eight weeks)

The Associate Editor appoints at least two referees for reviewing the paper and uploads their comments on the discussion platform within six weeks. The referees’ comments may be anonymous, but referees are encouraged to allow their reports to be attributed. Referees are asked to provide short reports that focus on two questions: (i) Is the contribution of the paper potentially significant? (ii) Is the analysis correct? [snip].

While the referees are reviewing the paper, the corresponding public discussion platform is open for eight weeks (the “discussion period”). On this platform, registered readers may review the paper by uploading comments (anonymously or with the name of the contributor published on the web). In addition, the referees´comments are posted on the Web site. Authors are asked to respond to these comments. All comments and responses are published and archived in the economics discussion paper section. This “public” peer review process lasts 8 weeks.

All registered readers can rate the comments by the criteria "extremly helpful, helpful, not helpful, unacceptable" (the results of these ratings are not made public). The editorial board of Economics reserves the right to delete unacceptable comments and exclude their commentators from the community of registered readers. [snip]

During the discussion period, the editorial staff sends the paper to potentially interested researchers (invited readers) to encourage them to post a comment in the discussion platform. [snip]

The author is free to respond to all comments.

Second Stage: Publication of the final version in Economics

Peer review completion

Based on the referee reports, the author replies, and the comments made by registered readers, the associate editor decides whether the paper is accepted or rejected for publication in Economics.

[snip]

If accepted, the discussion paper or its revised version is published in Economics. The Economics Discussion Paper with all comments is permanently archived and remains accessible to the public for documenting the paper’s history.

Market-based evaluation of published articles

Readers are asked to rate articles in economics on a scale from one to five ... [snip]

Readers are free to upload comments.

Authors are free to respond to all comments.

Authors are free to upload revised versions at any time.

Statistics on ratings, downloads and citations are collected to compile rankings of papers in the various fields of economics. Based on the rankings, prizes for the most outstanding papers in special fields are awarded each year.

[http://www.economics-ejournal.org/about-economics/two-stage-publication-process]

ELPUB2008: Open Scholarship: Authority, Community and Sustainability in the Age of Web 2.0

The International Conference on Electronic Publishing is entering its twelfth year and ELPUB2008 marks the first time the conference will be held in North America. The Knowledge Media Design Institute at the University of Toronto is pleased to serve as the host [site] ... [snip].


Call For Papers
Scholarly communications, in particular scholarly publications, are undergoing tremendous changes. Researchers, universities, funding bodies, research libraries, and publishers are responding in different ways, from active experimentation, adaptation, to strong resistance.

The ELPUB 2008 conference will focus on key issues on the future of scholarly communications resulting from the intersection of semantic web technologies, the development of cyberinfrastructure for humanities and the sciences , and new dissemination channels and business models. We welcome a wide variety of papers from members of these communities whose research and experiments are transforming the nature of scholarly communications.

Topics include but are not restricted to:

• New Publishing models, tools, services and roles
• New scholarly constructs and discourse method
• Innovative business models for scholarly publishing
• Multilingual and multimodal interfaces
• Services and technology for specific user communities, media, and content
• Content search, analysis and retrieval
• Interoperability, scalability and middleware infrastructure to facilitate awareness and discovery
• Personalisation technologies (e.g. social tagging, folksonomies, RSS, microformats)
• Metadata creation, usage and interoperability
• Semantic web issues
• Data mining, text harvesting, and dynamic formatting
• User generated content and its relation to publisher's content
• Usage and citation impact
• Security, privacy and copyright issues
• Digital preservation, content authentication
• Recommendations, guidelines, interoperability standards

Author Guidelines
Contributions are invited for the following categories:

• Single papers (abstract minimum of 1,000 and maximum of 1500 words)
• Tutorial (abstract minimum of 500 and maximum of 1500 words)
• Workshop (abstract max of 1000 words) - Poster (abstract max of 500 words)
• Demonstration (abstract max of 500 words)

Abstracts must be submitted following the instructions on the conference website (http://www.elpub.net).

Key Dates

January 20th 2008: Deadline for submission of abstracts (in all categories)

February 28, 2008: Authors will be notified of the acceptance of submitted papers and workshop proposals.

April 11th, 2008: Final papers must be received.

See Website For Detailed Author Instructions

Posters (A1-format) and demonstration materials should be brought by their authors at the conference time.

Only abstracts of these contributions will be published in the conference proceedings. Information on requirements for Workshops and tutorials proposals will soon be posted on the website.

All submissions are subjected to peer review (double-blind) and accepted by the international ELPUB Programme Committee.

Accepted full papers will be published in the conference proceedings. Printed proceedings are distributed during the conference.

Electronic versions of the contributions will be archived (http://elpub.scix.net)

[http://www.elpub.net/]

MediaCommons: A Digital Scholarly Network

MediaCommons, a project-in-development with support from the Institute for the Future of the Book (part of the Annenberg Center for Communication at USC) and the MacArthur Foundation, will be a network in which scholars, students, and other interested members of the public can help to shift the focus of scholarship back to the circulation of discourse. This network will be community-driven, responding flexibly to the needs and desires of its users. It will also be multi-nodal, providing access to a wide range of intellectual writing and media production, including forms such as blogs, wikis, and journals, as well as digitally networked scholarly monographs. Larger-scale publishing projects will be developed with an editorial board that will also function as stewards of the larger network.

What you see here now is simply an early stage along the way toward that network. We’re using the site now to test out some possible future features, such as In Media Res, and to solicit proposals for our initial large-scale projects. But we’re also using the site’s blog to plan in public, to generate conversation about what MediaCommons ought to become.

Our hope is that the interpenetration of the different forms of discourse will not simply shift the locus of publishing from print to screen, but will actually transform what it means to “publish,” allowing the author, the publisher, and the reader all to make the process of such discourse just as visible as its product. In so doing, new communities will be able to get involved in academic discourse, and new processes and products will emerge, leading to new forms of digital scholarship and pedagogy.For this reason, we want our readers and our writers intimately involved in MediaCommons not just after its fuller realization, but in its preliminary stages of development. Get involved in the various conversations around the blog, videos and project proposals. Submit a video yourself, or better yet, a proposal for a larger publishing project. Help us set the agenda for the future of publishing in media studies.

[snip]

More information about the genesis of MediaCommons is available ... :

• Introducing MediaCommons
• MediaCommons 2: Renewed Publics, Revised Pedagogies
• MediaCommons 3: Continuing the discussion
• Initial Responses to MediaCommons

Kathleen Fitzpatrick and Avi Santo, Editors

[http://mediacommons.futureofthebook.org/about]