Search
Advanced Search

Article-Level Metrics Information

This page contains information about each of the article-level metrics that we track. Summary tables of 'average usage' are also available, as well as a page containing a technical description of our usage data in particular; and a summary Excel file containing the full data set.

Background

At PLoS, we believe that research articles should primarily be judged on their individual merits, rather than on the basis of the journal in which they were published. In March 2009, we inaugurated a program to provide "article-level metrics" on every article across all journals. This suite of relevant indicators of impact helps users determine the value of an article to them and to their scientific community. The regularly updated data fall into the following categories:

They are described further in the sections below.

Article-Level Metrics (ALMs) leverage the acceleration of research communication made possible by the networked landscape of researcher tools and services. Also by incorporating the manifold ways in which research is disseminated, these article impact indicators are made available rapidly after publication and are continually updated. It is important to note that the behavior of metrics varies by time (and needless to say by field and research area). For example, some metrics tend to accrue slowly over time; some are quicker to do so. Newly published articles will typically show lower levels of activity (for any given metric) for the initial weeks or months after publication than older articles. Further discussion of known limitations to individual metrics is detailed in the section below.

PLoS is committed to the open provision of these metrics; we encourage researchers to investigate and analyze them in new and interesting ways. Therefore, the entire dataset of all ALMs are made available as a summary Excel file. This file will be updated periodically. We also provide an API and accompanying documentation for the automatic retrieval of the full set of ALM data.

Article-Level Metrics Suite

Article Usage Data

Online usage via the PLoS platform is presented according to industry standard definitions of usage and is COUNTER 3-compliant (this is a standard which has been developed to report the usage of an entire journal in the context of a subscribing library, however we have applied the same rules to our own data analysis).

PLoS articles are provided in three different formats– HTML (browser view), PDF (often the preferred method when printing an article), and the original XML (back-end information, which generates the HTML and PDF files) – and we record the online activity of users across these three formats. These “article views” (divided by the three types of file format) are provided as an aggregate metric or broken down, month-by-month, in graphical format. Detailed, technical information relating to PLoS usage data can be found at the Usage Help page.

We also display COUNTER 3-compliant PubMed Central (PMC) usage data for each article. PMC individually counts the number of html page views and PDF downloads of the article on their site. The results are only made available to PLoS once a month, not in real-time. As a result, articles may experience a lag with the display of PMC data of up to one month. This will also impact the data shown on recently published articles, which may not show PMC usage data for their first month of publication. The total article usage data displayed on the Metrics Tab is an aggregate of both PLoS and PMC usage.

Online usage data should be interpreted with caution. In general, it is dependent on the age of the article and its subject area. To assist in the interpretation of usage data, PLoS provides summary tables showing useful average figures. In addition, interested researchers can download the entire dataset as an Excel file (updated periodically).

Robot activity may impact usage data. While we comply with COUNTER 3’s requirements to exclude its defined list of robots from reports, we employ more stringent benchmarks with an expanded list. PLoS will accommodate new COUNTER standards (e.g., standards devoted to article-level usage or expanded numbers of robot services to exclude) in the future as they come into effect. Be aware that PMC exclude a different (smaller) list of robots than is applied to the PLoS data.

We also provide detailed, technical information about our online usage data.

Citation Information

PLoS provides citation data on each article as computed by the following third-party citation measuring services: Scopus, Web of Science, PubMed Central and CrossRef. Each displays a single number (article citations) and links to a landing page containing information related to the citing articles.

Citation counts often vary between services, as each draws upon a different database. To attain the most comprehensive view of citations, consult all lists and "de-duplicate" the results.

Although Google Scholar extends the search for article citations beyond formal scholarly literature, it does not offer a formal mechanism to capture the data (i.e., an API service). As such, we provide a direct link to search article citations on Google Scholar’s site, but we are unable to provide a citation figure as measured by Google Scholar.

Blog Coverage

Blog posts serve as a common dissemination channel for articles published in PLoS journals. To identify and link to them from each article, we use third party blog aggregators: Researchblogging.org and Nature Blogs. For each service, we provide a numerical indicator of how many blog articles they have identified (relating to the article in question) and a link to their landing page containing the blogs counted.

The blog activity reported depends on the method of aggregation specific to each service. To attain the most comprehensive picture of how many (and which) blogs cite the article in question, consult all the constitutive lists and de-duplicate repeated entries.

Linking to blog coverage is not yet comprehensive: we rely on the ability of third parties to find blog postings and match them to PLoS articles. In many cases, blog authors do not reference the article in a way that allows for automated aggregation, and the aggregating services we link to cover only a selection of all possible blogs. Therefore, there will potentially be many more blogs about an article than these aggregators are able to identify. In recognition of this issue, we provide a link to a search on Google Blogs that may provide additional links to unlisted blogs.

The "trackback" functionality provides another way for bloggers to link to an article, and for us to automatically show that link on the PLoS site. If you are a blogger, we encourage you to use trackbacks and reference the article by its Digital Object Identifier (DOI). The trackback URL address can be found on the “Related Content” tab at each article.

Social Tools for Reference Management, Recommendation, and Evaluation

With the establishment of a networked landscape in research, researchers today employ a host of tools from which to manage references; disseminate articles; and evaluate each other’s work. PLoS has integrated the leading channels within these three areas into the ALM data suite to offer a more comprehensive view of the article’s impact after publication.

We provide activity data from common online reference management services which allow researchers to bookmark papers, collate references, and share sources with their community. Specifically, we capture data from the primary providers – CiteULike, Connotea, and Mendeley – – to indicate how many times the research article in question has been bookmarked by individual researchers or research groups. Each is linked to a landing page that allows users to navigate to other services such as subject tags and other bookmarked articles.

The CiteULike landing page captures total number of individuals and groups who have added the article to their CiteULike bookmarking account. There may be multiple users attached to each posting on this landing page, and they are found hyperlinked by the article listing. For example, the listing with the description: "posted by UserX along with 2 people and 1 group" will have a total count of 4.

We also capture data that track the dissemination activity of articles through the online mechanisms of Facebook. Given the ease and scope of digital propagation, researchers increasingly employ this social channel to recommend articles. This activity thus represents interest in the article, in a similar manner as usage data and provides insight into the reach of the article. The Facebook count reflects the aggregate number of Facebook Likes, "shares," "posts," and "comments" on an article (as reported by the Facebook API). At this time, the Facebook service does not offer a landing page which displays the entire collection of events included in the aggregate counts, and so they are not linked in the same manner as the other data sources.

We believe that appropriate use of the social network data types will aid the discovery of related papers as well as reveal the article’s readership reach. In collaboration with Cameron Neylon, this informational video discusses the power of such metrics as a research and discovery tool.

PLoS Reader Evaluation

The PLoS publishing platform allows users to leave Comments or Notes about an entire article or specific parts of the article, respectively. Comments cannot be anonymous and must adhere to PLoS’s guidelines for commenting. Commentators must declare competing interests (when they exist). PLoS staff monitors all comments.

Although we provide information on the number of Comment and Note threads that have been created, each Comment/Note thread may also contain multiple replies, which are not separately counted in the Metrics tab. (They are incorporated in the summary Excel file). Under the Comments tab, the full text of Comments and Notes can be found, accompanied by all replies. Please be aware that users may leave text comments when making a “Star” rating of an article - these comments can only be accessed by clicking on the Rating itself (although, again, the summary Excel file indicates which articles have Comments of this type).

The PLoS platform also supports the ability of users to leave "Star" ratings on articles. As with Comments and Notes, users may not be anonymous. We provide information on the number of ratings as well as a breakdown of the ratings in each of the three categories, and the overall average rating. The actual rating information and originating user information is available through the links.(This page may also include text notes made by that user).

Known Issues with Article-Level Metrics

Known limitations with ALMs include the following:

Robot activity may impact online usage data. PLoS has excluded all that are identified on this growing list, however PMC will be excluding a different list. No robot list is exhaustive and some level of robot usage will undoubtedly remain in the data.

Differences in PLoS usage data for article published prior to July 1st, 2005: Usage data reported for these articles is shown as an HTML view but actually represents a “combined” figure made up of the 3 view types. This primarily affects articles published in PLoS Biology and PLoS Medicine. Usage between HTML, PDF and XML view types cannot be separated due to problems with early log files before July 1st, 2005.

PubMed Central usage data: PMC statistics are COUNTER 3- compliant to the extent that they exclude internal use and crawlers/bots, and do not count duplicate requests for HTML pages or PDFs that are made within the limits specified by the standard. They are not compliant in that NLM does not provide usage data by specific IP, user, or organization. PMC began to make their usage data available to PLoS on January of 2010. Articles published before that point will not have PMC data prior to that time. We receive monthly reports from PMC of the prior month’s usage and so there may be a lag in the display of data up to one month’s time.

Scopus Citations: Scopus sometimes significantly undercounts the number of citations to a specific article. This is due to the existence of double records in their database for many PLoS articles. Hence, citations are spread across both records. PLoS is working with Scopus to improve their database in this respect, and so Scopus citation counts may increase in the future.

ISI Web of Science® Citations: The Web of Science® count reflects the sum total of citations for an article across all years and all five citations indices in their database. Individual users who search from their account may obtain lower results than the quoted Times Cited count if they do not have full access to the complete suite of ISI Web of Science® databases. To read more about how the ISI Web of Science® counts citing articles, please visit their help file.

CrossRef Citations: These citations to the article are provided by the CrossRef Cited-by Linking service. The data are limited to journals participating in CrossRef's Cited-by Linking service.

Although PLoS originally provided blog coverage data from Postgenomic, the service was closed down by Nature Publishing Group in 2010.Blog coverage data from BlogLines was also a live data source, but the service was discontinued by its original host institution. Both data sources have been removed from the display.

“Go-live” Dates for Different Data Sources and Functionalities: PLoS gradually expanded the set of data channels over time. Article published before the data source was integrated may not contain any data. Also, Commenting, Note making, Star rating and Trackback functionality were introduced at different points in time between journals. Articles published before this functionality was made live will typically show fewer comments/notes/ratings than articles published after this date. In addition, PLoS migrated the technology platforms for PLoS Biology, PLoS Medicine, PLoS Pathogens, PLoS Computational Biology, and PLoS Genetics, but it was not possible to include the "posting" dates of the comments that had been made up until that point. For these articles, all commenting data shows a "posting date" of the date of migration. The original posting date can be found by clicking into each comment.

All site content, except where otherwise noted, is licensed under a Creative Commons Attribution License.