This page contains information about each of the article-level metrics that we track. Summary tables of 'average usage' are also available, as well as a page containing a technical description of our usage data in particular; and a summary Excel file containing the full data set.
At PLOS, we believe that research articles should primarily be judged on their individual merits, rather than on the basis of the journal in which they were published. In March 2009, we inaugurated a program to provide "article-level metrics" on every article across all journals. This suite of relevant indicators of impact helps users determine the value of an article to them and to their scientific community. The regularly updated data fall into the following categories:
They are described further in the sections below.
Article-Level Metrics (ALMs) leverage the acceleration of research communication made possible by the networked landscape of researcher tools and services. Also by incorporating the manifold ways in which research is disseminated, these article impact indicators are made available rapidly after publication and are continually updated. It is important to note that the behavior of metrics varies by time (and needless to say by field and research area). For example, some metrics tend to accrue slowly over time; some are quicker to do so. Newly published articles will typically show lower levels of activity (for any given metric) for the initial weeks or months after publication than older articles. Further discussion of known limitations to individual metrics is detailed in the section below.
PLOS is committed to the open provision of these metrics; we encourage researchers to investigate and analyze them in new and interesting ways. Therefore, the entire dataset of all ALMs are made available as a summary Excel file. This file will be updated periodically. We also provide an API and accompanying documentation for the automatic retrieval of the full set of ALM data.
Online usage via the PLOS platform is presented according to industry standard definitions of usage and is COUNTER 3-compliant (this is a standard which has been developed to report the usage of an entire journal in the context of a subscribing library, however we have applied the same rules to our own data analysis).
PLOS articles are provided in three different formats– HTML (browser view), PDF (often the preferred method when printing an article), and the original XML (back-end information, which generates the HTML and PDF files) – and we record the online activity of users across these three formats. These "article views" (divided by the three types of file format) are provided as an aggregate metric or broken down, month-by-month, in graphical format. Detailed, technical information relating to PLOS usage data can be found at the Usage Help page.
We also display COUNTER 3-compliant PubMed Central (PMC) usage data for each article. PMC individually counts the number of html page views and PDF downloads of the article on their site. The results are only made available to PLOS once a month, not in real-time. As a result, articles may experience a lag with the display of PMC data of up to one month. This will also impact the data shown on recently published articles, which may not show PMC usage data for their first month of publication. The total article usage data displayed on the Metrics Tab is an aggregate of both PLOS and PMC usage.
Online usage data should be interpreted with caution. In general, it is dependent on the age of the article and its subject area. To assist in the interpretation of usage data, PLOS provides summary tables showing useful average figures. In addition, interested researchers can download the entire dataset as an Excel file (updated periodically).
Robot activity may impact usage data. While we comply with COUNTER 3's requirements to exclude its defined list of robots from reports, we employ more stringent benchmarks with an expanded list. PLOS will accommodate new COUNTER standards (e.g., standards devoted to article-level usage or expanded numbers of robot services to exclude) in the future as they come into effect. Be aware that PMC exclude a different (smaller) list of robots than is applied to the PLOS data.
Article-Level Metrics Article Usage charts are produced with Highcharts. Highcharts is not free for commercial use.
We also provide detailed, technical information about our online usage data.
PLOS provides citation data on each article as computed by the following third-party citation measuring services: Scopus, Web of Science, PubMed Central and CrossRef. Each displays a single number (article citations) and links to a landing page containing information related to the citing articles. PLOS also collects data for articles cited in Wikipedia encyclopedia entries. There is no external link for the Wikipedia landing page at this time.
Citation counts often vary between services, as each draws upon a different database. To attain the most comprehensive view of citations, consult all lists and "de-duplicate" the results.
Although Google Scholar extends the search for article citations beyond formal scholarly literature, it does not offer a formal mechanism to capture the data (i.e., an API service). As such, we provide a direct link to search article citations on Google Scholar's site, but we are unable to provide a citation figure as measured by Google Scholar.
We provide activity data from common online reference management services which allow researchers to bookmark papers, collate references, and share sources with their community. Specifically, we capture data from the primary providers – CiteULike and Mendeley – to indicate how many times the research article in question has been bookmarked by individual researchers or research groups. Each is linked to a landing page that allows users to navigate to other services such as subject tags and other bookmarked articles. For data that is displayed as an aggregate number, we display the breakdown of these counts in a table that appears as you hover over the respective tile.
The CiteULike landing page captures total number of individuals and groups who have added the article to their CiteULike bookmarking account. There may be multiple users attached to each posting on this landing page, and they are found hyperlinked by the article listing. For example, the listing with the description: "posted by UserX along with 2 people and 1 group" will have a total count of 4. The Mendeley count is an aggregate of the number of individuals and groups who have added the article.
With the establishment of a networked landscape in research, researchers today employ a host of tools from which to share and discuss each other's work. PLOS has integrated the leading channels within these three areas into the ALM data suite to offer a more comprehensive view of the article's impact after publication.
We believe that appropriate use of the social network data types will aid the discovery of related papers as well as reveal the article's readership reach. In collaboration with Cameron Neylon, this informational video discusses the power of such metrics as a research and discovery tool.
Blog posts serve as a common dissemination channel for articles published in PLOS journals. To identify and link to them from each article, we use third party blog aggregators: Researchblogging.org and Nature Blogs. For each service, we provide a numerical indicator of how many blog articles they have identified (relating to the article in question) and a link to their landing page containing the blogs counted.
The blog activity reported depends on the method of aggregation specific to each service. To attain the most comprehensive picture of how many (and which) blogs cite the article in question, consult all the constitutive lists and de-duplicate repeated entries. Linking to blog coverage is not yet comprehensive: we rely on the ability of third parties to find blog postings and match them to PLOS articles. In many cases, blog authors do not reference the article in a way that allows for automated aggregation, and the aggregating services we link to cover only a selection of all possible blogs. Therefore, there will potentially be many more blogs about an article than these aggregators are able to identify. In recognition of this issue, we provide a link to a search on Google Blogs that may provide additional links to unlisted blogs.
We also track the dissemination activity of articles through the online mechanisms of Twitter and Facebook. Given the ease and scope of digital propagation, researchers increasingly employ this social channel to recommend articles. This activity thus represents interest in the article, in a similar manner as usage data and provides insight into the reach of the article.
We track the sharing of PLOS articles on Twitter and provide the total number of tweets. We also display an archive of the tweets captured so that users can review the content of each tweet. The Facebook count reflects the aggregate number of Facebook Likes, "shares," "posts," and "comments" on an article (as reported by the Facebook API). At this time, the Facebook service does not offer a landing page which displays the entire collection of events included in the aggregate counts, and so they are not linked in the same manner as the other data sources.
We also track article discussion on the PLOS publishing platform, which allows users to leave Comments about an entire article or specific parts of the article, respectively. Comments cannot be anonymous and must adhere to PLOS's guidelines for commenting. Commentators must declare competing interests (when they exist). PLOS staff monitors all comments. Although we provide information on the number of Comment threads that have been created, each Comment thread may also contain multiple replies, which are not separately counted in the Metrics tab. Under the Comments tab, the full text can be found, accompanied by all replies.
We provide data on sources that capture formal endorsements of PLOS research articles via a platform such as an online recommendation channel. F1000Prime is a directory of recommended articles by their expert team of scientists and clinical researchers in biology and medicine. Each reviewer provides a numerical score to their qualitative evaluation. The metric is thus a sum of the scores from all the reviews, which the recommended article has received.
PLOS ALM relative metrics are a set of summary statistics aimed to provide context to primary ALM data. These metrics are statistical interpretations of the direct measurements that come directly from each of the ALM sources, and can be used to inform readers of the scope, meaning, and coverage of the primary data. The relative metrics set is still expanding, but at the moment, we offer the following: views-downloads ratio and average usage. The latter is in Beta mode as we continue to receive user feedback to refine the metric.
Views-downloads ratio: Though the research community engages with the article in numerous ways, the most basic mode is article access ("Usage"). The views-downloads ratio gives readers a view into the level of engagement on this particular dimension, based on the premise that a PDF download denotes expanded consideration of an article. We divide the HTML views by the PDF downloads to arrive at the percentage of views leading to a download. This ratio automatically normalizes articles by article age.
Limitations: In the event that an external link points directly to an article's PDF download, readers will arrive at the PDF download without creating an imprint on the HTML page. This occurrence will slightly inflate the proportion of downloads and lead to a lower views-downloads ratio.
Average Usage: The average usage calculation is a proximate measure of how a paper's total usage compares to the overall usage of a group of related papers. We define this reference set based on likeness across two dimensions, subject area and age, and then calculate the average from the usage count amongst this group. We use the median as the measure for central tendency.
We group articles based on age so as to control for large-scale institutional changes in reader behavior over time. We then group based on the seed article's subject matter, using its first and second level subject categories based on user selection. From this reference set of related papers, we calculate the average usage on a month-by-month basis. In this fashion, month X usage of the focal article is compared directly to the average usage for month X usage of all papers in its cohort and controls for the small variations in publication age amongst this set. Results are displayed as a line overlaid on the historical cumulative usage chart of the seed article.
Limitations: The context provided in the average usage is limited to the PLOS corpus as usage data is not widely available beyond PLOS journals at the moment. We begin displaying the average usage three months after an article's publication date to ensure there is sufficient data to compute a summary statistic. Due to the average usage methodological design, which groups articles based on publication year, an article may display a curtailed line depending on its publication date relative to the calendar year. Younger articles will reach a point in which they have been fully calculated and no longer have data to contribute to the monthly calculation. We cease to calculate the average usage at the point in which the first article drops out of the reference set. In the event that a reference set does not meet the minimum size for statistical significance (less than 25 articles) or usage data availability disruptions occur for an article in the reference set, the average usage is not computed. In the former, we suppress the subject category so that it is does not appear as a selection item. This situation may arise in certain niche subject areas that are not frequently covered. In the latter, the average usage line will be displayed up to the point in which such event occurs.
Known limitations with ALMs include the following:
Robot activity may impact online usage data. PLOS has excluded all that are identified on this growing list, however PMC will be excluding a different list. No robot list is exhaustive and some level of robot usage will undoubtedly remain in the data.
PLOS usage data: For article published prior to July 1st, 2005, usage data reported for these articles is shown as an HTML view but actually represents a "combined" figure made up of the 3 view types (HTML, PDF and XML) which could not be separated due to issues with early log files.
On April 14, 2014, we upgraded our COUNTER usage monitoring system to refine the management of user logs. This has been applied to all article activity that occurred from January 1, 2014 onwards.
PubMed Central usage data: PMC statistics are COUNTER 3- compliant to the extent that they exclude internal use and crawlers/bots, and do not count duplicate requests for HTML pages or PDFs that are made within the limits specified by the standard. They are not compliant in that NLM does not provide usage data by specific IP, user, or organization. PMC began to make their usage data available to PLOS on January of 2010. Articles published before that point will not have PMC data prior to that time. We receive monthly reports from PMC of the prior month's usage and so there may be a lag in the display of data up to one month's time.
Scopus Citations: Scopus sometimes significantly undercounts the number of citations to a specific article. This is due to the existence of double records in their database for many PLOS articles. Hence, citations are spread across both records. PLOS is working with Scopus to improve their database in this respect, and so Scopus citation counts may increase in the future.
ISI Web of ScienceⓇ Citations: The Web of ScienceⓇ count reflects the sum total of citations for an article across all years and all five citations indices in their database. Individual users who search from their account may obtain lower results than the quoted Times Cited count if they do not have full access to the complete suite of ISI Web of Science® databases. To read more about how the ISI Web of ScienceⓇ counts citing articles, please visit their help file.
CrossRef Citations: These citations to the article are provided by the CrossRef Cited-by Linking service. The data are limited to journals participating in CrossRef's Cited-by Linking service.
"Go-live" Dates for Different Data Sources and Functionalities: PLOS gradually expanded the set of data channels over time. Article published before the data source was integrated may not contain any data. Also, Commenting, Note making, Star rating and Trackback functionality were introduced at different points in time between journals. Articles published before this functionality was made live will typically show fewer comments/notes/ratings than articles published after this date. In addition, PLOS migrated the technology platforms for PLOS Biology, PLOS Medicine, PLOS Pathogens, PLOS Computational Biology, and PLOS Genetics, but it was not possible to include the "posting" dates of the comments that had been made up until that point. For these articles, all commenting data shows a "posting date" of the date of migration. The original posting date can be found by clicking into each comment.
Twitter: We began collecting tweets for PLOS articles on June 1, 2012, and the Twitter ALM count does not include data prior to this time period. We search and pull tweets based upon the DOI of the article, which is embedded in the article URL. Shortened URLs substantially modify this original URL structure, however. To the extent possible we attempt to reconstruct the long-form, original tweet URL and collect it for display.
The following have discontinued their services since the launch of ALMs: