Advertisement
Essay

Essay Essays are opinion pieces on a topic of broad interest to a general medical audience.

See all article types »

Progress in Using Systematic Reviews of Animal Studies to Improve Translational Research

  • C. R. Hooijmans mail,

    C.Hooijmans@cdl.umcn.nl

    Affiliation: Radboud University Nijmegen Medical Centre, SYRCLE at Central Animal Laboratory, Nijmegen, The Netherlands

    X
  • M. Ritskes-Hoitinga

    Affiliation: Radboud University Nijmegen Medical Centre, SYRCLE at Central Animal Laboratory, Nijmegen, The Netherlands

    X
  • Published: July 16, 2013
  • DOI: 10.1371/journal.pmed.1001482

Summary Points

  • During the last decade, new developments and initiatives have been introduced to improve the quality and translational value of animal research. We summarize these here, focusing on quality of study conduct, reporting, and replication; systematic reviews and meta-analyses; and study registration, publication bias, and data sharing.
  • Systematic reviews of animal studies should be conducted routinely. Funding agencies should subsidize systematic reviews, not simply for transparency, but also to avoid waste of financial resources and unnecessary duplication of animal studies.
  • An international register for animal studies should be established and funded. Journals should publish “negative and neutral” results, and promote data sharing.
  • Improving the quality and translation of animal research requires co-operation from the wider scientific community, journals, researchers, regulators, funding bodies, peer reviewers, and patients. Systematic reviews of preclinical studies should be included in the Cochrane Library.

Introduction

The scientific rationale for systematic reviews of animal studies was first outlined in an influential commentary published in 2002 [1]. The authors drew attention to the need for more rigorous assessment of animal studies before beginning studies in patients. They suggested that systematic reviews of all relevant animal experiments should be a prerequisite for the design of new clinical trials. Their concerns were prompted by a systematic review of animal studies of Nimodipine in focal cerebral ischemia, which showed insufficient evidence to justify involvement of over 7,000 patients in drug trials [2].

We summarize developments since 2002, focusing on the scientific rationale for systematic reviews of animal studies and the limitations and pitfalls. Moreover, we suggest further improvements in animal research to maximise its contribution to evidence-based translational research.

Translational Challenges

Although it may be preferable to study humans when seeking knowledge about human biology or the human response to interventions, it may be unethical or unfeasible. In these circumstances animal models are regularly used. One of the advantages of animal studies is the ability to study a relatively homogeneous group of animals instead of a heterogeneous group of patients. In addition, animal studies offer a wider range of possibilities to examine toxicity of interventions or study pathology and mechanisms of disease; most clinical trials only focus on clinical efficacy. Nevertheless, new therapies or interventions shown to be effective in animal studies are often less effective or ineffective in clinical trials [3]. Sometimes, interventions are even harmful to humans [2],[4],[5].

Several challenges exist to successfully translating the outcomes from animal research to humans in a clinical setting:

(1) Biological differences between species and strains

Genetic and species differences between animals and humans, but also within animal species, strains, and cell lines are often disregarded in the design of animal studies [6][8]. Ignoring biological differences between species and strains results in flawed design and unreliable outcomes, incurs unnecessary costs, and uses more experimental animals than necessary.

(2) Poor methodological quality of animal experiments

In many animal experiments, important methodological issues, such as randomization and blinding, are neglected [9][12]. In addition, the statistical methods used to analyze results are often flawed [11]. These failures mean that basic research cannot be replicated and may cause an overestimation of the efficacy of interventions [13]. Although clinical trials in humans also suffer from biases, preclinical animal studies appear to be associated with even greater risks [14].

(3) Differences in the design of experimental animal studies and clinical trials

Animal studies designed to decide whether or not to take an intervention forward to clinical trials, use study protocols that differ from clinical studies. For example, many of the animal studies investigating the effect of probiotics on pancreatitis administered probiotics before inducing pancreatitis, whereas in the clinical trial probiotics were given to patients already presenting with signs of pancreatitis [5].

(4) Insufficient reporting of details of animals, methods, and materials

Characteristics of the design of animal studies, such as the strain, gender, age and weight, and housing conditions of the animals used, are known to influence results. Failures in reporting these details skew the interpretation of study results and subsequent translation into clinical benefits.

(5) Publication bias

Not reporting experiments with negative or neutral results leads to an overestimation of the effect of an intervention [15]. Publication bias plays a role in both clinical trials and animal studies, but is believed to be more problematic in animal studies (14). In experimental stroke studies, for example, an estimated 14% of animal studies are unreported [16].

Scientific Rationale for Systematic Reviews of Animal Studies

In light of these translational challenges, it is no surprise that results in clinical trials often deviate from results of animal studies [17]. Systematic reviews are an important part of the solution because of the need to better understand how animal research informs clinical research [18][22].

By systematically reviewing the literature and ensuring that all animal studies are published regardless of outcome, unnecessary duplication of often expensive animal experiments can be reduced [23]. Moreover, systematic reviews can contribute to improving translation of animal research to humans in several ways: information about safety and efficacy of treatments that is hard to obtain from individual studies; misinterpretation of evidence is diminished because systematic reviews expose biases and inadequacies in the methodology of individual studies [24]; and differences in design between animal studies, and between animal and human studies, become transparent, and the most optimal animal models for evaluating a treatment before testing in humans can be selected [25],[26].

Systematic reviews of animal studies also have limitations. First, reviews are time consuming. Second, because of poor reporting and methodological inadequacies in animal studies, reviews may be unable to produce precise and reliable overall effect estimates. For now, systematic reviews should be primarily used to determine the direction of the effect, and the factors affecting the size and direction of the effect. Third, systematic reviews will be unable to address the issue of the low external validity of (certain) animal models to the extent that this is due to biological differences between humans and other animal species. [27]. Nevertheless, overall, the advantages of systematic reviews outweigh the limitations.

Improving the Translational Value of Animal Studies

Since 2002, when the scientific rationale for systematic reviews of animal studies was published [1], important developments and initiatives have aimed to improve the quality and translational value of animal research:

(i) Quality of study conduct, reporting, and replication

In 2009, the NC3Rs surveyed the quality of reporting, experimental design, and statistical analysis in 271 published animal studies [11]. The survey revealed a number of weaknesses in experimental design, statistical analysis, and reporting, and prompted publication of two sets of guidelines. The Animal Research: Reporting In Vivo Experiments (ARRIVE) guidelines [28] and the Gold Standard Publication Checklist (GSPC) [29] are based on the CONSORT guidelines for reporting clinical trials, and offer checklists for improving reporting of animal research. These initiatives may help to reduce bias in the conduct and reporting of animal studies, and thereby improve their predictive value [12]. Major scientific and medical journals have included the ARRIVE guidelines in their guidance to authors, and The Netherlands Organisation for Health Research and Development (ZonMW) urges researchers who receive funding from this organisation to use the GSPC. Moreover, in the UK, the Wellcome Trust, the Biotechnology and Biological Sciences Research Council, and the Medical Research Council have signed up to the ARRIVE guidelines. ZonMW has also issued a special funding programme for further development of education and supervision in this field.

Furthermore, an online interactive educational programme, known as 3Rs-REDUCTION (http://www.3rs-reduction.co.uk/) has been developed for researchers to improve the design of animal experiments and to better consider genetic variation in animal (toxicology) testing. Workshops arranged by the Fund for the Replacement of Animals in Medical Experiments (FRAME) complement this programme, and aim to educate animal researchers in methods and practices to use the minimum number of animals necessary for achieving the objectives of a study.

A key feature of good experimental practice is reproducibility. Reproducibility, or the lack of it, is a familiar issue for the scientific community, and the Science Exchange, PLOS ONE, Figshare, and Mendeley have recently launched the Reproducibility Initiative (https://www.scienceexchange.com/reproduc​ibility).

This initiative aims to identify and reward high quality, reproducible research. Results can be submitted for independent and “blinded” validation. Validated studies receive a “Certificate of Reproducibility,” acknowledging that their results have been independently reproduced as part of the Reproducibility Initiative. Scientists are able to publish the replicated results as an independent publication in the PLOS ONE Reproducibility Collection, and can share their data using the Figshare Reproducibility Collection repository. The director of the US National Office of Public Health Genomics says that the initiative “will begin to address huge gaps in the first of many translational steps from scientific discoveries to improving health.”

(ii) Systematic reviews and meta-analyses

Other initiatives are directly focused on the conduct of systematic reviews and meta-analysis of animal studies. In 2004, Pound and colleagues [30] called for a large-scale programme of systematic reviews of animal studies to help improve the quality of evidence derived from animal data. That same year, the chair of the National Institute for Clinical Excellence called for “detailed scrutiny of the totality of all the available evidence” to assess the “real predictive power” of the preclinical biological studies used in the research and development process [31].

In response to these appeals, an international group of animal researchers established the Collaborative Approach to Meta Analysis and Review of Animal Data from Experimental Studies (CAMARADES). This group focused particularly on systematic reviews of animal studies evaluating interventions for stroke and other neurological conditions.

In 2005, the Nuffield Council on Bioethics published a report stressing that more systematic reviews of animal studies are needed to understand the ethical and scientific issues in animal research [32], and urged funders of animal research to support systematic reviews. Simultaneously, the American Council on Science and Health called on US government agencies to adopt new guidelines for toxicology studies to identify cancer-inducing substances. The council insisted that modern methods be used to review systematically “the totality of evidence from animal studies, just as is done for studies on humans, rather than giving excessive weight to any one or two animal test results in one species” [33]. At the same time, a British charity, SABRE Research UK, was established by patients and researchers to promote systematic reviews of animal studies for better health care, and to protect patients and research volunteers from unsound research.

In 2008, the SYstematic Review Centre for Laboratory animal Experimentation (SYRCLE, previously named 3R Research Centre) was established in Nijmegen (The Netherlands) to improve the scientific quality and transparency of animal research, and to develop educational material and guidelines for systematic reviews of animal studies. In 2012, the Dutch Parliament adopted a motion in which the government was requested to ensure that systematic reviews also become the norm for animal experiments, just as is the case for clinical studies.

Although it is too early to provide evidence that systematic reviews lead to improved translational value of animal research, there are some examples [3],[34],[35]. The systematic review of Hirst et al., which was conducted after establishment of temozolomide as first line chemotherapy for malignant glioma, showed that temozolomide, also improved survival and reduced tumour volume in experimental glioma, just like in humans, even after accounting for publication bias [34]. In addition, the design of an ongoing trial on hypothermia in acute ischemic stroke (EuroHYP-1) was explicitly based on a systematic review of animal data [35]. Evidence-based selection of animal models will also improve translation, as the use of less suitable animal models is decreased [8],[25].

Scientists are increasingly convinced of the need of systematic reviews of animal studies. The number of systematic reviews of animal studies increased from 86 to 244 between 2005 and 2010. However, systematic reviews of animal studies are still relatively rare [15].

(iii) Study registration, publication bias, and data sharing

Obtaining data originating from preclinical animal studies is a priority, and calls have been made for prospective registration of all animal experiments [3],[24]. Although a registration system does not currently exist, initiatives have begun. The Dutch parliament adopted a second motion in February 2012 that defined a general database/registry of animal studies with the aim of preventing unnecessary duplication and reducing publication bias. The Netherlands Knowledge Centre on Alternatives to animal use (NKCA) is tasked with outlining the steps needed to implement this motion. A register of animal studies has different pitfalls and challenges compared to a registry of clinical studies.

Another strategy for obtaining data originating from preclinical animal studies, is publishing negative and neutral results, and promoting data sharing. In 2012, Begley and Ellis called for opportunities to present negative data, and suggested that journal editors should initiate this cultural change [36]. In addition, funding agencies and reviewers must acknowledge that negative data are just as useful as positive data. The Dutch ZonMw recently developed a special funding programme for animal research entitled, “More knowledge with fewer animals,” including a module providing funding to scientists for publishing negative or neutral results.

Several initiatives aimed at data sharing are being developed. First, REACH, the European Community Regulation on chemicals and their safe use (EC 1907/2006) (http://echa.europa.eu/chemicals-in-our-l​ife/animal-testing-under-reach) encourages information from tests to establish that the hazardous properties of chemicals are shared between registrants. Sharing of results of tests in vertebrate animals is already mandatory. The European Chemicals Agency, ECHA, facilitates sharing of data and information by companies, scientists, and citizens.

Second, Figshare was launched in 2011. Figshare allows researchers to publish research findings in an easily citable, sharable, and discoverable manner. By opening up the peer review process, researchers can publish neutral results, avoid the file drawer effect, and help make scientific research more efficient. Figshare uses creative commons licensing to share research data and allow users to retain ownership.

Third, F1000Research, the first open science journal, was launched in 2012. It is an open access journal for life scientists, offering immediate publication, transparent post-publication peer review, and full data deposition and sharing. All scientifically sound articles are accepted. This revolutionary development ensures and stimulates full openness and transparency.

Future Horizons

Although these initiatives are promising, there is no room for complacency. We must start by converting ideas into daily scientific practice.

More systematic reviews of animal studies are needed. Making systematic reviews of animal studies a routine is our scientific and societal responsibility, just as with clinical studies in humans. As a minimum, we urge researchers to conduct a systematic search for all relevant experimental animal studies before designing or starting a new animal experiment or clinical trial.

An international initiative to register animal studies must be started. In addition, journal editors must accept publications with so-called negative and neutral results, and promote data sharing.

Funding agencies should stimulate and fund systematic reviews. A recent article on forbes.com estimates that some major drug companies spend between US$4–$US11 billion per drug, once failure rates are factored in. Systematic reviews disclose inadequacies in methodology of individual studies. This helps improve future study design, and reduce failure rate of animal studies of new drugs [23].

Specifically, funding agencies can mandate systematic reviews of animal experiments as part of a funding. This will make the choice of animal models more evidence-based and provide better protection for human patients.

From a societal, ethical, and scientific perspective, laboratory animals must be used more efficiently. Better training of scientists who intend to work with animals, and improved research methods will assist the translational process, help the economic and ethical aspects of animal studies, and promote better health care and improved safety for patients and research participants.

Achieving success will depend upon co-operation between scientific journals, authors, animal research regulators, funding bodies, and peer reviewers. Progress also depends on the clinical research community demanding better quality pre-clinical research, and finding ways to improve communication with animal researchers whose pre-clinical work they may rely on [3].

Finally, a big step forward would be for organizations focused on systematic reviews of animal data, such as SYRCLE and CAMARADES, to join the worldwide network of the Cochrane Collaboration to improve links and cross-fertilisation between preclinical and clinical research, and facilitate translation from “bench to bedside.”

Acknowledgments

We gratefully acknowledge the valuable input of and discussions with Sir Iain Chalmers of the James Lind Library, Susan Green of SABRE Research UK, and Rob de Vries of SYRCLE.

Author Contributions

Wrote the first draft of the manuscript: CRH MR-H. Contributed to the writing of the manuscript: CRH MR-H. ICMJE criteria for authorship read and met: CRH MR-H. Agree with manuscript results and conclusions: CRH MR-H.

References

  1. 1. Sandercock P, Roberts I (2002) Systematic reviews of animal experiments. Lancet 360: 586. doi: 10.1016/s0140-6736(02)09812-4
  2. 2. Horn J, de Haan RJ, Vermeulen M, Luiten PG, Limburg M (2001) Nimodipine in animal model experiments of focal cerebral ischemia: a systematic review. Stroke 32: 2433–2438. doi: 10.1161/hs1001.096009
  3. 3. Perel P, Roberts I, Sena E, Wheble P, Briscoe C, et al. (2007) Comparison of treatment effects between animal experiments and clinical trials: systematic review. BMJ 334: 197. doi: 10.1136/bmj.39048.407928.be
  4. 4. Kenter MJ, Cohen AF (2006) Establishing risk of human experimentation with drugs: lessons from TGN1412. Lancet 368: 1387–1391. doi: 10.1016/s0140-6736(06)69562-7
  5. 5. Hooijmans CR, de Vries RB, Rovers MM, Gooszen HG, Ritskes-Hoitinga M (2012) The effects of probiotic supplementation on experimental acute pancreatitis: a systematic review and meta-analysis. PLoS One 7: e48811 doi:10.1371/journal.pone.0048811.
  6. 6. Festing MF (2009) Fifty years after Russell and Burch, toxicologists continue to ignore genetic variation in their test animals. Altern Lab Anim 37: 1–5.
  7. 7. Festing MF (2004) The choice of animal model and reduction. Altern Lab Anim 32 Suppl 2: 59–64.
  8. 8. van Drongelen J, Hooijmans CR, Lotgering FK, Smits P, Spaanderman ME (2012) Adaptive changes of mesenteric arteries in pregnancy: a meta-analysis. Am J Physiol Heart Circ Physiol 303: H639–657. doi: 10.1152/ajpheart.00617.2011
  9. 9. Bebarta V, Luyten D, Heard K (2003) Emergency medicine animal research: does use of randomization and blinding affect the results? Acad Emerg Med 10: 684–687. doi: 10.1111/j.1553-2712.2003.tb00056.x
  10. 10. Mignini LE, Khan KS (2006) Methodological quality of systematic reviews of animal studies: a survey of reviews of basic research. BMC Med Res Methodol 6: 10.
  11. 11. Kilkenny C, Parsons N, Kadyszewski E, Festing MF, Cuthill IC, et al. (2009) Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS One 4: e7824 doi:10.1371/journal.pone.0007824.
  12. 12. Landis SC, Amara SG, Asadullah K, Austin CP, Blumenstein R, et al. (2012) A call for transparent reporting to optimize the predictive value of preclinical research. Nature 490: 187–191. doi: 10.1038/nature11556
  13. 13. Macleod MR, van der Worp HB, Sena ES, Howells DW, Dirnagl U, et al. (2008) Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality. Stroke 39: 2824–2829. doi: 10.1161/strokeaha.108.515957
  14. 14. Ioannidis JP (2012) Extrapolating from animals to humans. Sci Transl Med 4: 151ps115.
  15. 15. Korevaar DA, Hooft L, ter Riet G (2011) Systematic reviews and meta-analyses of preclinical studies: publication bias in laboratory animal experiments. Lab Anim 45: 225–230. doi: 10.1258/la.2011.010121
  16. 16. Sena ES, van der Worp HB, Bath PM, Howells DW, Macleod MR (2010) Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol 8: e1000344 doi:10.1371/journal.pbio.1000344.
  17. 17. Bracken MB (2009) Why animal studies are often poor predictors of human reactions to exposure. J R Soc Med 102: 120–122. doi: 10.1258/jrsm.2008.08k033
  18. 18. Ioannidis JP (2006) Evolution and translation of research findings: from bench to where? PLoS Clin Trials 1: e36 doi:10.1371/journal.pctr.0010036.
  19. 19. Hackam DG (2007) Translating animal research into clinical benefit. Bmj 334: 163–164. doi: 10.1136/bmj.39104.362951.80
  20. 20. Hackam DG, Redelmeier DA (2006) Translation of research evidence from animals to humans. JAMA 296: 1731–1732. doi: 10.1001/jama.296.14.1731
  21. 21. Levin LA, Danesh-Meyer HV (2010) Lost in translation: bumps in the road between bench and bedside. JAMA 303: 1533–1534. doi: 10.1001/jama.2010.463
  22. 22. Hooijmans CR, Rovers M, de Vries RB, Leenaars M, Ritskes-Hoitinga M (2012) An initiative to facilitate well-informed decision-making in laboratory animal research: report of the First International Symposium on Systematic Reviews in Laboratory Animal Science. Lab Anim 46: 356–357. doi: 10.1258/la.2012.012052
  23. 23. Sena ES, Briscoe CL, Howells DW, Donnan GA, Sandercock PA, et al. (2010) Factors affecting the apparent efficacy and safety of tissue plasminogen activator in thrombotic occlusion models of stroke: systematic review and meta-analysis. J Cereb Blood Flow Metab 30: 1905–1913. doi: 10.1038/jcbfm.2010.116
  24. 24. Roberts I, Kwan I, Evans P, Haig S (2002) Does animal experimentation inform human healthcare? Observations from a systematic review of international animal experiments on fluid resuscitation. BMJ 324: 474–476. doi: 10.1136/bmj.324.7335.474
  25. 25. de Vries RB, Buma P, Leenaars M, Ritskes-Hoitinga M, Gordijn B (2012) Reducing the number of laboratory animals used in tissue engineering research by restricting the variety of animal models. Articular cartilage tissue engineering as a case study. Tissue Eng Part B Rev 18: 427–435. doi: 10.1089/ten.teb.2012.0059
  26. 26. Hooijmans CR, Pasker-de Jong PC, de Vries RB, Ritskes-Hoitinga M (2012) The effects of long-term omega-3 fatty acid supplementation on cognition and Alzheimer's pathology in animal models of Alzheimer's disease: a systematic review and meta-analysis. J Alzheimers Dis 28: 191–209.
  27. 27. Greek R, Menache A (2013) Systematic reviews of animal models: methodology versus epistemology. Int J Med Sci 10: 206–221. doi: 10.7150/ijms.5529
  28. 28. Kilkenny C, Browne W, Cuthill IC, Emerson M, Altman DG, et al. (2010) Animal research: reporting in vivo experiments: the ARRIVE guidelines. Br J Pharmacol 160: 1577–1579. doi: 10.1111/j.1476-5381.2010.00872.x
  29. 29. Hooijmans CR, Leenaars M, Ritskes-Hoitinga M (2010) A gold standard publication checklist to improve the quality of animal studies, to fully integrate the Three Rs, and to make systematic reviews more feasible. Altern Lab Anim 38: 167–182.
  30. 30. Pound P, Ebrahim S, Sandercock P, Bracken MB, Roberts I (2004) Where is the evidence that animal research benefits humans? BMJ 328: 514–517. doi: 10.1136/bmj.328.7438.514
  31. 31. Rawlins MD (2004) Cutting the cost of drug development? Nat Rev Drug Discov 3: 360–364. doi: 10.1038/nrd1347
  32. 32. Nuffield, Council, on, Bioethics (2005) The ethics of research involving animals. London.
  33. 33. American, Council, on, Science, and, et al. (2005) America's war on “carcinogens”: reassessing the use of animal tests predict human cancer risk. Meister K, editor. New York: American Council on Science and Health.
  34. 34. Hirst TC, Vesterinen HM, Sena ES, Egan KJ, Macleod MR, et al. (2013) Systematic review and meta-analysis of temozolomide in animal models of glioma: was clinical efficacy predicted? Br J Cancer 108: 64–71.
  35. 35. van der Worp HB, Macleod MR, Kollmar R (2010) European Stroke Research Network for H (2010) Therapeutic hypothermia for acute ischemic stroke: ready to start large randomized trials? J Cereb Blood Flow Metab 30: 1079–1093. doi: 10.1038/jcbfm.2010.44
  36. 36. Begley CG, Ellis LM (2012) Drug development: Raise standards for preclinical cancer research. Nature 483: 531–533. doi: 10.1038/483531a