Guidelines and Guidance

Guidelines and Guidance The Guidelines and Guidance section contains advice on conducting and reporting medical research.

See all article types »

Global Research Priorities to Better Understand the Burden of Iatrogenic Harm in Primary Care: An International Delphi Exercise

  • Kathrin M. Cresswell,

    Affiliation: The School of Health in Social Science, The University of Edinburgh, Edinburgh, United Kingdom

  • Sukhmeet S. Panesar,

    Affiliation: Centre for Population Health Sciences, The University of Edinburgh, Edinburgh, United Kingdom

  • Sarah A. Salvilla,

    Affiliation: Centre for Population Health Sciences, The University of Edinburgh, Edinburgh, United Kingdom

  • Andrew Carson-Stevens,

    Affiliations: Centre for Population Health Sciences, The University of Edinburgh, Edinburgh, United Kingdom, Cochrane Institute of Primary Care & Public Health, School of Medicine, Cardiff University, Cardiff, United Kingdom

  • Itziar Larizgoitia,

    Affiliation: WHO, Geneva, Switzerland

  • Liam J. Donaldson,

    Affiliation: Department of Surgery and Cancer, Imperial College London, London, United Kingdom

  • David Bates,

    Affiliations: Division of General Internal Medicine, Brigham and Women's Hospital, Boston, Massachusetts, United States of America, Harvard Medical School, Boston, Massachusetts, United States of America, Department of Health Policy and Management, Boston, Massachusetts, United States of America

  • Aziz Sheikh mail,

    Affiliations: Centre for Population Health Sciences, The University of Edinburgh, Edinburgh, United Kingdom, Division of General Internal Medicine, Brigham and Women's Hospital, Boston, Massachusetts, United States of America, Harvard Medical School, Boston, Massachusetts, United States of America

  • on behalf of the World Health Organization's (WHO) Safer Primary Care Expert Working Group

    Membership of the World Health Organization's (WHO) Safer Primary Care Expert Working Group is included in the Acknowledgments.

  • Published: November 19, 2013
  • DOI: 10.1371/journal.pmed.1001554

Summary Points

  • There is a need to identify and reach agreement on key foci for patient safety research in primary care contexts and understand how these priorities differ between low-, middle-, and high-income settings.
  • We conducted a modified Delphi exercise, which was distributed to an international panel of experts in patient safety and primary care.
  • Family practice and pharmacy were considered the main contexts on which to focus attention in order to advance patient safety in primary care across all income categories. Other clinical contexts prioritised included community midwifery and nursing in low-income countries and care homes in high-income countries.
  • The sources of patient safety incidents requiring further study across all economic settings that were identified were communication between health care professionals and with patients, teamwork within the health care team, laboratory and diagnostic imaging investigations, issues relating to data management, transitions between different care settings, and chart/patient record completeness.
  • This work lays the foundation for a range of research initiatives that aim to promote a more comprehensive appreciation of the burden of unsafe primary care, develop understanding of the main areas of risk, and identify interventions that can enhance the safety of primary care provision internationally.

Tackling the Provision of Unsafe Primary Care Internationally

It is now well established that medical errors are common and that these can result in considerable morbidity and mortality [1][3]. Much of this evidence, however, comes from hospital settings in industrialised countries where considerable progress has been made in describing the epidemiology of errors, understanding underlying contributing factors, and, more recently, taking steps to intervene to enhance patient safety [4].

In contrast, much less is known about the frequency of patient safety incidents and preventability of harm in primary care, particularly in low- and middle-income countries (Box 1). This is of concern as, in many parts of the world, primary care-based services now provide the first point of contact with health systems and often play a key role in coordinating more specialist care provision [5][8]. The increasing move to primary care-based health systems internationally [9],[10] adds further impetus to the urgent need for research into the frequency and preventability of patient safety incidents, but this is complicated by the considerable variation in population needs, economic and political circumstances, structures of health systems, and manifestations of primary care globally.

Box 1. Glossary of Key Definitions

Primary care

“Primary care is the provision of integrated, accessible health care services by clinicians who are accountable for:

addressing a large majority of personal health care needs

developing a sustained partnership with patients

practicing in the context of family and community” [10].

Patient safety

“Patient safety is the reduction of risk of unnecessary harm associated with health care to an acceptable minimum. An acceptable minimum refers to the collective notions of given current knowledge, resources available and the context in which care was delivered weighed against the risk of non-treatment or other treatment” [28].


“Harm implies impairment of structure or function of the body and/or any deleterious effect arising there from, including disease, injury, suffering, disability and death, and may be physical, social or psychological. Disease is a physiological or psychological dysfunction. Injury is damage to tissues caused by an agent or event and suffering is the experience of anything subjectively unpleasant. Suffering includes pain, malaise, nausea, depression, agitation, alarm, fear and grief. Disability implies any type of impairment of body structure or function, activity limitation and/or restriction of participation in society, associated with past or present harm” [28].

Low-, middle-, and high-income countries

“For operational and analytical purposes, the World Bank's main criterion for classifying economies is gross national income (GNI) per capita. In previous editions of our publications, this term was referred to as gross national product (GNP). Based on its GNI per capita, every economy is classified as low income, middle income (subdivided into lower middle and upper middle), or high income” [29].

In an attempt to support the development of a more comprehensive evidence-base, the World Health Organization (WHO) convened an international group of experts to discuss, debate, and advise on directions to bridge knowledge gaps around safe primary care, which would also serve to catalyse research in these areas internationally. A key strand of this foundational work was to identify a shared vision on relevant contexts of primary care and areas that would need further study to better understand the burden of harm in primary care settings internationally.

Developing Agreement on Primary Care Contexts and Priority Areas

We conducted a three-stage modified Delphi exercise, aiming to seek agreement on the most important contexts of primary care and the potential causes of patient safety incidents in different economic settings [11],[12]. This exercise was undertaken during a two-day expert meeting in February 2012 at the WHO headquarters in Geneva, Switzerland. AS, DB, and IL jointly chaired this meeting, which consisted of presentations, discussion groups, and plenary sessions focusing on understanding the challenges of assessing patient safety in primary care in low-, middle-, and high-income settings [13][15].

The Delphi technique has been widely used to help promote agreement amongst international experts. Key strengths of this process include the fact that it does not force consensus, but rather it can help to identify where agreement does and does not exist [16][19]. Although the general purpose and procedures have been retained in modified versions, some important differences in methods relate to: (1) approaches to managing interactions between experts, (2) the design of the initial item generation, and (3) feedback of individual scores [20][22].

Identifying Experts

We identified experts from academic, policy, and clinical backgrounds with expertise relating to patient safety in primary care settings. This initial long-list was generated by drawing on existing WHO contacts, profiling academic institutions with relevant expertise in the area, identifying authors of published studies, and by searching Google Scholar. Selected experts were invited to participate at the meeting. Provisions were made to facilitate representation from low- and middle-income countries in different geographical regions, and by meeting travel costs where financial considerations were a possible barrier to attendance.

Generating Candidate Statements and Prioritisation Exercise

Participants were provided with a review of the literature surrounding the frequency of patient safety incidents, burden of harm, and preventability of these incidents in primary care. Data were collected in three iterative stages with corresponding data collection forms (see Text S1). The forms were distributed and collected face-to-face by members of the research team (KMC, SAS, SSP, and ACS). Opportunity for free text comments was provided throughout and each participant was assigned a number for anonymisation purposes.

The forms included a list of candidate areas identified from the literature, which were grouped into three sections with corresponding statements for low-, middle-, and high-income countries (see Text S1). After piloting, the list was shared with the experts at the beginning of day 1 of the meeting, asking participants to add additional items. This list was amended on the basis of participant feedback.

The amended list formed the basis for the second round of data collection. Here, experts were asked to score items in terms of importance (frequency of occurrence, severity of outcome, preventability, inequity of occurrence) on a 9-point Likert-type scale for each income category ranging from 1 = “not important” to 9 = “extremely important” [15]. Data were independently scrutinised and transcribed by two members of the research team (KMC and ACS) into Microsoft Excel spread sheets and the median score for all items and the percentage agreement for items scoring 7, 8, or 9 (“usually important,” “very important,” and “extremely important,” respectively, i.e., the highest scores) were calculated.

The medians and percentage agreements obtained for each item were then included in the revised questionnaire that formed the basis for round 3 of data collection, giving participants the opportunity to revise their scoring on the basis of other participants' rankings. The third questionnaire was distributed and collected at the end of day 1, followed by calculation of the percentage agreement with individual items.

Items with an agreement of >80% in each section at the end of the Delphi exercise were fed back to participants on day 2 of the meeting. Rather than feeding back the actual distribution of panellists' ratings on the prior round, because of time and resource constraints the process fed back the median rating and percentage agreement of the prior round for the 7, 8, or 9 category as a proxy for the full distribution. This was followed by a plenary discussion, which gave participants the opportunity to collectively discuss emerging conclusions and recommendations. It enabled an exploration of areas of convergence and divergence across countries and perspectives, giving participants the opportunity to air concerns and discuss potential next steps.

Overarching Considerations to Improve Understanding of the Extent of Unsafe Primary Care

We distributed 40 questionnaires in round 1. Of these, 37 questionnaires were completed. Reasons for non-completion were participants either failing to return the questionnaire (n = 1) or leaving the meeting early (n = 2). No potential participant explicitly refused to participate in the study. Of the 37 questionnaires distributed in round 2, 34 were returned (non-responses were again mainly because some participants needed to leave the meeting early (n = 3)). In round 3, we distributed 30 questionnaires to the remaining participants (the other four had left the meeting), all of which were completed. The overall response rate was therefore 30/40 (75%; see Figure 1). Key characteristics of those who completed all three rounds of the Delphi exercise are detailed in Table 1.


Figure 1. Flow-diagram of the three-stage modified Delphi process.


Table 1. Key participant characteristics of experts in all three rounds of the modified Delphi exercise.


Overall, there was over 80% agreement across 15 items in low-income country contexts, 16 items in middle-income country contexts, and 16 items in high-income country contexts. Family practice and pharmacy were important primary care contexts across all income categories (Table 2). Additional contexts identified as warranting particular attention were community midwifery and nursing in low-income countries, and care homes in high-income countries (Table 2).


Table 2. Primary care contexts that were considered to be important by over 80% of participants after round 3.


Tables 3 and 4 summarise the factors responsible for patient safety incidents that were identified as particularly needing further investigation across income settings. Important additional items in low- and middle-income settings included counterfeit drugs and errors in the execution of clinical tasks, whilst additional items in high-income settings were systems management and technology-related issues (Table 5).


Table 3. Causes of patient safety incidents and their associated harm that were considered in need of further study by over 80% of participants after round 3 across all income settings.


Table 4. Five key causes of patient safety incidents that required further study across countries with different levels of income.


Table 5. Items relating to causes of patient safety incidents and associated harm that were considered to be important by over 80% of participants after round 3 in low-, middle-, and high-income settings.


Participants also prioritised the importance of cross-cutting systems' issues (Table 6). As can be seen, a range of interventional, regulatory, and methodological issues emerged; it is noteworthy that improved education and training for primary care workers received unanimous support.


Table 6. Cross-cutting items that were considered to be important to focus on by over 80% of participants after round 3.


Overall, we identified family practice and pharmacy as the main contexts to focus attention on in order to advance patient safety in primary care across all income categories. The sources of patient safety incidents requiring further study identified across all economic settings were communication between health care professionals and with patients, teamwork within the health care team, laboratory and diagnostic imaging investigations, issues relating to data management, transitions between different care settings, and chart/patient record completeness.

Strengths and Limitations of the Approach Employed

This work provides a foundation from which to focus efforts on how to better quantify the extent of iatrogenic harm in primary care and, in due course, to develop interventions to enhance the safety of primary care provision globally [5],[6]. The exercise allowed us to identify areas for research into safer primary care, focusing on areas with the greatest propensity for harm and where prevention was considered feasible. Incorporating this exercise within a two-day face-to-face meeting helped to ensure good response rates retaining the majority of participants, and allowed feeding back results to participants in real time, taking their comments into account in analysis activities and questionnaire design. The iterative element of the exercise helped to inform the meeting's proceedings and ensuing discussions. We provided participants with the opportunity to discuss emerging areas of agreement, thereby contributing to the face validity of the results. The different backgrounds and expertise of participants created the opportunity to explore the specific challenges associated with primary care provision across a range of geographical and political contexts.

Our work also has some important limitations. The total number of participants was limited by resource constraints. Furthermore, as the focus was on policy and research deliberations, we had an over-representation of high-income country-based scholars and doctors, potentially influencing the results in favour of medical concerns important to those working in industrialised countries. That said, it is important to note that community pharmacy, nursing, and midwifery emerged as consistent priority areas. Some participants acknowledged that they had limited insights into provision of care in low-income settings, which may have influenced their ability to make informed judgements. Expansion of this exercise, involving additional experts from wider professional domains and world settings may therefore generate additional important insights.

Because of resource and time constraints in collating data over a narrow time window, we were unable to include a reminder of participants' own prior ratings, which may to an extent have been mitigated by the fact that all three rounds were conducted on the same day. We were also unable to provide additional information on distributions of ratings that could point to potentially diverging ratings and hence disagreements amongst experts that might not be reflected in the summary scores. We did however examine the raw data from each round for ambiguous items, but did not detect any such instances.

Finally, it should be noted that some known important patient safety issues in primary care such as injection safety were not included in this exercise because of a lack of specific expertise among participants, though it was noted as an important gap [23].

Implications and Unresolved Issues

Some of the existing knowledge underlying the measurement, causal factors, and interventions to enhance patient safety in primary care may be applicable across a wide array of income settings. This knowledge may be particularly relevant in relation to common contexts of care provision, including the central positioning of family practice and community pharmacy in health systems globally, although there may be other contextual and institutional factors at play that need to be better understood. The expert consultation strongly advocated the need for further research surrounding the frequency and preventability of patient safety incidents in primary care.

The discipline of patient safety is built on the premise that harm arises from errors in a multifactorial chain of events [24],[25]. The underlying assumption is that if systems (i.e., organisations and networks of organisations) and working conditions within these organisations can be optimised, then the occurrence of adverse events is less likely. This systems approach is increasingly being applied as the patient safety culture of institutions and systems matures. Our findings support this trend, as reflected in the cross-cutting areas identified that all relate to improving ways of working collaboratively [26],[27].

The work accomplished in this meeting can now be used as a starting point to inform and focus efforts in relation to epidemiological investigations that are urgently needed, particularly in low- and middle-income country contexts. These insights can then be used to develop interventions that aim to reduce risks of iatrogenic harm and improve health outcomes. Once tested, effective interventions need to be incorporated into local and international policy making in order to ensure that findings are effectively translated into practice.


Family practice and pharmacy were identified as important contexts across all income categories. Particular areas identified as warranting further investigation included communication between health care professionals and with patients, teamwork within the health care team, laboratory and diagnostic imaging investigations, issues relating to data management, transitions between different care settings, and chart/patient record completeness. The WHO will be issuing a roadmap within the next 12 months to ensure that the momentum from this important initiative is maintained.

Supporting Information

Text S1.

Sample data collection form.




We thank those who supported this work in various forms and are not included as authors. These include the members of the World Health Organization's (WHO) Safer Primary Care Expert Working Group: Carlos Aibar (Universidad de Zaragoza, Spain), Hamad Al-Bulushi (Ministry of Health, Oman), Buthaina Al-Mudaf (Ministry of Health, Kuwait), Hisham Aljadhey (King Saud University, Saudi Arabia), Fawzi Amin (Ministry of Health, Bahrain), Anthony Avery (The University of Nottingham, United Kingdom), Pierre Barker (Institute for Healthcare Improvement, United States of America), Jean Brami (Department of Evaluation, ANAES, France), Perpetual Chikobvu (University of the Free State, South Africa), Aneez Esmail (The University of Manchester, United Kingdom), John Hickner (Cleveland Clinic, United States of America), Neil Houston (Primary Care at Dollar Health Centre, United Kingdom), Tawfik Khoja (Health Ministers Council for Gulf Cooperation, Saudi Arabia), Maaike Langelaan (Netherlands Institute for Healthcare Research, Netherlands), Mondher Letaief (University Hospital of Monastir, Tunisia), Chaojie Liu (La Trobe University, Australia), Rajan Madhok (Global Association of Physicians of Indian Origin, United Kingdom/India), Meredith Makeham (University of New South Wales, Australia), Philippe Michel (Bordeaux University Hospital, France), Yakoub Neyaz (Ministry of Health, Saudi Arabia), Ludovic Reveiz Herault (AMRO/HSS/PR, United States of America), Gurdev Singh (Patient Safety Research Center, United States of America), Ranjit Singh (Patient Safety Research Center, Unites States of America), Andreas Soennichsen (Paracelsus Medical University, Austria), Nicole Spieker (PharmAccess Foundation, Netherlands), Hans Trier (Danish Society of Patient Safety, Denmark), Amardeep Thind (The University of Western Ontario, Canada), Nana Twum-Danso (Africa Operations at Institute for Healthcare Improvement, Ghana), Wim Verstappen (Radboud University Nijmegen Medical Centre, Netherlands), Katharine Wallis (University of Otago, New Zealand), Stuart Whittaker (COHSASA, South Africa), Benedetta Allegranzi (WHO), Edward Kelley (WHO), Angela Diane Lashoher (WHO), Shamsuzzoha Babar Syed (WHO), Antonio Villafaina (WHO), Yonatan Yohannes (WHO), Kevin Wang (WHO) Maria-Carmen Audera-Lopez (WHO), Marie-Paule Kieny (WHO). We are also most grateful to the institutions that sponsored participants to the meeting.

Author Contributions

Analyzed the data: KMC SSP SAS AC-S. Wrote the first draft of the manuscript: KMC AS. Contributed to the writing of the manuscript: SSP SAS AC-S IL LJD DB WHOSPCEWG. ICMJE criteria for authorship read and met: KMC SSP SAS AC-S IL LJD DB AS. Agree with manuscript results and conclusions: KMC SSP SAS AC-S IL LJD DB AS WHOSPCEWG. KMC and SSP led the drafting of the protocol, which received critical input from SAS and AC-S.


  1. 1. The Health Founation (2011) Improving safety in primary care. Available:​proving-safety-in-primary-care-research-​scan/. Accessed 7 September 2013.
  2. 2. Fiss T, Dreier A, Meinke C, van den Berg N, Ritter CA, et al. (2011) Frequency of inappropriate drugs in primary care: analysis of a sample of immobile patients who received periodic home visits. Age Ageing 40: 66–73. doi: 10.1093/ageing/afq106
  3. 3. Woolf SH, Kuzel AJ, Dovey SM, Phillips RL Jr (2004) A string of mistakes: the importance of cascade analysis in describing, counting, and preventing medical errors. Ann Fam Med 2: 317–26. doi: 10.1370/afm.126
  4. 4. Makeham M, Dovey S, Runciman W, Larizgiotia I (2008) Methods and measures used in primary care patient safety research. Available:​h/methods_measures/makeham_dovey_full.pd​f. Accessed 7 September 2013.
  5. 5. Donaldson SL (2005) World alliance for patient safety: forward programme. Geneva: WHO.
  6. 6. Bates DW, Larizgoitia I, Prasopa-Plaizier N, Jha AK (2009) Global priorities for patient safety research. BMJ 338: b1775. doi: 10.1136/bmj.b1775
  7. 7. Starfield B (1998) Primary care: balancing health needs, services and technology. New York: Oxford University Press.
  8. 8. Reerink IH, Sauerborn R (1996) Quality of primary health care in developing countries: recent experiences and future directions. Int J Qual Health Care 8: 131–9. doi: 10.1093/intqhc/8.2.131
  9. 9. WHO (1978) Declaration of Alma-Ata. In International Conference on Primary Health Care. Alma-Ata, USSR: WHO.
  10. 10. Institute of Medicine (1996) Primary care: America's health in a new era. Available:​d=5152&page=27. Accessed 7 September 2013.
  11. 11. Mir G, Salway S, Kai J, Karlsen S, Bhopal R, et al. (2013) Principles for research on ethnicity and health: the Leeds Consensus Statement. Eur J Public Health 23: 504–510. doi: 10.1093/eurpub/cks028
  12. 12. Avery AJ, Savelyich BS, Sheikh A, Cantrill J, Morris CJ, et al. (2005) Identifying and establishing consensus on the most important safety features of GP computer systems: e-Delphi study. Inform Prim Care 13: 3–12.
  13. 13. Jones J, Hunter D (1995) Consensus methods for medical and health services research. BMJ 311: 376–80. doi: 10.1136/bmj.311.7001.376
  14. 14. Cuhls K (2012) Delphi method. Available:​phiMethod.pdf. Accessed 7 September 2013.
  15. 15. Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, et al. (1986) A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care 2: 53–63. doi: 10.1017/s0266462300002774
  16. 16. Elwyn G, O'Connor A, Stacey D, Volk R, Edwards A, et al. (2006) Developing a quality criteria framework for patient decision aids: online international Delphi consensus process. BMJ 333: 417–23. doi: 10.1136/
  17. 17. Ferri CP, Prince M, Brayne C, Brodaty H, Fratiglioni L, et al. (2005) Global prevalence of dementia: a Delphi consensus study. Lancet 366: 2112–7. doi: 10.1016/s0140-6736(05)67889-0
  18. 18. Shekelle P, Woolf SH, Eccles M, Grimshaw J (1999) Clinical guidelines: developing guidelines. BMJ 318: 593–6. doi: 10.1136/bmj.318.7183.593
  19. 19. Shekelle P (2004) The appropriateness method. Med Decis Making 24: 228–31. doi: 10.1177/0272989x04264212
  20. 20. Stewart J, O'Halloran C, Harrigan P, Spencer JA, Barton JR, et al. (1999) Identifying appropriate tasks for the preregistration year: modified Delphi technique. BMJ 319: 224–9. doi: 10.1136/bmj.319.7204.224
  21. 21. Mullen PM (2003) Delphi: myths and reality. J Health Organ Manag 17: 37–52. doi: 10.1108/14777260310469319
  22. 22. De Meyrick J (2012) The Delphi method and health research. Health Education 103: 7–16. doi: 10.1108/09654280310459112
  23. 23. WHO (2012) Injection safety. Available: Accessed 7 September 2013.
  24. 24. Institute of Medicine (1999) To err is human: building a safer health system. Washington (D.C.): National Academy Press.
  25. 25. Reason J (2000) Human error: models and management. BMJ 320: 768–70. doi: 10.1136/bmj.320.7237.768
  26. 26. Preston C, Cheater F, Baker R (1999) Left in limbo: patients' views on care processes across the primary/secondary interface. Quality in Health Care 8: 16–21. doi: 10.1136/qshc.8.1.16
  27. 27. Roland M (1995) R&D priorities in relation to the interface between primary and secondary care. Report to the Central Research and Development Committee. Leicester: Department of Health.
  28. 28. WHO (2009) More than words. Conceptual Framework for the International Classification for Patient Safety. Available:​ntation/taxonomy/icps_technical_report_e​n.pdf. Accessed 7 September 2013.
  29. 29. World Bank (2012) How we classify countries. Available:​classifications. Accessed 7 September 2013.