Advertisement
Essay

Essay Essays are opinion pieces on a topic of broad interest to a general medical audience.

See all article types »

Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation

  • Lorraine Catwell,

    Affiliation: Centre for Population Health Sciences, The University of Edinburgh, Edinburgh, Scotland

    X
  • Aziz Sheikh mail

    aziz.sheikh@ed.ac.uk

    Affiliation: Centre for Population Health Sciences, The University of Edinburgh, Edinburgh, Scotland

    X
  • Published: August 18, 2009
  • DOI: 10.1371/journal.pmed.1000126
  • Featured in PLOS Collections

This is the first in a monthly series of three articles on evaluating eHealth.

Summary Points

  • eHealth interventions will play a substantial role in shaping health care systems in the 21st century.
  • Until eHealth interventions are “fit for purpose”, health care professionals are unlikely to adopt them and this risks implementation failure.
  • eHealth developments should be viewed as interventions, and evaluated as new drugs or management programmes, recognising the challenges of evaluating complex interventions.
  • We propose a means to evaluate eHealth interventions while they are being designed, developed, and deployed.
  • We argue that continuous systematic evaluations of eHealth interventions are needed.

There is now considerable interest internationally in exploiting the potential of information communication technology (ICT) systems to improve the quality, safety, and efficiency of health care. Given that the adoption of ICT systems by health care providers is some 25 to 30 years behind many other private and public sectors [1], there is an understandable sense of urgency with which these eHealth initiatives are now being commissioned, developed, and deployed, typically at considerable expense [2][5]. The American Recovery and Reinvestment Act of 2009 [6], which includes $34 billion to incentivise health care professionals to “use a certified EHR (electronic health record) technology in a ‘meaningful manner’,” is a recent high-profile example of the sums of money that are being invested in eHealth [7].

Whilst eHealth interventions undoubtedly have the potential to play a substantial role in shaping and helping to create health care systems that are fit for the 21st century [8], experience has repeatedly shown that if attempts are made to implement poorly designed systems, there is a real danger that not only will the anticipated benefits fail to be realised [4],[5],[8][10], but also that vast sums of money will have been squandered in the process. Worse still, patients' safety may also be compromised [11][13].

The argument about the need rigorously to evaluate medical technologies themselves as well as their social and economic impact is not new [14][16]. However, those responsible for ICT developments must appreciate that health information systems should be evaluated with the same rigour as a new drug or treatment programme, otherwise decisions about future deployments of ICT in the health sector may be determined by social, economic, and/or political circumstances, rather than by robust scientific evidence.

Health care is in what Heathfield has described as “a catch 22 situation” [17]: until we develop eHealth interventions that are “fit for purpose” [18]—and early evaluations suggest they are frequently not [19]—health care professionals are, justifiably, reluctant to adopt these new technologies. The paradox is that unless we have the means to demonstrate the true benefits of these systems, which requires integrating these technologies throughout the health care industry, we will never have the necessary evidence to support the case for ICT in health care [17].

So, while financial incentives to adopt certain eHealth interventions may be understandable [6],[7], they should never be the main reason for their adoption. Instead, society must be able to judge the true value of eHealth interventions in its own right. Therefore a means simultaneously to evaluate eHealth interventions while they are being developed and deployed is required [20]. In this article, we argue for continuous systematic multifaceted evaluations—throughout the lifecycle of eHealth interventions—on the grounds that such an evaluative approach is likely to provide timely and relevant insights that can help to assess the short-, medium-, and long-term safety, effectiveness, and cost-effectiveness of eHealth interventions. The suggested lifecycle–based approach to evaluation should [20], we believe, become the norm rather than the exception, as is currently the case.

What Is eHealth?

In the past, the term “medical technology” was often used to describe the set of techniques, drugs, equipment, and procedures used by health care professionals to deliver medical care to individuals. ICT deployments would historically therefore have been considered under this heading [14]. Today, eHealth is the term more commonly used in relation to ICT deployments in health care; although there have been several attempts to define eHealth (see Box 1 for some examples), there is still no universal agreement on the precise meaning of this term.

Box 1. Example definitions of eHealth

  • “…a consumer-centred model of health care where stakeholders collaborate, utilising ICTs including Internet technologies to manage health, arrange, deliver and account for care and manage health care systems” [4].
  • “… today's tools for substantial productivity gains, while providing tomorrow's instrument for restructured, citizen-centred health systems” [10].
  • “… describes the application of information and communications technologies (ICT) across the whole range of functions that affect healthcare, from diagnosis to follow-up. It is the means to deliver responsive healthcare tailored to the needs of the citizen” [21].

Consider, for example, telemedicine interventions, i.e., the provision of health care services across distances by such means as telemonitoring devices (e.g., teleradiology and telecardiology), teleconsulting, or even telesurgery [21][24]. Numerous electronic medical records systems are used, for example, to record details of patients' health and their medication [25],[26]. Finally there are the many health portals that provide a means to access medical records and health-related information over a secure network, such as Google Health, Microsoft HealthVault, and the National Health Service's HealthSpace [27][29].

All of these systems have been defined as eHealth technologies [4],[10],[21], but all may serve very different purposes and all may have very different target user groups. For this reason, we suggest that any definition of eHealth should encompass the full spectrum of ICTs, whilst appreciating the context of use and the value they can bring to society. One such definition that includes these various facets is that proposed by Pagliari [30], who defined eHealth as:

“…an emerging field of medical informatics, referring to the organisation and delivery of health services and information using the Internet and related technologies. In a broader sense, the term characterises not only a technical development, but also a new way of working, an attitude, and a commitment for networked, global thinking, to improve healthcare locally, regionally and worldwide by using information and communication technology.”

The Anticipated Benefits of eHealth

As has been the case in many other sectors, it is widely believed the introduction of ICT systems within health care, combined with the necessary social (i.e., organisational and behavioural) changes [31], will substantially reduce costs and improve efficiency [10]; it is also anticipated that eHealth will lead to a reduction in the high number of patients who are inadvertently harmed by medical errors and violations [12].

For example, proponents of the introduction of electronic health records, which are currently being introduced in England, Scotland, France, Canada, Australia, and the USA, anticipate that such tools will lead to improvements in the recording, storing, retrieving, and sharing of patient information both within and between various stakeholder groups. Their hope is that this will, in turn, translate into improvements in the delivery of health and social care [26].

Similarly, telemedicine initiatives [21][24], which increasingly are being deployed in the context of the management of people with long-term conditions, have the ability to transcend many of the challenges health care professionals face in providing equitable, accessible, and high quality care to people living in remote locations and/or those who are housebound [4],[10]; such interventions can also improve the convenience of care by delivering it to people in the comfort of their own homes [4],[10]. Given that health care systems are (directly or indirectly) a major emitter of carbon gasses, the widespread use of telemedicine could also result in ecological benefits.

Another important initiative that offers considerable potential benefit is the development of health portals, which provide health care professionals, patients, their families, caregivers, and the public at large with direct access to health records as well as to relevant and accurate medical information at the touch of a button [27][29].

The potential benefits of these developments are considerable and multidimensional. It is important to note however that potential benefits do not equate with those that have been empirically demonstrated. More importantly, we must not ignore the potential risks associated with the implementation of ICT in complex environments such as health care services.

Recognising the Risk of Harm

As noted above, a number of governments around the world are currently engaging in truly epic programmes to roll out eHealth interventions as quickly as possible throughout the health care sector [32]. However, it is of concern that, in this rush, relatively little time, thought, or resources have been devoted to assessing the potential risks associated with eHealth interventions [11][13],[33][36].

eHealth may compromise patient safety in a number of ways. Take the example of poorly developed computer decision support functionality, which can result in the issuing of erroneous prescribing support and advice, as was demonstrated by Fernando et al. in their assessment of primary care prescribing software [33].

Another cause for concern is the roll-out of applications that allow round-the-clock, multiple access points to patient data through health portals such as Google Health, Microsoft's HealthVault, and the National Health Service's HealthSpace [27][29]. These applications have been designed to allow users to read, amend, and share their medical records with others, be they health care professionals, support groups, caregivers, family, friends, and/or other patients. But what mechanisms are in place to prevent cyber criminals from accessing or stealing sensitive data [34][36]? How will these and other eHealth applications be suitably evaluated so they are (and remain) secure portals for storing and exchanging potentially sensitive information [18]?

Box 2 details some high profile examples of ways in which eHealth deployments may have compromised patient safety and/or confidentiality.

Box 2. Examples of problems associated with eHealth projects

  • Large investments in eHealth may, by diverting resources result in a shortfall in funding for basic infrastructure, equipment, and staffing elsewhere in the system. For example, some community health centres in South Asia do not have the facilities or the surgical staff to carry out such basic procedures as a caesarean section, so investing in developments such as telemedicine, which are only likely to be accessible to a minority, would exacerbate the digital divide and existing health inequities [11].
  • eHealth applications that are inappropriately specified, have functional errors, are unreliable, user-unfriendly, or in a poorly prepared or supported environment can put patients' and the health service at risk. For example: (i) patients from Michigan were wrongly coded as being dead on medical bills; (ii) increased workloads on clinical users can in turn decrease in the quality of patient care; and (iii) delays in answering emergency calls due to problems with emergency dispatching systems may lead to delays in emergency treatment [12].
  • An unexpected increase in mortality was observed after the implementation of a commercially sold computerised physician order entry system; this increase may have been due to “system integration failure” and/or “human-machine interface flaws” [13].
  • Installation of “Trojan horse” software on end-user computers can capture: key stroke information, files stored on hard drives and Microsoft Outlook e-Mail files, all of which allow cyber criminals to steal electronic health records [36].

Implications for Evaluation

In order to maximise benefits and minimise risks, eHealth interventions need to be subject to the same independent scrutiny as any other health care intervention prior to implementation, i.e., they need to be suitably evaluated. Such evaluations need to begin with a clear description of a problem or need; for example, the need to improve access to health care information for both professionals and patients. A chain of reasoning then must take place that leads from the statement of the problem or need to the formulation of a possible solution (see Box 3) [37]. If any part of that chain is missing, it is highly probable that a poor-quality solution, or even a wrong solution, will be developed. As noted above, in the context of health care the implications of such failures may be particularly profound [11][13], but an evaluation programme capable of evaluating each part of this chain will help ensure that the right solution is developed and delivered for the need, whilst also recognising the importance of local contextual considerations.

Box 3. Suggested seven steps from statement of a problem to definition of a solution [37]

  1. Drivers: Clearly articulate why change is needed (e.g., problems posed by paper-based records in allowing ready multiple user access to patient records when needed).
  2. Vision: Realistically define possible responses to those drivers, i.e., what the revised model of delivering care will look like (e.g., patient records will be readily accessible from anywhere within the health care institution/setting).
  3. Goals: Explain how a project will move toward realising this vision (e.g., electronic health records will be deployed through a web-based secure network).
  4. Business objectives: Define how success will be measured (e.g., health care professionals, patients, their caregivers, and the public at large will have access to electronic health information over a secure network, from anywhere at any time, within x years at y cost). It is important that these timelines and costs are realistic.
  5. Business requirements: Define the detailed capabilities that will be needed in order to achieve these business objectives (e.g., detailed technical specifications; access to authority; social [i.e. behavioural and organisational] changes).
  6. Design: Propose possible solutions to meet the need and these requirements (e.g., commissioning of a custom-made Web-based electronic health record system).
  7. Solution: Develop and implement the solution and assess whether the problems have been resolved and the anticipated benefits realised within the proposed time scale and allocated budget.

Within biomedicine, randomised controlled trials (RCTs) are often seen as the “gold standard” methodological approach, rightly so because of their unique ability to control for the impact of known and unknown confounding factors [16]. But whilst RCTs and similar experimental design methods may be appropriate for studying interventions under controlled clinical conditions, these design methodologies alone are often less well suited to evaluate the impact of eHealth interventions in a complex environment or to study the effect they have on the delivery of care [16],[17]. The main reason for this deficiency is that studies adopting an experimental design approach fail to take sufficient account of the contextual considerations, which play a major role in the success or failure of the intervention being studied. It is therefore often difficult simply to generalise from results obtained using RCTs when studying complex interventions such as eHealth technologies [16],[17],[20].

In light of these considerations, and the current dilemma regarding investing in eHealth [17], we propose an alternative more comprehensive overall evaluation approach, one that encourages a multifaceted, multidisciplined approach and facilitates continuous systematic evaluations throughout the lifecycle of an eHealth intervention. The proposed approach takes into account sociotechnical and contextual considerations [38] and is capable of ensuring that each part of the chain of reasoning is adhered to [37]. Figure 1 depicts our suggested evaluation approach.

thumbnail

Figure 1. Schematic for simplified evaluation process.

doi:10.1371/journal.pmed.1000126.g001

The overall aim of this model is to maximise the benefits while minimising any risks associated with the eHealth intervention. This balance is achieved by iterative formative evaluations at four key stages of the eHealth intervention's lifecycle [20]. This model has the additional advantage of providing a means to understand the implementation process [17].

From the very beginning of an idea, for example, the replacement of paper-based records with electronic health records, it is important to be able to describe the vision, i.e., what will the new health service look like, including identifying in as much detail as possible measures of success/benefits, potential risk/costs, main stakeholders, and potential social changes [31]. The aim here is to gauge whether or not the idea has any perceived merit and, if so, which areas, if any, may need to be reconsidered. In this respect it is important that design teams take a multifaceted and multidisciplined approach to documenting the complex relationships between the political, social, organisational, and technical worlds. At some point, the rich picture of the real world needs to be developed into a conceptual model so that stakeholders can reflect critically on the drivers, vision, and goals of the project and agree whether or not such a programme of change is appropriate and feasible (see Figure 1: Phase I).

Only once the initial idea has been debated and accepted by the majority of key stakeholders should one proceed to the next stage, i.e., requirements elicitation and analyses. In this stage, design teams need to gain a thorough understanding of the stakeholders' needs, concerns, values, and beliefs, and define (as far as possible) what the eventual system will be expected to provide. It is important that this initial elicitation stage goes beyond functional and technical requirements and considers, for example, accessibility, acceptability, and affordability issues. Formative iterative evaluations using simple prototypes of the eHealth intervention may be useful at this stage to assist with the communicating of ideas, building a common understanding, agreeing to objectives, and securing stakeholder buy-in (see Figure 1: Phase II).

The third phase of the project involves the design, development, and testing of a system, including assessment and adoption of the social changes necessary to make the new system work [31]. Once a working model of the system is available, empirical evaluations can be completed, which could include the collection of quantitative and/or qualitative data, depending on the goals and scope of the study and the stage of development [20]. This stage of the evaluation process is likely to highlight any design faults and/or training needs. Therefore it is extremely important to take this opportunity to refine the design and/or address training needs before the system is rolled out to further sites (see Figure 1: Phase III).

The final stage is to implement and deploy the (working and accepted) system across the health sector, and in the process to undertake a series of formative evaluations of the system in operation under normal/everyday conditions. A summative evaluation should also be conducted to verify whether or not the new system meets the purpose (requirements) for which it was designed, whether or not the associated benefits have been realised, and whether or not there are any risks to patients and/or the health care system. At this stage, it is important to demonstrate that the original need has been satisfactorily met, opportunities to improve the system are highlighted, and “drivers for change” identified, whereby the cycle begins again (see Figure 1: Phase IV).

Conclusions

eHealth interventions have considerable potential to transform the health sector, hopefully for the better. As with any other intervention, however, the risk of harm exists, so policymakers, commissioners, clinicians, and patients alike need to remain aware of this possibility. If we are to maximise the benefits associated with eHealth interventions whilst minimising risks, we must be able simultaneously to evaluate eHealth interventions while they are being designed, developed, and deployed [20].

In this article, we have proposed a novel approach to evaluation that we believe addresses this need, while facilitating an evaluation of events leading up to the new system and continuing long after it has been implemented. Such systematic evaluation matters, because at the end of the day it must be hard scientific evidence that informs key policy decisions, rather than, as is currently so often the case, industry lobbying, political expediency, or enthusiasm to implement technology simply because it exists.

Author Contributions

ICMJE criteria for authorship read and met: LC AS. Wrote the first draft of the paper: LC. Contributed to the writing of the paper: LC AS.

References

  1. 1. Smadu M (2007) Supporting a Healthy Nation and a Health Economy Brief to the House of Commons Standing Committee on Finance. Available: http://www.cna-aiic.ca/CNA/documents/pdf​/publications/Pre-Budget_Brief_Aug_2007_​e.pdf. Accessed 22 July 2009.
  2. 2. National Center for Health Statistics Health (2007) United States, 2007, With Chartbook on Trends in the Health of Americans Hyattsville (Maryland). Available: http://www.cdc.gov/nchs/data/hus/hus07.p​df. Accessed 22 July 2009.
  3. 3. Brantley D, Lamey-Cummings K, Spivack R (2004) Innovation, Demand & Investment in Telehealth. Office of Technology Policy U. S. Department of Commerce. Available: http://www.telehealthlawcenter.org/loada​ttachment.php?attachmentid=117_1104_253. Accessed 22 July 2009.
  4. 4. Alvarez RC (2002) The promise of e-Health – a Canadian perspective. Ehealth International. Available: http://www.pubmedcentral.nih.gov/picrend​er.fcgi?artid=135525&blobtype=pdf. Accessed 22 July 2009.
  5. 5. Iakovidis I (2008) Assessing the benefits of eHealth – are we getting any now or do we still have to wait? Information Society and Media European Communities. Available: http://www.medcareshop.be/healthcare/hit​/seminaries2008/1.IliasIakovidis.pdf. Accessed 22 July 2009.
  6. 6. American Rein and Recovery Act of 2009 H. R. 1.US Government Information Office. Available: http://frwebgate.access.gpo.gov/cgi-bin/​getdoc.cgi?dbname=111_cong_bills&docid=f​:h1enr.pdf. Accessed 22 July 2009.
  7. 7. Dierker L, du Pont L (2009) American Reinvestment and Recovery Act: States and State-level HIE Foundation of Research and Education of AHIMA. Available: http://healthinfo.montana.edu/ARRA_Impli​cations%20for%20SLHIE%203_5_09.ppt. Accessed 22 July 2009.
  8. 8. Stead WW, Lin HS (2009) Computational Technology for Effective Health Care: Immediate Steps and Strategic Directions. National Academy of Sciences (pre-publication copy). Available: http://www.nlm.nih.gov/pubs/reports/comp​tech_prepub.pdf. Accessed 22 July 2009.
  9. 9. Rodrigues RJ (2008) Compelling issues for adoption of e-health. The Commonwealth Health Ministers Reference Book 2008. Available: http://www.ehealthstrategies.com/files/C​ommonwealth_MOH_Apr08.pdf. Accessed 22 July 2009.
  10. 10. Commission of the European Communities (2004) e-Health – making healthcare better for European citizens: An action plan for a European e-Health Area. Available: http://eur-lex.europa.eu/LexUriServ/LexU​riServ.do?uri=COM:2004:0356:FIN:EN:PDF. Accessed 22 July 2009.
  11. 11. Mudur G (2004) The great technological divide. BMJ 328: 788. Available: http://www.bmj.com/cgi/content/full/328/​7443/788. Accessed 22 July 2009.
  12. 12. Ammenwerth E, Shaw NT (2005) Bad Health Informatics Can Kill – Is Evaluation the Answer? Editorial. Methods Inf Med 44: 1–3. Available: http://iig.umit.at/dokumente/z29.pdf. Accessed 22 July 2009.
  13. 13. Han YY, Carcillo JA, Ventkataraman ST, Clark RSB, Watson SR, et al. (2005) Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 116: 1506–1512. Available: http://pediatrics.aappublications.org/cg​i/content/full/116/6/1506?maxtoshow=&HIT​S=10&hits=10&RESULTFORMAT=&fulltext=cpoe​&searchid=1&FIRSTINDEX=0&resourcetype=HW​CIT Accessed 22 July 2009.
  14. 14. Office of Technology Assessment (1976) Development of Medical Technology: Opportunities for Assessment. United States Congress Office of Technology Assessment NTIS order #PB-258117. Available: http://govinfo.library.unt.edu/ota/Ota_5​/DATA/1976/7617.PDF. Accessed 22 July 2009.
  15. 15. Ammenwerth E, Brender J, Nykänen P, Prokosch RM, Talmon J (2004) Visions and strategies to improve evaluation of health information systems: Reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Inform 73: 479–491. Available: http://66.102.1.104/scholar?q=cache:obLF​colvQXkJ:scholar.google.com/&hl=en Accessed 22 July 2009.
  16. 16. Kaplan B (2001) Evaluating informatics applications – some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform 64: 39–56. Available: http://66.102.1.104/scholar?q=cache:w4_u​GY7cGi0J:scholar.google.com/&hl=en Accessed 22 July 2009.
  17. 17. Heathfield H (1996) Current evaluations of information technology in health care are often inadequate. BMJ 313: 1008. Available: http://www.bmj.com/cgi/content/full/313/​7063/1008 Accessed 22 July 2009.
  18. 18. Car J, Black A, Anandan C, Cresswell K, Pagliari C, et al. (2008) The Impact of eHealth on the Quality & Safety of Healthcare A Systematic Overview & Synthesis of the Literature. Report for the NHS Connecting for Health Evaluation Programme March 2008. Available: http://www.pcpoh.bham.ac.uk/publichealth​/cfhep/documents/NHS_CFHEP_001_Final_Rep​ort.pdf. Accessed 22 July 2009.
  19. 19. Greenhalgh T, Stramer K, Bratan T, Byrne E, Russell J, et al. (2008) Summary Care Record Early Adopter programme: An independent evaluation by University College London. London: University College LondonUniversity College London. Available: http://www.ucl.ac.uk/openlearning/docume​nts/scrie2008.pdf. Accessed 22 July 2009.
  20. 20. Kushniruk AW, Patel VL (2004) Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 37: 56–76. Available: http://140.128.151.3/%E9%99%B3%E6%B0%B8%​E7%A6%8F/new%20Paper/Cognitive%20and%20u​sability%20engineering%20methods%20for%2​0the%20evaluation%20of%20clinical%20info​rmation%20systems%20.pdf. Accessed 22 July 2009.
  21. 21. Silber D (2003) The Case for eHealth. Presented at the European Commission's first high-level conference on eHealth May 22/23 2003. European Institute of Public Administration. Available: http://ec.europa.eu/information_society/​eeurope/ehealth/conference/2003/doc/the_​case_for_eHealth.pdf. Accessed 22 July 2009.
  22. 22. McLean S, Sheikh A (2009) Does telehealthcare offer a patient-centred way forward for the community-based management of long-term respiratory disease? Prim Care Respir J E-pub ahead of print. doi:10.3132/pcrj.2009.00006. Available: http://www.thepcrj.org/journ/aop/SG00155​.pdf Accessed 22 July 2009.
  23. 23. Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions on Telemedicine for the benefit of patients, healthcare systems and society. Commission of the European Communities (2008) Available: http://eur-lex.europa.eu/LexUriServ/LexU​riServ.do?uri=COM:2008:0689:FIN:EN:PDF. Accessed 22 July 2009.
  24. 24. American College of Physicians (2008) Position Paper. American College of Physicians. http://www.acponline.org/advocacy/where_​we_stand/policy/ehealth.pdf. Accessed 22 July 2009.
  25. 25. Garets D, Davis M (2006) Electronic Medical Records vs. Electronic Health Records: Yes, There is a Difference. A HIMSS Analytics White Paper. Chicago. Available: http://www.himssanalytics.org/docs/WP_EM​R_EHR.pdf. Accessed 22 July 2009.
  26. 26. Pagliari C, Detmet D, Singleton P (2007) Electronic Personal Health Records Emergence and Implications for the UK A Report to the Nuffield Trust. London. Available: http://nuffieldtrust.org.uk/ecomm/files/​Elec%20Personal%20Records%20II.pdf. Accessed 22 July 2009.
  27. 27. Google Health Available: https://www.google.com/accounts/ServiceL​ogin?service=health&nui=1&continue=https​%3A%2F%2Fwww.google.com%2Fhealth%2Fp%2F&​followup=https%3A%2F%2Fwww.google.com%2F​health%2Fp%2F&rm=hide. Accessed 22 July 2009.
  28. 28. Microsoft HealthVault Available: http://www.healthvault.com/. Accessed 22 July 2009.
  29. 29. NHS HealthSpace Available: https://www.healthspace.nhs.uk/visitor/d​efault.aspx. Accessed 22 July 2009.
  30. 30. Pagliari C, Sloan D, Gregor P, Sullivan F, Detmer D, et al. (2005) What Is eHealth (4): A Scoping Exercise to Map the Field. J Med Internet Res 7: e9. Available: http://www.jmir.org/2005/1/e9/. Accessed 22 July 2009.
  31. 31. World Bank (1996) The World Bank Participation Sourcebook. Available: http://www-wds.worldbank.org/external/de​fault/main?pagePK=64193027&piPK=64187937​&theSitePK=523679&menuPK=64187510&search​MenuPK=64187283&siteName=WDS&entityID=00​0009265_3961214175537. Accessed 22 July 2009.
  32. 32. Sauer C, Willcocks L (2007) Unreasonable expectations – NHS IT, Greek choruses and the games institutions play around mega-programmes. J Inform Technol 22: 195–201.
  33. 33. Fernando B, Boki SP, Savelyich A, Avery J, Sheikh A, et al. (2004) Prescribing safety features of general practice computer systems: evaluation using simulated test cases. BMJ 328: 1171–1172.
  34. 34. Vamosi R (2007) Study finds electronic health records vulnerable. News Blog. Available: http://news.cnet.com/8301-10784_3-977998​6-7.html. Accessed 22 July 2009.
  35. 35. Goodin D (2009) Prime Minister's health records breached in database attack. Scottish rich and powerful victimized. The Register. Available: http://www.theregister.co.uk/2009/03/02/​nhs_database_breach/print.html. Accessed 22 July 2009.
  36. 36. Brewin B (2008) Cyber criminals overseas steal U.S. electronic health records. Nextgov. Available: http://www.nextgov.com/nextgov/ng_200805​16_2203.php. Accessed 22 July 2009.
  37. 37. Beauchamp G (2007) Business Analysis—Delivering The Right Solution to the Right Problem (2007). http://www.businessanalystsolutions.com/​BA%20Chain%20of%20Reasoning.pdf. Accessed 22 July 2009.
  38. 38. Black A, Car J, Majeed A, Sheikh A (2008) Strategic considerations for improving the quality of eHealth research: we need to improve the quality and capacity of academia to undertake informatics research. Inform Primary Care 16: 175–177.