Introduction

Diabetes is a common and serious chronic disease associated with impaired quality of life, premature mortality, and significant economic costs [1]–[3]. Patients with diabetes require complex care to manage multiple risk factors such glycemia, blood pressure, and lipids, and to screen for and treat the many complications of the disease. Clinical practice guidelines assist health care providers to deliver this complex care by synthesizing the enormous literature on diabetes management into specific recommendations for care [4]–[6]. However, gaps between guideline recommendations and actual care delivered to patients with diabetes are well documented [7]–[10]. Therefore, the implementation of evidence-based guideline recommendations needs to be improved.

Although multifaceted and intensive quality improvement interventions have been shown to improve processes of care for diabetes patients [11]–[14], they are often difficult to widely implement because of their complexity and cost. In contrast, printed educational materials for clinician education are one of the most commonly used approaches for quality improvement and for increasing guideline adherence, because they are familiar, relatively inexpensive, and easily scalable to reach large populations [15]. A systematic review of methods to improve guideline adherence found that dissemination of educational materials led to an absolute improvement of more than 8% for dichotomous outcomes [15]. Therefore, when the Canadian Diabetes Association (CDA) updated their national clinical practice guidelines for diabetes in December 2008 [5], they created a quality improvement strategy of mailed educational toolkits to family physicians targeting a sequence of key themes from the guidelines. The first theme was cardiovascular disease screening and treatment, since it is the most prevalent complication of the disease and since evidence-based preventative interventions are readily available. The objective of this study was to evaluate the effectiveness of this educational toolkit to improve management of cardiovascular risk factors and outcomes of cardiovascular disease in people with diabetes.

Methods

We conducted a pragmatic cluster randomized controlled trial. A detailed description of the intervention and allocation has been previously published (Text S1) [16].

Results

We randomized 4,007 family practices in Ontario, to which 933,789 patients with diabetes were assigned in the administrative data study (Figure 1). In the clinical data study, we approached 372 intervention practices and 395 control practices to recruit 40 of each, and we examined the records of 1,592 randomly selected patients at high risk for cardiovascular disease. The pre-specified sample size was met. The baseline characteristics of patients and practices were well balanced (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Flow chart of practice and patient randomization in the trial.

https://doi.org/10.1371/journal.pmed.1001588.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Patient and practice baseline characteristics, by study group.

https://doi.org/10.1371/journal.pmed.1001588.t001

Administrative Data Study

The primary outcome, death and non-fatal myocardial infarction, occurred in 11,736 (2.5%) patients in the intervention group, and 11,536 (2.5%) patients in the control group (odds ratio 1.00, 95% confidence interval 0.96–1.03, p = 0.77). The other clinical event outcomes did not show benefit with the intervention (Table 2). Two of the processes of care secondary outcomes related to coronary artery disease screening were statistically significantly worse in the intervention group: electrocardiograms (38.8% versus 40.2%, odds ratio 0.96, 95% confidence interval 0.93–0.99, p = 0.02) and cardiac stress tests or nuclear imaging (7.8% versus 8.1%, odds ratio 0.96, 95% confidence interval 0.93–1.00, p = 0.04). The intervention did not significantly increase initiation of medications targeting cardiovascular risk factors or glycemia. Fewer patients in the intervention group were started on nitrates than in the control group (odds ratio 0.96, 95% confidence interval 0.92–1.00, p = 0.03).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Primary and secondary outcomes of the toolkit intervention.

https://doi.org/10.1371/journal.pmed.1001588.t002

Discussion

Printed educational materials in the form of a cardiovascular disease toolkit did not improve quality of care or outcomes in a population with diabetes. Some of the secondary outcomes were statistically significantly worse in the intervention group, although we cannot exclude that these represent chance findings due to multiple hypotheses testing. Thus, this quality improvement intervention not only failed to improve care, but if anything may have adversely affected some of the secondary outcomes.

The study has several strengths to highlight. It used a pragmatic randomized trial design to obtain a valid estimate of the effectiveness of the toolkit in real world clinical care, not just in an idealized efficacy trial setting. By randomizing in clusters at the practice level, the risk for contamination was minimized. The study included nearly 1 million patients—the entire population of patients with diabetes in Canada's most populous province—making it the largest ever randomized trial in diabetes. Using administrative data sources to evaluate outcomes ensured complete data collection with no loss to follow-up or missing data. The lack of detailed clinical data available in these data sources was supplemented with data collected directly from patient charts in the clinical data study. However, there are some possible limitations to note. First, the toolkit was developed without a specific quality improvement or educational theory to guide its content or delivery, which might otherwise have increased the likelihood of it leading to clinically important improvements in care [20]. Although the toolkit addressed clinician knowledge, other barriers to implementation of evidence-based care (such as time, patient adherence or comorbidity, or organization of health care delivery) were not addressed [21]. Second, the follow-up time of the study was short, which may have been insufficient time in which to see changes in clinical events. However, process measures and drug prescriptions could very easily be influenced within the short time frame of the study, and yet the intervention had no impact on these secondary outcomes as well. Third, although practices participating in the clinical data study were randomly selected, they may have been subject to volunteer bias as those who agreed to participate could be different from the large number who did not. However, it is unlikely that this would influence the validity of the study, as such bias would have affected both intervention and control practices equally. Finally, in the clinical data study, utilization of statins and of angiotensin converting enzyme inhibitors or angiotensin receptor blockers were both >85%, even in the control group. Thus, utilization may already have reached a ceiling, beyond which further improvements from an intervention would be unlikely.

The possibility that the intervention might have worsened care is unexpected. It is of course possible that the four statistically significant findings showing harm were chance findings due to multiple hypothesis testing, but notably there were no results in the opposite direction. Therefore, we feel it necessary to speculate on possible explanations for why this intervention could have worsened care, should these findings be real. Although its content was focused on cardiovascular disease, the toolkit was disseminated by the Canadian Diabetes Association and it was temporally linked with the newly released national diabetes practice guidelines. Hence, the intervention may have inadvertently focused the attention of physicians onto other aspects of diabetes care, and away from broad cardiovascular disease risk management, which might have been perceived of as a “mere” comorbidity of diabetes. Lack of attention to cardiovascular risk factor management would be detrimental to patient outcomes, especially since cardiovascular disease is the most common cause of death for diabetic patients [2],[22],[23]. These findings highlight the importance of monitoring for unintended consequences of quality improvement interventions [24].

The previous literature has demonstrated that the benefits of printed educational interventions are, at best, modest. A systematic review of methods to improve practice guideline adherence demonstrated an absolute improvement of 8% for educational materials [15]. A more recent Cochrane review found that printed educational materials led to a median absolute improvement in performance of only 2% [25]. A diabetes-specific review of quality improvement strategies to improve glycemic control showed that clinician education interventions had the lowest impact of any quality improvement approach, with a mean reduction in HbA1c of 0.16% [26]. Studies of printed materials specifically tied to clinical practice guidelines also showed modest benefits. A small English study randomized 42 family physicians to receive an algorithm for monitoring and treatment of hypertension of diabetic patients based on practice guidelines, but found no difference in blood pressure control between the intervention and control groups [27]. However, some processes of care were slightly improved: patients in the intervention group were prescribed higher doses of antihypertensive medications, and had more physician visits to monitor blood pressure. In a larger Canadian study, family physicians were randomized to receive a one-page summary of a three-year-old practice guideline on anti-anginal therapy from the local medical governing body [28]. No differences were noted in prescription of β-blockers, antiplatelet agents, or lipid-lowering drugs between groups in the 7,000 patients reviewed. We had hoped that the intervention in our study might have had greater impact because it was produced and disseminated by the CDA, an advocacy and professional organization with credibility among family physicians for diabetes messaging, and because the intervention was released in conjunction with a well-publicized update to national diabetes practice guidelines, increasing the salience of the intervention and its content. However, these differences of this intervention from those previously studied did not result in improved outcomes.

In summary, the cardiovascular disease management toolkit failed to reduce clinical events or improve quality of care for patients with diabetes. It may actually have led to worsening in some secondary outcomes, though this may represent a chance finding. Despite years of evidence that printed educational materials have, at best, only a modest impact on quality of care, they remain a very commonly used intervention for quality improvement. Therefore, even the remote potential for them to cause true harm is worrying, as the public health consequences could be significant. Quality improvement interventions are often developed by clinicians or policy makers in an unsystematic way. The results of this study highlight the need for a rigorous and scientifically based approach to the development, dissemination, and evaluation of quality improvement interventions.

Supporting Information

Acknowledgments

We are deeply indebted to the Canadian Diabetes Association, and in particular Ian Blumer, Alice Cheng, and Carolyn Gall Casey, for their support of this study. We also thank Alex Kopp, analyst at the Institute for Clinical Evaluative Sciences, for his assistance with the randomization of physician practices, and Donald Redelmeier for his comments on an earlier draft of this manuscript.