Patient and Community Domain

Overall, the patient and community domain illustrated significant improvement from baseline to 2008 (65.3%–84.5% p<0.0001). The substantial increases in this domain were primarily driven by the improvements in the engagement of village councils (shura-e-sehie) in the management oversight of health facilities measured by meeting records at the health facilities (<35% in 2004 to 95% in 2008). Client satisfaction and perception of quality, critical determinants to service utilization, were rated high (>70%) in all survey years. The annual increase for patient perception of quality and village council activities, estimated by GEE was significant (p = 0.02 and p<0.0001) (Table 3), although the size of the actual increases in patient-perceived quality was minimal, over the years.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. BSC indicators with statistically significant increments in mean scores between 2004 and 2008 using GEE robust estimates.

https://doi.org/10.1371/journal.pmed.1001066.t003

Staff Perspectives and Satisfaction Domain

Provider satisfaction included multiple measures of various health system attributes and support factors including training, professional development opportunities, availability of essential equipment and medicines according to BPHS guidelines, and working relationships with various stakeholders. There was a slight, but significant, increase in this index indicator (+6%) to about 70% in 2008. In 2008, median scores for providers reporting timely salary payments increased by 20% from baseline levels. Financial incentives and regular payments are important contributory factors for provider motivation and retention, and the finding suggests improved management of implementing agencies.

Capacity for Service Provision

Compared with other domains, the most dramatic improvements were evident in the capacity for service provision domain (47.4–76.4, p<0.0001). All system preparedness factors for service provision; essential medicines, equipment, clinical guidelines, laboratory functionality, health provider adequacy, skill mix, and provider knowledge and training showed significant improvements since 2004. The evidence illustrates the investment efforts to adhere to BPHS standards to ensure capacity for service provision, a critical prerequisite for ensuring optimal quality of care and health outcomes. The annual progressive increase estimated by GEE was significant for all indicators in this domain (Table 3).

Service Provision

Significant improvements were evident for adherence to clinical standards for patient history and physical exam as well as counseling, although the latter was still at suboptimal levels in 2008 (48% national median score). There was a progressive increase in average new patient visits per month, which reached peak levels (median score of 85) in 2008. Scores were significantly higher for provision of both antenatal (+36%, p<0.0001) and delivery care services (+46%, p<0.0001) from baseline levels. Despite these improvements, average patient consultation time diminished over the years, which may have resulted from higher levels of service utilization following the removal of user fees. Regression modeling demonstrated a strong statistical association between the capacity for service provision (coefficient 0.45, p<0.0001), with quality of service provision in 2008. Applying the GEE modeling, there was a significant increase (p<0.0001) in adherence to standards for patient history taking and patient counseling during the 5-y period.

Financial Systems

Modest improvements were evident in the existence of patient user fee guidelines (+6.5% by 2007). All sampled facilities provided user fee exemptions for poor patients, raising the national median score to 100 in 2007, a 17.5% (p<0.0001) increase from baseline levels. Discussions are underway to reconfigure this domain to include other characteristics of health facility and hospital financial management.

Overall Vision for Pro-poor and Pro-female Services

One of the primary goals of the BPHS was to ensure equity of service delivery to the poor and optimize utilization of females for outpatient services. A concentration index of 50 would indicate no differences in service utilization, and a higher index would indicate higher utilization of females or poorer patients. This objective was partially achieved, as female patients increasingly constituted a higher proportion of new outpatient visits (+4.9%, p<0.0001), but there was minimal improvement in scores for equity of utilization or patient satisfaction as measured through concentration indices of users over the years.

Leadership and Transparency

Lack of leadership support has often been cited as a barrier to successful scorecard implementation [10]. However, in the Afghanistan context, the support of the executive leadership in policy and planning was a key enabling factor for the significant achievements in national performance scores. The BSC served as a strategic management tool to ensure capacity, identify factors contributing to performance deficiencies, and determine policy and resource innovations to promote good governance [15]. Despite some initial skepticism and reluctance to adopt the tool, over the years there have been several “champions” who leveraged the interest and support of other investors in the health sector to sustain the application of the BSC. In a postconflict environment, heavily reliant on external donor assistance, the BSC has been instrumental in enhancing transparency in the decision-making process, creating a culture of accountability by policy leaders to manage performance-based contracts. Improved performance trends helped demonstrate positive results to the public to rebuild trust in the government health care system [13],[18].

Evidence of Improved Health Service Capacity and Delivery

As a performance assessment tool, the BSC has provided a comprehensive framework of multiple service delivery and system elements, illustrating impressive improvements to achieve the BPHS standards for optimal service provision. The trends provide the evidence of investments to enhance health infrastructure, staffing and resource capacity, and subsequent impact on service provision. The customer domains of patient and provider satisfaction, and the financial domain (through 2007) also showed significant improvements during the 5 y. However, efforts must be made to motivate providers to improve counseling quality by providing appropriate instructions to patients on home care, conditions for immediate return to facility, and correct administration of prescriptions. Time spent on patient consultation and counseling, a significant predictor of quality of care [18],[21], showed a decreasing trend over the years with levels returning to baseline scores in 2008. Without additional health system level resource investments complemented by patient triaging, improving counseling quality will continue to be a challenge.

We believe that the apparent dramatic improvements for some indicators and minimal changes in others over the 5-y period have three plausible reasons: (1) The median performance score for service capacity and service provision domains was at much lower levels in 2004, (47 for service capacity and 40 for service provision) than the scores for overall patient satisfaction (>80 in 2004), patient perception of quality (>75% in 2004). Apparently the improvements were more dramatic for domains that had low scores in 2004, than those that were at higher levels. (2) The presentation of the scorecard illustrating poorly performing domains and the ongoing efforts of donor priorities to establish health infrastructure and resource capacity resulted in additional performance improvement strategies to improve the scores for these indicators. (3) The possibility of a courtesy bias resulting in high levels of patient satisfaction over the years has also been hypothesized in a later section.

BSC Facilitated Performance Benchmarking

In addition to enhancing improvements in service capacity and delivery, the BSC facilitated historical and provincial benchmarking and determining the effectiveness of various contracting mechanisms [20]. Benchmarking performance has been demonstrated to be an effective strategy, to foster healthy competition and create a culture of evidence to determine factors of success or failure, particularly in decentralized health systems, and is still considered a relevant strategy for the Afghan context, despite the provincial variations in the health care environment particularly security and geographic access [22],[23].

Remaining Challenges

Despite the progress on health system performance, Afghanistan continues to face challenges to ensure equitable service delivery. Service utilization for female patients is still dependent on the availability of female providers. Though the acceleration of the midwifery training programs has tried to meet the increasing demand for female providers, the data from the national assessments does not provide information on the national health workforce because of the sampling strategy employed. The human resource database indicated that female providers currently constitute only 24% of the workforce, with considerable variation across provinces. The overall proportion of health provider shortage estimated at 39% (based on BPHS standards) is still a severe deficit, compounded by worsening security [1]. Poor provider motivation can lead to suboptimal quality due to nonadherence to clinical guidelines, harmful practices, and poor provider–patient interactions [24]. Results-based financing mechanisms, similar to those implemented in Rwanda [25], have been introduced recently to incentivize providers to enhance motivation and performance. Though external assistance and public health spending has increased substantially in the last few years, better targeting of services is critical to ensure equity.

Limitations in the Design and Execution of the BSC

It is also important to highlight some of the limitations in scorecard design and execution for policy makers to debate the utility of the Afghanistan scorecard tool to measure health service delivery. Aside from the classic Hawthorne bias of performance improvements due to the fact that providers are aware of being observed, the quality of care assessments did not have gold standards by which the accuracy of diagnosis or treatment could be ascertained. High levels of patient satisfaction in this context may also be attributed to a courtesy bias, as patients with limited physical and financial access to services, may fear the consequences of negative feedback, resulting in termination of services [26]. Leveraging the support of village health councils to ensure community feedback may alleviate some of this fear and enhance community trust of the health system.

Another limitation to the scorecard design is the lack of measures to assess the impact of investments at the population level, as currently the assessments are organized around health facilities and the users of services. Although the BSC was not designed to measure important health impact like mortality rates or to evaluate the causal relationships that link to such measures of population health, future strategies will try to link household surveys to the facility-based assessments to examine causal associations between facility interventions and observed changes in population-based outcomes. These could include effectiveness and efficiency measures to demonstrate the overall performance of the health system.

To ensure equitable access to services, the establishment of subcenters, mobile clinics, and health posts has accelerated in recent years with more than 22,000 community health workers trained and deployed nationally to provide health promotion and first level of care in rural communities. Efforts are underway to reconfigure the scorecards to integrate community provider performance measures and assess investments made to the base of the health system pyramid to demonstrate results as communities and patients engage to become coproducers of health [27],[28].

A commonly cited barrier of scorecard implementation is the progression from a measurement tool to a performance management tool for continuous quality improvement to ensure social accountability and balance other financial considerations [29]. It is hypothesized that these quality improvement strategies may offset some of the costs of BSC reconfigurations necessitated by the changing health care environment [9]. The national quality assurance committee in Afghanistan, has been instrumental, in prioritizing performance deficiencies identified from routinely obtained health management information and supervision systems, as well as the NHSPA, in order to identify and address management constraints [30]. We recognize the importance of rigorous survey designs to assign cause and effect to improvements instituted as it provides a record of strategic inflection points when policies changes and resources were appropriated to improve systems and performance. Deficient performance such as inadequate provider knowledge or stock out resulted in improved efforts for refresher training and ensuring adequate stocks of essential drugs and review of NGO performance when issuing new contracts [18].

Cascading Scorecard Results to Frontline Providers

Translating the scorecard strategy into easily comprehensible performance measures to demystify the scorecard at the peripheral levels is important for the frontline providers to comprehend the value of the measurement strategy [11]. In Afghanistan, dissemination of scorecard results was primarily targeted to the policy leaders and contracting and implementing agencies responsible for the oversight of the health system at the national and provincial level, but there was minimal evidence on its use at the facility level. The importance of packaging and cascading the scorecard results to the frontline constituents cannot be undermined as this is critical for creating autonomy in the peripheral units to address context specific performance deficiencies through rapid quality improvement cycle efforts [9],[11].

Impediments in the Health Care Environment

The lack of explicit systems and policies, minimal infrastructure, high corruption levels, low literacy rates, and resistance to measurement, identified as barriers for successful scorecard execution, are also inherent in Afghanistan's health care environment [31]. However, the major challenge will be to sustain the momentum of BSC performance assessments in the context of worsening security, and therefore the long-term effectiveness of the current scorecard system warrants further scrutiny, particularly if donor assistance diminishes in the future. Further analysis applying multivariate statistical modeling controlling for confounders will be required to determine the major contributors for improving health service delivery in this context [32].

Future Scorecard Reconfigurations

Transitions in the health system landscape of leadership and political priorities, changes in the health system architecture of financing and resource structures, guidelines and policies, epidemiological transitions, and security constraints will necessitate periodic reconfigurations to ensure the scorecard's relevance and effectiveness. The Ministry and its stakeholders are considering the integration of indicators to determine effectiveness of financial management operations and performance to replace the user fee indicators formerly in the financial management domain. As all provinces have achieved the 2004 performance benchmarks, new targets will be required in the future, as well as the inclusion of indicators for emerging health priorities to modify the scorecard.

The costs for conducting the annual national performance assessment to generate the scorecards are estimated at 2.5% of the overall cost of delivering the BPHS. As the government becomes more self-reliant, the scorecard may be scaled down to a few key performance indicators that can be derived from data obtained from health information systems supplemented by routine rapid facility assessments to minimize survey costs.

Enhanced Organizational Learning Promoting a Culture of Measurement

Few national performance measures were available when the BSC was first initiated for the primary health care sector. As the health system matured, several parallel measurement systems were standardized using the BSC prototype, including scorecards to determine performance of the essential package of hospital services and more recently the ongoing routine assessments for the health management information systems. The scorecard enables a single organizational measure, and reduces the burden of managing fragmented performance assessments focused on few health service delivery elements. All surveys and measures have been conducted at the national level to include all secure provinces addressing vertical scale-up and assessment mechanisms for inclusion of insecure provinces are in consideration.

Conclusion

Emerging from decades of war and continued insecurity, Afghanistan has successfully pioneered the integration of the BSC at the national and provincial levels, to improve the delivery of basic health services. The BSC provides a dashboard of indicators integrating the building blocks of health systems, promoting effective governance and leadership, facilitating good stewardship and managerial autonomy for timely decisions on policy and service delivery [33]. Despite the promising results so far, the successful execution of the BSC will depend on its adaptive ability and sustained efforts of the MOPH leadership to accommodate dynamic and complex changes in the health care environment. As the global momentum for health systems strengthening accelerates, the balanced scorecard offers a promising measure of comprehensive health system performance to examine the effectiveness of strategies and innovations executed, particularly in resource constrained environments.