Citation: Byass P (2010) The Imperfect World of Global Health Estimates. PLoS Med 7(11): e1001006. doi:10.1371/journal.pmed.1001006
Published: November 30, 2010
Copyright: © 2010 Peter Byass. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This work was undertaken within the Umeå Centre for Global Health Research, with support from FAS, the Swedish Council for Working Life and Social Research (grant no. 2006-1512). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The author is a member of the Editorial board of PLoS Medicine, deputy editor of Global Health Action, a member of the editorial board of Population Health Metrics (a journal based at the Institute for Health Metrics and Evaluation), and a member of the UN Maternal Mortality Estimation Inter-Agency Technical Advisory Group, hosted by WHO, Geneva. He has visited the Institute for Health Metrics and Evaluation, Seattle, at their expense.
Provenance: Commissioned; not externally peer reviewed. The idea for a commissioned cluster of articles on global health estimates came from Ties Boerma and Colin Mathers at the World Health Organization.
- Global estimates of population health are currently needed because of the shortage of adequate quality population-based data. These estimates are complex, because they need to combine relatively complete data from industrialised countries with sometimes very scanty data from developing countries.
- Two major sources of such estimates are agencies within the United Nations system and (mostly northern) academic institutions, which differ in their approaches.
- Appropriate strategies for ensuring the robustness and transparency of estimates are very important. Long-term strategies must be geared towards improving the quantity and quality of bottom-up data, rather than developing ever more complex estimation methods.
- Ultimately, the world must be able to measure population health from reliable individual data rather than relying on estimates.
How Did We Get Here?
Global measurement has long been contentious. Three hundred years ago, the exact size and shape of the world were a matter of scientific controversy and estimation. New ways of measurement were developed, and by the early 18th century that uncertainty ceased . One hundred years ago, Sweden started to account for all deaths in its population, by cause, age, and sex , and annual summary tables based on individual death registration on a national basis have been published ever since. Today, despite increasing globalisation, there is still no similar universal individual registration of vital events in many countries, and consequently we find ourselves in an era of global estimates of population health. These global estimates are complex amalgams of detailed national measures from countries with universal registration and the best available data—which are often scanty—from other settings. Hopefully the long-term aim of the global health community is to move beyond this era of estimates, towards the relative certainty of accounting for individual health globally. Meanwhile, the purpose of this article is to explore issues and tensions around these currently necessary global estimates.
As discussed previously , fundamental links between poverty and data availability mean that global estimates are not trivial to construct. Unlike global estimates of geophysical parameters, which have been revolutionised by remote sensing and satellite surveillance , population health estimates must continue to rely in some way on grass-roots data about individual people. But lack of standards and different approaches are causing contention and confusion.
Why Is There Debate about Current Global Estimates?
Current global estimates mainly come from one of two sources: (1) the United Nations (UN) and its specialised agencies (such as the World Health Organization and the United Nations Children's Fund [UNICEF]) or (2) northern academic institutions. There are important underlying differences between the estimates from these sources, as shown in Figure 1. Why southern academic institutions are not more engaged in the process of developing global estimates, given that the major uncertainties within most estimates centre on southern data, is a further question of interest.
Figure 1. Approaches to global estimates by UN agencies and academic groups.doi:10.1371/journal.pmed.1001006.g001
The UN Estimates
To contextualise estimates originating from the UN, it is necessary to understand the nature of the UN system. The UN is fundamentally constructed as a member organisation for most of the world's nation states, and member countries are therefore its constituency. This structure has significant practical implications for the construction of global estimates. The UN system has a direct entrée to member countries at the governmental level and, although unable to exercise compulsion, may be able to access otherwise unpublished national data. After UN experts have processed and modelled available data, member countries are commonly consulted on estimates before publication. UN estimates can then be published under the UN imprimatur , although often staff and external advisors also prepare articles for publication in peer-reviewed journals . This approach has been criticised , perhaps unfairly.
Estimates from Academic Institutions
Academic institutions, by contrast, have to rely largely on public-domain data and previously published studies, often leading to the use of mixed methods involving direct and meta-analyses –. As independent institutions, they have no obligation to consult externally, and countries' first sight of such estimates may well be on publication. While this may lead to interesting post-publication debate and controversy, there can be a problem in that a published paper is still likely to be taken as the definitive version, disregarding any implications of subsequent interactions. Such estimates are, however, normally disseminated via peer-reviewed journals, which should be assumed to assure a paper's quality, as with any other scientific output. However, questions have been raised as to how to effectively peer-review papers describing complex estimates . An interesting development might be the inclusion of a country consultation phase before publication of estimates from academic institutions, making public the issues that thereby arise.
Two Tales of Maternal Mortality
Some of the recent debate and contention around the source of global estimates emerged during 2010 when two separate estimates of global maternal mortality were published. One set of estimates originated from the Institute for Health Metrics and Evaluation (IHME) , and the other was the latest update of the UN inter-agency estimates of maternal mortality . It is impossible to conclude which is the more “correct” set of estimates, because if that were measurable as a matter of fact, the estimates would be redundant anyway. However, it is interesting to compare some key issues in the approaches and conclusions of these different estimates. Headline figures were very similar and well within each other's uncertainty intervals (342,900 and 358,000 maternal deaths worldwide and maternal mortality ratios of 251 and 260 per 100,000, respectively, for 2008). However, there may be important differences at the country level or for specific causes, depending on the data and methods used.
On the critical issue of estimating how many deaths among pregnant women might be associated with HIV/AIDS, very different approaches were used, with widely different conclusions (61,400 and 42,000 deaths, respectively, for 2008). Bottom-up data on deaths associated with both pregnancy and HIV/AIDS—particularly from Africa—are very scanty because, even where verbal autopsies are performed to ascertain cause of death, very often only a single cause is recorded . It is not surprising that where data are critically lacking, different approaches to estimation yield different results. The overlap between pregnancy and HIV/AIDS is an area in which urgent reforms in methods and procedures are needed in order to provide appropriate multiple-cause data, rather than developing more complex estimation methods.
How Can the Robustness and Transparency of Global Estimates Be Ensured?
Because estimates are estimates, and not measurements, it is relatively easy for the proponents of particular estimates to claim high quality and reliability, and for the detractors to question the same, with little scope for objective adjudication. The key factor for robustness is the extent of available data, linked, of course, to sound methods. Transparency involves using all available data of quality and relevance, while usually imposing some explicit framework of rules as to what constitutes usable data. Methodological strategies then also need to be set out in a fully transparent manner. Any epidemiological interpretation depends on an understanding of the provenance and sampling basis of the underlying data, which leads to estimates of uncertainty. However, some of the data used in global estimates are so many stages removed from their origin that associated estimates of uncertainty themselves become very complex and hard to understand. Unfortunately, the gaps and uncertainties around data in many instances drive researchers to ever-increasing levels of methodological complexity in attempts to compensate, and transparency may be obscured by these complexities. This can rapidly lead to an “Emperor's New Clothes” syndrome in which only the cognoscenti truly understand the underlying basis of complex estimates, while the vast majority may be reluctant to admit that the detail is beyond their comprehension.
Where Do We Need to Go from Here?
The undeniable long-term aim must be to foster more and more production of high-quality and complete population data from locations that are as yet devoid of usable material. This implies a bottom-up philosophy emphasising the need to connect with population data at source . If a gradual process of filling in such gaps in global data is realistic, then one would also hope that in parallel with increasing completeness of data there will be reductions in the complexity of appropriate estimation methods. This would lead towards the ideal situation, in which global estimates would become a thing of the past as the world's population actually became measurable.
One potential obstacle to this process is that the world may become so used to the concept of global estimates that insufficient effort will be invested in improving bottom-up data. Even now, there are signs that journal editors can find complex global estimates more enticing to publish than relatively detailed epidemiological descriptions of within-country data. For example, four recent sets of global estimates published in The Lancet have been “fast-tracked” for reasons that are not entirely clear ,,,; since they actually present long-term estimates that have no urgent health care implications, perhaps this is symptomatic of the self-importance that such estimates seem to be acquiring. It is very important that the concept of global estimates does not acquire an undeserved supremacy over conventional analyses of detailed data, if we want to nurture a culture of encouraging and promoting good-quality data at source, with local analyses and interpretation . Strategies to bring all countries to a common standard of high-quality and sustainable health information systems need to be prioritised .
For the foreseeable future, the global health community is likely to be locked in a love–hate relationship with global estimates of population health. We wish we did not need these estimates and could instead rely on objective assessments based on quality data. But we know we do not yet have enough of those data, and so for now we do need estimates as a resource on which to base health policy and planning decisions. However, we should not expect to wait three hundred years, nor even one hundred years, before superseding global estimates with global measurements.
Global Health Estimates: Where Do We Go from Here?
- Work towards better availability and quality of data
- Enhance country capacity to develop, analyse, and interpret local data
- Develop common data standards
- Focus on better data rather than more complex estimation methods
- Improve robustness of data and methodologies used in estimates
- Manage a gradual transition from estimates to measurements
ICMJE criteria for authorship read and met: PB. Agree with the manuscript's results and conclusions: PB. Wrote the paper: PB.
- 1. Westfall RS (1983) Never at rest: a biography of Isaac Newton. Cambridge: Cambridge University Press.
- 2. Kungliga Statistika Centralbyrån (1915) Dödorsacker År 1911. Stockholm: Kungliga Statistika Centralbyrån.
- 3. Byass P (2009) The unequal world of health data. PLoS Med 6: e1000155. doi:10.1371/journal.pmed.1000155.
- 4. van Donkelaar A, Martin RV, Brauer M, Kahn R, Levy R, et al. (2010) Global estimates of ambient fine particulate matter concentrations from satellite-based aerosol optical depth: development and application. Environ Health Perspect 118: 847–855.
- 5. United Nations Children's Fund, World Health Organization, World Bank, United Nations DESA/Population Division (2010) Levels and trends in child mortality—report 2010. Available: http://www.childinfo.org/files/Child_Mortality_Report_2010.pdf. Accessed 17 September 2010.
- 6. You D, Jones G, Hill K, Wardlaw T, Chopra M (2010) Levels and trends in child mortality, 1990–2009. Lancet 376: 931–933.
- 7. (2010) Soul searching at the UN. Lancet 376: 1026.
- 8. Black RE, Cousens S, Johnson HL, Lawn JE, Rudan I, et al. (2010) Global, regional, and national causes of child mortality in 2008: a systematic analysis. Lancet 375: 1969–1987.
- 9. Dellicour S, Tatem AJ, Guerra CA, Snow RW, ter Kuile FO (2010) Quantifying the number of pregnancies at risk of malaria in 2007: a demographic study. PLoS Med 7: e1000221. doi:10.1371/journal.pmed.1000221.
- 10. Rajaratnam JK, Marcus JR, Flaxman AD, Wang H, Levin-Rector A, et al. (2010) Neonatal, postneonatal, childhood, and under-5 mortality for 187 countries, 1970–2010: a systematic analysis of progress towards Millennium Development Goal 4. Lancet 375: 1988–2008.
- 11. Graham WJ, Braunholtz DA, Campbell OM (2010) New modelled estimates of maternal mortality. Lancet 375: 1963.
- 12. Hogan MC, Foreman KJ, Naghavi M, Ahn SY, Wang M, et al. (2010) Maternal mortality for 181 countries, 1980–2008: a systematic analysis of progress towards Millennium Development Goal 5. Lancet 375: 1609–1623.
- 13. World Health Organization, United Nations Children's Fund, United Nations Population Fund, World Bank (2010) Trends in maternal mortality: 1990 to 2008. Geneva: World Health Organization. Available: http://whqlibdoc.who.int/publications/2010/9789241500265_eng.pdf. Accessed 20 September 2010.
- 14. Byass P, Kahn K, Fottrell E, Collinson MA, Tollman SM (2010) Moving from data on deaths to public health policy in Agincourt, South Africa: approaches to analysing and understanding verbal autopsy findings. PLoS Med 7: e1000325. doi:10.1371/journal.pmed.1000325.
- 15. Boerma T, AbouZahr C (2005) Monitoring global health: bottom up approach is more likely to be successful. BMJ 330: 195–196.
- 16. Rajaratnam JK, Marcus JR, Levin-Rector A, Chalupka AN, Wang H, et al. (2010) Worldwide mortality in men and women aged 15–59 years from 1970 to 2010: a systematic analysis. Lancet 375: 1704–1720.
- 17. Ravishankar N, Gubbins P, Cooley RJ, Leach-Kemon K, Michaud CM, et al. (2009) Financing of global health: tracking development assistance for health from 1990 to 2007. Lancet 373: 2113–2124.
- 18. Bangha M, Diagne A, Bawah A, Sankoh O (2010) Monitoring the millennium development goals: the potential role of the INDEPTH Network. Glob Health Action 3: 5517. doi:10.3402/gha.v3i0.5517.
- 19. Chan M, Kazatchkine M, Lob-Levyt J, Obaid T, Schweizer J, et al. (2010) Meeting the demand for results and accountability: a call for action on health data from eight global health agencies. PLoS Med 7: e1000223. doi:10.1371/journal.pmed.1000223.