Ontario citizens were treated to a bonus in their Globe and Mail newspaper in July. The Government of Ontario included an insert entitled, Hospital Report 2001. The report described a "balanced scorecard" derived from a joint initiative of the Ontario Hospital Association, the Government of Ontario, the University of Toronto and the Canadian Institute for Health Information.
The Honorable Tony Clement, Minister of Health, and David McKinnon, President of the Ontario Hospital Association, included notes of praise for the scorecard, remarking that it was the result of a commitment made in 1997.
Note that the term 'balanced scorecard' is derived from a series of articles and a book now turned into a movement in management circles of the same name (for more information see Workforce Performance, August 1999 "Review of the Balanced Scorecard", http://www.opm.gov/perform/articles/1999/aug99-3.htm). The Ontario scorecard report lists four distinct perspectives against which all hospitals in Ontario are given a score. The four perspectives are:
1 patient satisfaction - This perspective includes what patients say about their stay in hospital, including opinions about the overall quality of care, the care received from doctors, nurses, and other health professionals, and finally, the housekeeping and food services in the hospital.
2 patient care - This category scored how patients fared during and after their hospital stay. For example, it rates whether they had access to specific medical technologies, how long patients stayed in hospital, how often patients experienced complications of care in hospital, and re-admission rates due to related conditions.
3 hospital finances - This category included what resources Ontario hospitals had and how they were using them. This included whether hospitals were running surpluses or deficits; how much hospitals spent on computer systems, x-ray machines, and other equipment; and how much staff time was spent on patient care, as opposed to other duties.
4 keeping pace with change - This category rated whether hospitals partnered with community agencies and updated their practices. For example, it noted how hospitals worked with other health care organizations, whether hospitals were using information technology to support patient care, and what patients thought about the coordination of the care while they were in hospital and after they left.
At first glance, these categories seem very sensible. Upon further inspection however, one starts to see some gaps. In our opinion, there are two key gaps which need to be addressed in the current Ontario hospital scorecard:
1. Reach - The reach of Ontario hospitals is important. For our purposes, reach can be defined as the groups and communities served by hospitals both collectively and individually. (For more on the concept of reach see Montague, S., 1997 "The Three Rs of Performance", http://www.pmn.net/scorecard/index.html). The detailed report, put together for the scorecard, recognizes the importance of the different communities served by hospitals by weighting patient satisfaction scores according to age and gender (Hospital Report 2001, p. 13). In other words, since persons of different age and gender groups tended to rate hospital services higher or lower than the average, the overall hospital score was weighted or "risk-adjusted", using multiple regression analysis, by the age and gender make-up of its patients. One could argue that age and gender would not be the only demographic determinants of systematic differences in satisfaction ratings, (e.g., what about socio-economic status?, ethnicity?, linguistic profile?) - but such a technical discussion is not the most important point here.
The important point is that reach is not explicitly shown for the hospitals in the scorecard. The specific circumstances of the people served by hospitals are not dealt with in the insert scorecard (nor is it apparently dealt with explicitly in the background reports, although the data available for the reports does include demographic reach data).
In our opinion, the exclusion of "reach" is a great oversight. The experience of professionals working with Performance Management Network has been that the nature of groups reached has a profound explanatory impact on both the results (i.e., user satisfaction levels and care outcomes in this case) and on the use of resources. For example, the use of resources, and as importantly, the predictability of resource use, might be directly linked to the nature of maladies treated. (Consider a hospital in an urban area treating the effects of violent crime, versus a hospital in a suburb with an aging demographic treating primarily lifestyle and chronic disease.)
Thus many of the indicators under each of the scorecard areas could logically be explained by differences in reach. Yet the consumers of this report, Ontario taxpayers, are not explicitly provided this information in a readily available way.
2. Outcome Results - According to our reading of the report and its background reports, the majority of the results indicators used by the OHA scorecard relate to patient satisfaction levels, internal hospital improvements, and certain aspects of hospital care - only some of which are real outcomes. We would define outcomes as results which are influenced by delivery agent (i.e. hospital) processes, but are not controlled by them (See Mayne, J., 2001, "Addressing Attribution Through Contribution Analysis: Using Performance Measures Sensibly" in The Canadian Journal of Evaluation, Volume 16, Number 1, pp. 1-24). So, for example, the use of specific medical technologies, or decisions around how long patients stay in hospital for certain procedures would be defined as process decisions of the hospital itself. These are not care "outcomes". The only apparent care outcomes in the current Ontario scorecard by our definition would relate to the re-admission rate of patients due to a related health problem, and complication rates for medical and surgical patients.
The problem here is that we have process outputs masquerading as outcomes, and no clear articulation of reach. This can be very dangerous. Imagine the case where hospitals are now going to be compensated on their scorecard results. Hospitals are therefore encouraged to shorten the patient stays, and perhaps even more insidiously, to adjust reach into the community. Note that Ontario already has a phenomenon where patients are seeking help elsewhere (for example in other provinces and in the United States) for the treatment of certain diseases and afflictions. Without a reach measure in the scorecard which addresses the proportions of the needs served by Ontario hospitals, reach will almost surely be 'gamed' to achieve the desired result. Predictable low-cost treatments to already healthy patients will be favored over unexpected and costly procedures. Those likely to complain or to have 'problems' may be treated through home services, or encouraged to go out of the province.
A balanced scorecard which includes reach and some additional results could help. Figure 1 below displays the current Ontario hospital measures against a 3 Rs scorecard. (For more on the Three Rs scorecard see http://www.pmn.net/scorecard/index.html, and http://www.pmn.net/contributions/score.htm ). Note the bias in the current Ontario scorecard towards resource efficiency and satisfaction results.
Figure 1 - Three Rs Core Components Used to Sort the Ontario Hospitals Balanced Scorecard Report
Figure 2 complements these indicators with possible additional measures which might achieve better balance for individual hospital ratings and for the hospital care picture in Ontario as a whole.
Figure 2 - Additional Measures and the Three Rs Approach
The suggested additional measures in Figure 2 would not be easy to collect in every case, nor would they be beyond controversy since they would likely require a good deal of qualitative interpretation. We believe, however, that such an alternative should be pursued in order to provide a truly balanced and citizen- focused depiction of hospital care services in Ontario.
For more information or to comment please contact Steve Montague or Lisa Fairweather.
copyright PMN 2001