 Steve Montague "... performance information should contain clear and concrete
performance expectations with a focus on outcome-results." "We are changing what we measure - we are moving from monitoring
inputs and tracking outcomes to focusing on the outputs required to
achieve the outcomes desired by Government." "The [Government Performance and] Review Act seeks to improve the
management of federal programs by shifting the focus of decision making
from staffing and activity levels to the results of federal
programs." So just where should public managers focus? From the quotes above, you might think that Canada, Australia, and the United States each pay attention to different aspects of public enterprise and program performance. Add into the mix the requirements for implementation of accrual-based accounting, "Government-Wide Indicators" and a "balanced scorecard"(1) and one can rapidly see a public manager's vision getting distinctly out-of-focus. Is it any wonder that managers sign up in droves for training in performance measurement? Unfortunately, they often take courses only to hear general platitudes and self-evident truisms from (so-called) experts. In many instances, the 'cases' presented provide only a vague relevance to their own situations -- leaving participants just as confused when they leave as when they came. Joseph Wholey, currently of the US General Accounting Office and long-time program evaluator, sees the problem as one of defining the term 'performance'. "Performance" is not an objective reality out there waiting to be measured and evaluated. "Performance" is socially constructed reality (Berger, P.L., and Luckmann, T. The Social Construction of Reality. Garden City, N.Y.: Doubleday, 1966). "Performance" exists in people's hearts and minds if it exists anywhere at all. "Performance" may include resource inputs; in particular, dollars and staff time. "Performance" may include outputs; products and services delivered to partners, clients, or the public. "Performance" may include intermediate outcomes such as client satisfaction, actions taken by other levels of government, or actions by those in the private sector (for example, to control pollution). "Performance" may include end outcomes or impacts such as changes in environmental quality or changes in health status. "Performance" may include unintended outcomes such as costs incurred by firms and individuals as they respond to environmental programs. "Performance" may include net impacts: what difference a program has made.(2) So performance can include any and all of these perspectives. What is a manager to do? In fact there is hope. A review of recent international guidance and practice suggests that international approaches have more in common than they might seem. The common ground is a logic model. The logic model is a diagram explaining the flow from inputs, activities, or processes to outputs, shorter-term and longer-term outcomes or impacts. The model helps analysts and managers alike to articulate the cause-effect theory of a program or service, as well as the roles and relationships implied by delivery. When distilled to its essence, a logic model should answer the questions WHY an initiative exists, WHAT short and intermediate term outcomes are expected to result from the initiative, WHO is reached, and HOW, through activities and outputs, the initiative is delivered. Canada's Office of the Auditor General offers one version of the logic model in its 1997 Report of the Auditor General of Canada as follows in Figure 1 below: Figure 1
Australian guidance in the 1998 document named, "How to Measure Outputs" also advocates a logic model approach. While starting with outputs and their attributes of price, timeliness, and quantity, the approach also refers to 'quality' and 'contribution to outcome' which provide for a logical connection between outputs and outcomes. See Figure 2. Figure 2: Identifying Output 101 - Community Housing(4)
Recent iterations of the logic model have included the concept of 'reach' or the users and stakeholders involved in the delivery of a program, service, or initiative. Along with the concept of reach there has been the recognition that a logic flow will have a 'hierarchy' of outputs and especially outcomes. In Canada, the Management Committee on Industry Portfolio - Science and Technology Evaluation Sub-Committee has advocated a 'performance framework' approach. Conceptually, resources (staff and operating funds) are used to perform
activities and create outputs. This is HOW one goes about
achieving objectives. These activities and outputs reach a target user
group either directly or with the aid of co-delivery partners and
stakeholders. This is WHO is affected by the activities
and outputs. As a result of the activities and outputs, the target client
group behaves differently, and immediate outcomes occur. This is
WHAT happens. Over the longer term, the changed behavior
leads to more extensive and consequential impacts, which if the program is
performing well, can be causally linked to program objectives. This
responds to WHY. Sources of information to measure
program performance can then be identified, and performance indicators
developed in terms of these themes for any given program or organization.
See Figure 3.
Source: Steve Montague, The Three Rs of Performance: Core Concepts for Planning, Measurement and Management, Performance Management Network Inc., 1997, Ottawa, Ontario, Canada, p. 168. US evaluator Michael Patton, in a recent version of his seminal text, Utilization-Focused Evaluation: The New Century Text, draws on a logic model two decades old to illustrate both reach and a hierarchy of outcomes. Figure 4 below displays the approach. Figure 4
 Source: Adapted from Claude Bennett 1979. Taken from Michael Quinn Patton, Utilization-Focused Evaluation: The New Century Text, Edition 3, Thousand Oaks, California, 1997, p 235. Note hierarchy items 1 and 2 relate to HOW, item 3 relates to WHO, items 4 through 6 relate to WHAT, and item 7 relates to WHY. Finally, in their excellent 1998 guidebook, Specifying Outcomes and Outputs, the Australian Federal Public Service provides a number of examples of output-outcome hierarchies - even for the difficult area of policy. Figure 5 below shows one such chain for retirement policy advice. Figure 5: Outcomes Hierarchy for Retirement Policy Advice
 Source: Taken from Department of Finance and Administration, Specifying Outcomes and Outputs: Implementing the Commonwealth's Accrual-based Outcomes and Outputs Framework, Commonwealth of Australia 1998, p 82. [Note the description of both who was involved in the outcome as well as what outcome was to be achieved.] In summary, a careful review of international practice suggests that the underlying ethos of the Canadian, Australian, and American systems is not so different after all. The approaches boil down to articulating a logical flow of desired events. The models describe the use of resources in terms of activities and outputs flowing to reach targeted users and stakeholders, resulting in a hierarchy of expected outcomes. The hierarchy starts with reactions and behavior changes in individuals and groups, leading to broader benefits and impacts at the community and societal level. Whether it is top-to-bottom, bottom-to-top or left-to-right--the logic flow is the same. Upon reflection, it appears that the admonitions to 'focus' on outputs or outcomes or 'key results' are misguided. The point is really to focus on the relationships, flows and trade-offs among these different components of performance. Public administrators across Western democracies will only improve management(5) once they take in the bigger picture-- guided by some form of logic model-- before setting out to focus on specific measurement areas. Under the surface, Canadian, Australian and U.S. approaches all promote the understanding of program logic before measurement. The recognition of logic before measurement will be a key element in the future success of performance management initiatives in these countries. Endnotes 1. The Working Group on Implementation of the Government Performance and Results Act recently described the concept as follows: The Scorecard looks at performance from four perspectives rather than from a single, traditional bottom line measure. Kaplan and Norton recognized that performance is not one-dimensional, that there are multiple contributors that were categorized as:
2. Joseph S. Wholey, "Clarifying Goals, Reporting Results," Progress and Future Directions in Evaluation, Jossey-Bass Publishers, San Francisco, Number 76, Winter 1997, p 97. 3. "Chapter 5: Reporting Performance in the Expenditure Management System," The Report of the Auditor General of Canada, April 1997, Exhibit 5.1. 4. How to Measure Outputs, Department of Finance and Administration, Commonwealth of Australia, 1998, p 4. 5. 'Management' has been defined by Drucker as the allocation of resources to achieve results. Thus understanding management requires an understanding of relationships and cause-effect flows. ©1999 Performance Management Network Inc. | |||||||||||||||||||||||||||||||||||||||||||||||||||||
