Performance Measurement, Management and Reporting for S&T Organizations - An Overview(*)George G. Teather
Steve Montague
Abstract
An integrated approach to performance measurement, management, and reporting is presented
which builds on the well-known logic diagram approach of evaluation theory. The addition of
explicit consideration of reach, defined as clients, co-delivery partners, and stakeholders, supports
a more holistic, balanced approach to the concept of performance, which has found acceptance
among S&T performers and central agencies in Canada and the U.S. The description of the
"performance framework approach" is supported by rationale for its use at both operational and
strategic levels of S&T management. Also included are discussions of recent complementary work
and examples of successful use of the approach.
Introduction
There has been a burgeoning interest in the performance of government programs in recent years.
This interest comes from several sources, including citizens' concerns about value received for
their tax dollars and managers' need to better understand program performance in order to make
strategic and operational decisions in an era of declining resources and government expenditure
reductions. The Government Performance and Results Act (GPRA) in the U.S. and similar
initiatives in other countries reflect this pressure. In Canada, Science and Technology (S&T) has
been singled out for improved performance measurement. A major year long review of federal
S&T, which began in mid 1994, involved both external consultations with the public, business,
universities, and other stakeholders as well as an internal review of S&T policies and programs in
all science-based departments and agencies. The government's response is contained in Science
and Technology for the New Century - A Federal Strategy (Canada 1996), which includes a
commitment to the assessment of federal S&T performance on a regular basis, exemplified in the
following quotation:
"Each department and agency will set S&T targets and objectives, establish . . .
performance indicators . . ."
In order to respond to these challenges, the S&T community in these and other countries is under
considerable pressure to develop mechanisms to determine and measure performance in a
credible, logical manner which will be understood by the government and other key stakeholders.
In recent years there have been numerous efforts to measure S&T performance with many
examples of good studies; unfortunately, they are interspersed with poor ones. Using an analogy
borrowed from the technology sphere, S&T performance measurement is still an emerging
capability, on the initial slope of the "S" curve, characterized by many competing initiatives with
varying degrees of quality and capability, each striving for acceptance and survival.
Drawing on more than ten years of experience in evaluation of government S&T organizations and
programs, the authors have developed an integrated approach to the consideration of S&T
performance which has found acceptance by S&T managers and government central agencies in
Canada and the U.S. In this paper, we will link this approach with a number of other recent
initiatives and complementary advances, some of which reside in the gray literature of government
reports. The intention is to present a comprehensive, coherent framework for understanding and
describing the role of S&T in the modern economy, a necessary precursor to measuring, managing,
and reporting on the performance of individual organizations or programs.
The Performance Framework Approach
In the late 1970s, the Canadian federal government institutionalized the use of the logic model originally introduced by Joe Wholey and others (Wholey 1980) as a basic tool for evaluation of federal programs. Consequently, there has been extensive experience in the use of the logic model since that time. The performance framework (Montague 1993; Montague 1994) (Figure 1) was developed from the logic model which was modified to include explicitly consideration of the "reach" of the program or organization under review. Reach defines the target clients, key co-delivery partners, and stakeholders which are the mechanisms through which activities and outputs are transformed into results. Rather than focus on impact, this approach considers performance in terms of the entire program in a holistic manner, linking resources to reach and results. This performance framework is congruent with Kaplan and Norton's "balanced scorecard" approach (Kaplan et al. 1996) of business management theory as they each describe successful performance in terms of a spectrum of factors, internal and external to the organization, which relate to both capability and results.
Figure 1. Performance framework
In addition to providing a rational approach to understanding the linkages between resource
utilization, resulting capability, and consequential results, the performance framework focuses
directly on management needs by responding to stakeholders' key questions in a straightforward
manner. The questions How? Who? What do we want? and Why? can be answered directly using
this approach. Some may be concerned that the causal linkage of activities and outputs to
objectives is intended to defend the status quo; in fact, the opposite is true. The performance
framework challenges both existing strategies and their operational implementation to demonstrate
performance against objectives or alternately provides a mechanism to consider alternate service
delivery approaches in terms of better defined performance objectives.
Figure 1 captures the key attributes of the performance framework approach. Conceptually,
resources (staff and operating funds) are used to perform activities and create outputs. This is HOW
one goes about achieving objectives. These activities and outputs reach a target client group either
directly or with the aid of co-delivery partners and stakeholders. This is WHO is affected by the
activities and outputs. As a result of the activities and outputs, the target client group behaves
differently and immediate impacts occur. This is WHAT happens. Over the longer term, the
changed behavior leads to more extensive and consequential impacts. If the program is performing
well, these changes can be causally linked to intended long-term program objectives. This
responds to WHY. Sources of information to measure program performance can then be identified,
and performance indicators can be developed in terms of these themes for any given program or
organization.
This performance framework approach has been used successfully to describe the performance of
many programs in a wide variety of disciplines. It has been adapted for use in the S&T domain by
the authors and colleagues and linked to other initiatives focusing on S&T policy and impact
measurement methodologies. This integrated systematic approach has received broad acceptance
from a number of S&T managers and stakeholders at the program and organizational level in both
Canada and the United States.
Rationale for Government S&T
The performance framework approach does not determine program objectives, but rather adopts
those objectives which have been developed through policy or program decisions by government
or senior management to describe performance. Consequently, the rationale and policies defining
the roles and purpose of government S&T need to be articulated within the performance framework
context to define intended long-tem impacts. Several recent initiatives have helped to better
understand and describe the role of government S&T in modern society.
Economists traditionally have used the non-appropriable externalities or "public good" nature of
much S&T activity to explain under investment by the private sector and the need for significant
government investment in S&T, basic R&D, standards, and related work (Arrow 1962). However,
as well as investing in S&T on behalf of the private sector, the government is also a major user of
S&T to make and implement policy and regulatory decisions to define and manage the society in
which we live. Because of the generic nature of much S&T knowledge, it provides a foundation for
use by both the public and private sectors and supports national competitiveness as defined
broadly to include a well educated, healthy population, and effectively operating society
underpinning the efficient production of goods and services which are competitive in price and
quality.
This concept was carried further by Greg Tassey, an economist with the National Institute of
Standards and Technology (NIST). Tassey notes the existence of technology infrastructure as a key
to economic development in Technology Infrastructure and Competitive Position (Tassey 1992).
Technology infrastructure comprises an economy's set of institutions and facilities relating to its
science base, generic technologies, applied technologies, and "infratechnologies," that is, technical
"tools" such as test methods and measurement techniques or protocols that affect the productivity
of research and the diffusion of innovation.
As mentioned previously, there has been expensive examination of the role of the government as
an investor in S&T to benefit the private sector, but rarely as a consumer using S&T to meet its
internal needs. As well as the obvious role of S&T in defense and public health, government S&T
organizations and resources have contributed to the achievement of government objectives in the
areas of agriculture, the environment, and construction, to name just a few. A Canadian study
(Canada 1993) on socioeconomic impacts of government S&T found that S&T was performed
broadly speaking for four purposes: building of S&T competence, policy development, policy
implementation, and industrial development. Much recent emphasis has been on this last category
and the impact of government S&T on direct wealth creation. This focus is also discussed in the
recent examination of the role of government laboratories in the U.S. by Papadakis (1995).
In fact, evaluators and analysts of S&T programs have come to recognize that "innovation" - the
essential core product of S&T - affects behaviors across a wide range of institutional actors in both
public and private sectors. The influence cannot and should not be constrained by simply
analyzing private or even narrowly defined social returns on investment (Lipsey et al. 1996).
Application of Performance Framework
The use of a performance framework model to respond to How? Who? What do we want? and
Why? facilitates an analysis of the behavior changes and benefits that occur within major
institutional actors as a result of S&T and related activities.
As an example, imagine the development of new software which results in greatly improved
images from remote sensing satellites. The private benefits stream of this innovation may be
minimal, as very few direct jobs or sales are created in the software firm developing the product.
After all, new software requires none of the production "gear-up" that would accompany a
machinery innovation. With competition in this field and difficulties in intellectual property
protection, imitators may soon in fact erode any private competitive advantage for the developing
firm.
But consider the broader behavioral effects on users of data from satellites resulting from this
innovation. With more precise and reliable information available, the ability to make natural
resource allocation decisions relating to agriculture, forestry, and environmental protection is
improved. Emergency response to natural phenomena such as landslides, storms, forest fires, and
oil spills can be better managed. In the longer term, more exacting mapping standards may emerge
leading to the development of world-class expertise in the field which in turn generates spinoffs in
scientific equipment, consulting, and various natural resource management services. All of these
benefits may accrue from as little as one software innovation in the right place at the right time.
The performance framework leads the analysis beyond the natural tendency to focus on immediate
direct impacts of each innovation (e.g., product sales) to an examination of a broad range of benefit
streams. These include behavioral changes beyond the advancement of knowledge and the
adoption of technology by specific users to innovation "system" effects relating to large institutions,
standards, and related sectors of the economy. Figure 2 shows a general application of the
framework to the S&T domain.
Figure 2. S&T Performance Framework
As well as defining objectives, use of the performance framework approach requires the collection
and analysis of performance-based information in terms of the categories of resources, reach, and
results.
In most cases, information on resources is relatively easy to obtain, since program management and
information systems have traditionally focused on resource utilization. Budget allocation,
categories of staff and outputs such as papers and reports published, and seminars held have been
readily available and extensively used as a proxy for impact and overall performance in the past.
However, a refereed publication, although a legitimate indicator of productivity and quality, has no
impact outside the laboratory which produced it until and unless someone else does something
different than they would without having read the article or heard about it at a conference or
seminar (citation is a legitimate indicator of impact).
Reach needs to understood conceptually as a precursor to data collection and analysis. Reach can
include many groups, the first being target and actual clients or recipients of the outputs. Another
could be those with complementary skills which, if induced to participate, can increase the
likelihood of achieving positive results dramatically. An example from recent experience is the
increased linkages between researchers and technology transfer specialists in universities or
government laboratories, which have been found to increase the successful transfer and utilization
of S&T outputs significantly. A third group is key stakeholders, who can provide credibility and
support. An example would be an industry association representing the target client group whose
support might induce members of the target client group to become clients. The last major
category of reach to be considered is the beneficiaries of the S&T activities beyond the direct
clients. For demonstration projects with one firm, this could be the larger industrial sector targeted
as potentially utilizing the technology.
Information on various aspects of reach has often not been previously considered as necessary and
is therefore not typically available. For example, performance-related analysis such as penetration
of intended target client groups can be problematic. For some S&T programs, target client groups
or recipients of outputs have not been fully identified. Targets can be as broad as the international
R&D community or as narrow as a single private firm within industrial sectors (i.e.,
pharmaceuticals) or government policy groups responsible for regulation as examples of
intermediate level targets. Often information systems do not capture client information, and
performance analysis in terms of reach (penetration of a target client group) is difficult to perform.
Reach is defined to include co-delivery partners. For many government S&T programs, effective
linkages with private sector partners or industry associations can have a major influence on
achievement of results. In the case of NIST and Canada's National Research Council of Canada
(NRC), private sector calibration laboratories are important means to reach the intended audience
of producers and users of measurement equipment.
Results, defined as "What do we want?" and "Why?", are particularly difficult to measure for many
S&T activities. The long pathway between S&T and ultimate impact, with the many intervening
factors which come in to play, including business cycles, interest rates, and politics, can make
attribution and causality difficult to determine. Immediate impacts are usually more directly
attributable to the S&T whereas, except in certain cases with few intervening factors, longer-term
impacts become more difficult to claim. In practice, even immediate impacts are often difficult to
determine, especially if client information is not kept, since it is usually necessary to have some
indication of the change in client behavior to assign impacts. Often a combination of quantitative
and qualitative information on service standards, client awareness, and use of S&T can be collected
by using client surveys, end-of-project feedback forms, and file analysis. In the authors'
experience, many S&T programs, while ignoring immediate impacts, attempt to determine results
in terms of longer-term impacts in spite of the difficulties, as a response to the need for
accountability and continued funding. A more balanced approach to measurement, capturing
indicators of both immediate and longer-term results is usually more useful to program
management and credible to stakeholders. While care needs to be taken to keep performance
measurement efficient, expanding the utilization and resulting benefits can compensate to some
extent.
There are a number of reports and books which identify and describe methods for determining the
immediate and longer-term results or impacts of S&T. Some are quite technical, as they are written
for an expert audience. One review intended for nonspecialists, mentioned previously, is a study
entitled Methods for Assessing the Socioeconomic Impacts of Government S&T (Canada 1993),
which describes and analyzes the major methods available and their applicability and provides an
extensive bibliography of published and gray literature from various countries. Table 1, from that
report, presents a summary of the applicability of various methodologies for R&D performed for
various purposes. In this table, traditional peer review has been modified to include greater input
on the potential of downstream utilization of research to complement the focus on research quality.
The partial indicators identified in this table are closely related to the general performance
framework approach being discussed, requiring the identification of a number of types of
information, each of which provides a partial indicator of impact. Following the approach
identified in this paper and making use of the appropriate methodologies in Table 1, it should be
possible to use several complementary methods to perform a credible assessment of the
performance of virtually any S&T program.
Table 1. Methods useful for assessment of past R&D
* Use of brackets signifies potential for use in particular circumstances.
The basic elements of the performance framework approach have been used by the authors since
the late 1980s in assessment work at the NRC and other S&T organizations in Canada. The 1990
Industrial Research Assistance Program (IRAP) Evaluation Study (Canada 1990) used the basic
performance framework approach, examining resources, reach, and results. The study included
extensive analysis of the penetration of IRAP into the Canadian Manufacturing Sector as well as the
immediate and longer-term impacts of IRAP assistance as reported by assisted firms. IRAP, a
technology extension program, was found to be highly incremental, and clients attributed a
considerable share of their success to IRAP assistance. This assessment was used as a reference
document by a 1991 Parliamentary Inquiry into the program as an important input to decision
making and was quoted extensively as the basis for conclusions and recommendations.
Another extensive assessment of the same program, documented in Assessment of Industrial
Research Assistance Program - Review Committee Report (Canada 1996), has just been completed
using a similar performance framework approach which provided a comparative analysis of
intended and unintended changes to the program five years later. The Review Committee
responsible for the assessment, made up of program stakeholders external to NRC, reported that the
performance framework approach was an effective method to collect credible evidence on the
overall performance of IRAP and to develop recommendations on key aspects of IRAP as input to a
new Strategic Plan for the next five years.
There are many other examples of successful use. The Canadian federal industry department,
Industry Canada, has developed a guide to assist managers in understanding and measuring
performance (Canada 1995), and the Canadian Technology Network (CTN), a recent initiative of
the federal government, adopted the framework approach to assist with monitoring and managing
both implementation and ongoing network performance. The document, An
Evaluation/Performance Framework for the Canadian Technology Network (Canada 1995),
contains an extensive description of the principles of the performance framework as well as a
practical example of the use of those principles to determine key performance characteristics for
CTN.
The performance framework approach is relevant to many levels of management and S&T decision making. Figure 3, reproduced for the CTN study (Canada 1995), demonstrates the relevance of performance information to various levels of management. Operationally, attention is primarily focused on resource management and delivery - with some reference to reach and immediate impacts. As the focus changes from program delivery to strategic and corporate to government level considerations, there is progressively more attention paid to longer-term impacts. For major S&T organizations and at the national level, there is a clear requirement to link program impacts to government S&T policy objectives.
Note: Information from the performance framework can apply at the central agency and corporate level for purposes of
accountability and higher management functions (policy and resourcing) as well as at tactical/operational levels
for resourcing, design, delivery, and operational control.
Figure 3. The relationship of key performance information on different management levels
Conclusions
Initial experience in the application of a performance framework for the analysis of S&T
performance has been promising. Frameworks developed for specific programs and organizations
have been shown to assist S&T performance planning, measurement, and reporting. The approach
helps resolve traditional conceptual difficulties such as inappropriate narrow considerations of
benefits and impacts, and provides a practical, consistent template for information collection,
analysis, and reporting on performance.
Arrow, Kenneth. "Economic Welfare and the Allocation of Resources for Invention." In The Rate and Direction of Economic Activity: Economic and Social Factors. Princeton: NEBR, 1962. Canada. Federal Government. Methods for Assessing the Socioeconomic Impacts of Government
S&T. May 1993.
Canada. Industry Canada. Focusing on Results: A Guide to Performance Measurement. March
1995.
Canada. Minister of Supply and Services. Science and Technology for the New Century - A
Federal Strategy. 1996.
Canada. Minister of Supply and Services. Science and Technology for the New Century - Industry
Portfolio's Action Plan. 1996.
Canada. National Research Council. An Evaluation/Performance Framework for the Canadian
Technology Network. August 1995.
Canada. National Research Council. Assessment of Industrial Research Assistance Program -
Review Committee Report. November 1996.
Canada. National Research Council. Industrial Research Assistance Program - Evaluation Study
Final Report. October 1990.
Kaplan, Robert, and David Norton. "Using the Balanced Scorecard as a Strategic Management
System." Harvard Business Review, January-February 1996, pp 75-85.
Lipsey, Richard G., and Ken Carlaw. "A Structuralist View of Innovation Policy." In The
Implications of Knowledge-Based Growth for Micro-Economic Policies, ed. Peter Howitt, Ministry
of Supply and Services. Canada: University of Calgary Press, 1996.
Montague, Steve. "The Three Rs of Performance-Based Management," Focus, December/January
1994, pp 26-28.
Montague, Steve. Performance Framework Approach to Public Management. January 1993.
Papadakis, Maria. "Federal Laboratory Missions, Products and Competitiveness." Journal of
Technology Transfer 20, April 1995, pp 54-67.
Tassey, Greg. Technology Infrastructure and Competitive Position. Norwell, MA: Kluwer, 1992.
Wholey, Joseph. Planning Useful Evaluation: Evaluability Assessment. Beverly Hills, CA: Sage,
1980.
George Teather has been conducting evaluations for the National Research Council of Canada
(NRC) for 12 years, examining a wide range of S&T activities, including research and development,
library services, information dissemination and technology assistance to firms. Before becoming an
evaluator, Mr. Teather co-authored a number of scientific articles as a researcher in NRC's
laboratories. He is a professional engineer with a Master's degree in Electrical Engineering and is at
present the Chair of the professional development committee of the Canadian Evaluation Society.
Steve Montague is a Director of Performance Management Network, a management consulting firm
focusing on performance-related issues. Mr. Montague has been involved in supporting
management decision making in public and private sectors organizations in S&T and other sectors
for more than 15 years. Over the past several years he has delivered many workshops and
presentations on performance-based management, including recently a number for senior Canadian
federal government managers. Mr. Montague has a Master's degree in Public Administration.
| ||||||||||||||||||||||||||||||||||||||
