Evaluation Guidelines TPG22-22

The NSW Evaluation TPG (2022) sets out mandatory policy requirements (compliance set out by NSW Treasury) and provides recommendations (reflecting best practice standards) and guidance to support consistent, high quality and transparent evaluations. Evaluations of government resourced ongoing and new initiatives should be prioritised, planned, conducted and reported with reference to the requirements, recommendations and guidance set out in this TPG.  

The TPG is supported by online evaluation workbooks and evaluation resources that can be used as needed. The Workbooks provide information to support monitoring and evaluation, including example templates. They are designed to be used flexibly and can be used as appropriate to the initiative, the context in which it is being evaluated, and the resources allocated to the evaluation. The Resources, including technical notes, provide detailed technical guidance to the terms and methods used in evaluation, and identify best-practice standards.

A logic model provides a high-level representation of how an initiative is intended to work. It can illustrate key activities and outputs, their outcomes and benefits, and the linkages between them.  

A logic model will be relevant, and can be further developed, over the life of an initiative, with the level of detail increasing with time. In its early development, a logic model will identify the intended benefits and objectives of government intervention. In the later stages of developing an initiative, the model should clearly detail how the initiative will achieve benefits and objectives. 

For large or complex initiatives (which include several connected sub-initiatives), it may be useful to develop an overarching logic model that summarises the key components of the initiative, as well as more detailed individual models for the sub-initiatives that make up the larger initiative.  

For further information on logic models, see Workbook I. Foundations for evaluation, Workbook VIII. Complex initiatives and Technical note. Logic models across the investment lifecycle.

Outputs are the products, services and infrastructure that result from the initiative activities.  

Outcomes are the changes that are attributable to the initiative outputs. Changes may be in economic, social, environmental or cultural conditions and may occur in the short, medium or long term. They may include changes in lives, status, health, surroundings, knowledge, attitudes, values, behaviours or satisfaction levels.  

For more information about the key stages in the logic model and the linkages between them, see Technical note. Logic models across the investment lifecycle and Workbook I. Foundations for Evaluation. 

The impacts of an initiative will include outcomes and associated benefits. In general, outcomes can be understood as the extent of changes that follow from an initiative, and benefits as the improvement in welfare related to that change (benefits are a measure of the value of outcomes to the NSW community – which may be monetary or non-monetary).

Monitoring is a continuous and systematic process of collecting and analysing information about the implementation and impacts of an initiative. Monitoring activities are important for collecting data required to undertake evaluations. 

Evaluation is a systematic and transparent process that can be used to assess the appropriateness, effectiveness, efficiency and net social benefits to the NSW community of an initiative once it has been implemented. Evaluation is most effective when supported by monitoring of the implementation and impacts (outcomes and benefits) of an initiative. Evaluation will usually investigate a broader range of information on initiative performance and context than is feasible to monitor on an ongoing basis, and involves research to address specific questions regarding the performance of the initiative. 

Monitoring and evaluation should be coordinated to provide evidence of an initiative’s performance.  

A monitoring and evaluation framework is a framework that plans for and guides monitoring and evaluation over the life of an initiative. The framework can be built upon a logic model for the initiative that identifies implementation (inputs, activities and outputs) and impacts (outcomes and benefits, and when these are due to be realised) (See Workbook I. Foundations for evaluation). 

For more information on Monitoring and Evaluation Frameworks, see Workbook II. Monitoring and evaluation framework.

Benefits management can inform but is not a substitute for evaluation.

Benefits management aims to maximise the likelihood of an initiative achieving its intended outcomes and benefits. On the other hand, evaluation is an independent process used to assess an initiative’s efficiency and effectiveness.

See Appendix 3 in the Benefits Management Guide for more information.

Cost-benefit analysis (CBA) is used to investigate the actual costs, outcomes and benefits of an implemented initiative and assess its net impact on social welfare.
Ex-post CBA may be conducted during the life of an initiative (also known as ‘in medias res’ or IMR CBA) and/or at the end of an initiative’s life.  

When conducted during the life of an initiative, an ex-post CBA (IMR) can provide evidence to: 

• review the total expected costs and benefits of an initiative in progress
• update an ex-ante CBA in light of new data collected
• assess whether outcomes and benefits are likely to continue
• inform adaptive management of a current initiative to maximise net benefits.

A conclusive ex-post CBA may be undertaken when all expenditures and benefits from an initiative are realised. For some initiatives, benefits may continue for some decades after implementation has commenced (and an IMR may more practically inform decision-making). An ex-post CBA completed at the conclusion of an initiative can be used to: 

• assess whether the initiative was a good investment, and  
• assess whether the ex-ante CBA, where undertaken, was broadly accurate.  

Ex-post CBA should be undertaken at the intervals that are useful to inform decision-making.  

For more information on the ex-post CBA, please see Technical note. Ex-post CBA.

Outcome evaluation examines if and how an initiative is leading to intended changes. It measures the extent of change and the degree to which the initiative has contributed to the observed changes. It can also be designed to capture unintended outcomes. Outcome evaluation can include questions about distribution and equity of outcomes.  

Outcome evaluation can be undertaken when the initiative is expected to achieve outcomes (across a sufficient sample size). It will provide an important foundation for economic evaluation, including CBA. 

Economic evaluation identifies and measures the impacts of an initiative relative to its costs, to provide an assessment of value for money or net social benefit. Economic evaluation will build on evidence collected from process and outcome evaluations, and from initiative monitoring (including under Benefits Realisation Management).  

Cost-benefit analysis (CBA) is a comprehensive examination of an initiative’s incremental impacts on welfare (including economic, social, environmental or cultural impacts) and assessment of whether the investment delivered a net social benefit to the people of New South Wales. CBA is the NSW Treasury preferred method for economic evaluation (ex-ante and ex-post) as it allows for the objective comparison of net social benefits across different initiatives and portfolios. It is recommended for large, complex or risky initiatives.  

For more information about outcome and economic evaluations, see the Evaluation TPG. 

The counterfactual is a consideration of what would likely have happened in the absence of an initiative, usually under a “do minimum” or “business as usual” (BAU) scenario as set out in the initiative’s  business case. In ex-post CBA, the ‘counterfactual’ includes projection of costs and benefits ‘without’ the initiative. The use of a counterfactual is to ensure that the assessment of the initiative is based on ‘additionality’ - that is, the ‘extra’ amount of costs and benefits over what would have been the case in the absence of the initiative.

Expertise external to the business and delivery team can be used to undertake the evaluation or aspects of the evaluation. This may be an evaluation unit or evaluators within the agency, independent consultancy teams or academic teams. The need for external expertise may depend on existing evaluation capability within the team, the technical expertise required for the evaluation, and any requirements for an independent perspective.

See Table 3 in the Evaluation Guidelines and Evaluation Workbook 5: Use the right expertise for more details.

Best-practice is to include resources for monitoring and evaluation in the initiative budget. All proposals seeking government resources are required to incorporate resourcing requirements for monitoring and evaluation into the business case funding submission (or set funding aside) or include a plan for when resourcing will be determined and obtained. The monitoring and evaluation budget should be appropriate to the evaluation requirements and scope, as well as the size of the initiative.  

Monitoring and evaluation costs will likely be within a range of one to five per cent of the initiative budget, depending on data needs and the scope of the evaluation. For a large infrastructure project, the relative cost may be at the lower end of this range. For a pilot initiative, where the intention is to test an initiative and provide learnings for a potential expansion, evaluation costs may be greater (from five to ten per cent, or above). Where available, reference the average budgets for evaluations that have been successfully undertaken that were of a similar type and scope.

For more information on budgeting for evaluations, see Workbook II. Monitoring and evaluation framework.