Manual of Performance Audit

Indeed, this process of change is still going on, mainly due to internal and external factors. The main external factors have been the enactment of the Government Accounts Act 2001 and the publication of a policy document entitled From policy budgets to policy accountability. The most important internal factors have been the Court’s mission and strategy. This manual has been conceived as a strategy-neutral document, and contains information on the analysis of policy objectives and policy information, and studies of outputs, outcomes and explanations.

These subjects are likely to affect to a greater or lesser degree any strategy adopted by the Court. The idea is that auditors should be able to select from this manual those sections that have a bearing on the audit they are intending to perform. The manual starts by outlining the framework in which the Court carries out efficiency and effectiveness audits, and by defining the most important terms. The manual also describes the type of decisions you can expect to take when planning and conducting an efficiency or effectiveness audit.

How you do this depends greatly on the nature of the problem and the domain in which the audit is to be conducted. For this reason, the manual does not contain any hard and fast rules for audit strategies. What you will need to do in each case is to explain in detail how far your audit intends to go (and this can range from simply finding out whether the objectives of a given policy have been achieved, to assessing the social impact and cost), what sort of data you intend to collect, how you are planning to collect them and how you intend to analyse them in such a way that you can answer the audit question.

The standards set by the Netherlands Court of Audit for government action are an important factor when it comes to formulating conclusions. Performance audits 7 Efficiency and effectiveness audits come with fixed standards for assessing the formulation of policy objectives (see section 5. 2. 3) and the quality of policy information (see section 5. 3. 4). No general standards have been set for assessing the way in which outputs and outcomes have been formulated.

In this case, you will need to formulate specific standards of your own, based on the stated objectives and the arrangements made in the relevant policy field (see also section 6. 2. 2). The same applies broadly speaking to the assessment of the effectiveness of government policy (see section 7. 3). In order to guarantee consistency in the way in which standards are formulated, these have been codified in the form of a standards database; every audit team is obliged to consult this database when compiling an audit proposal and to refine it once the audit has been completed.

This Performance Audit Manual replaces the Performance Audit Manual (1996), the Policy Outputs Audit Manual (1997) and the Efficiency Audit Manual (2001). The authors of this manual have used material from all three previous manuals. The manual has been designed to tie in with the material used in the course on audit methods and techniques. Performance audits 8 1 Introduction 1. 1 Efficiency and effectiveness audits conducted by the Netherlands Court of Audit The efficiency and effectiveness audits conducted by the Netherlands Court of Audit are designed to measure outputs and outcomes.

There are a number of different ways of looking at outputs and outcomes, however: we can analyse the degree to which the body in question has achieved its objectives, we can analyse the effectiveness of policy and we can analyse the efficiency of outcomes and outputs. 1 These different types of audit are layered: in order to audit the effectiveness of government policy, the assumption is that you already have information on the degree to which the government has been successful in achieving its policy aims.

In addition, an awareness of the effectiveness of policy is required in order to reach a judgement on the efficiency of the outcomes. Against this background, the different types of audit – ranging from measuring a particular body’s success in achieving its policy objectives to measuring its efficiency in doing so – are growing both ever more complex and ever more demanding in terms of the quality of the policy information on which they are based. 1. 1. 1 Have the objectives been achieved?

An audit of the achievement of objectives involves analysing the actual social situation after a policy has been implemented, and comparing this with the policy objectives, i. e. the social situation the policy was intended to create. It is not possible, however, to establish a causal link between the government’s performance and the degree to which certain effects have been produced. In other words, it does not allow you to make any pronouncements about the extent to which a minister’s policy has been successful or not.

This type of audit is particularly relevant to the Court’s remit (i. . improving the operation of central government and/or legal 1 There is also a fourth type of audit: an analysis of the extent to which the target group has been reached. This type of audit is discussed in chapter 6; see also section 1. 2. 2 below. Performance audits 9 persons with statutory tasks) if it is reasonable to expect the government body in question to have enjoyed only limited success in achieving its aims or if you want to build a platform for a more in-depth analysis of the effectiveness of government policy.

For this reason, the strategy described in chapter 6 for assessing the achievement of objectives is as open as possible to more precise effectiveness audits. 2 In some cases, it is either too early or impossible for other reasons to establish whether the ultimate objectives have been attained, in which case it may be useful to try and establish whether the auditee has reached certain predefined milestones along the way. This may also be important where a minister is not actually responsible for the achievement of a given aim, but is responsible for ensuring that certain milestones are reached.

This applies in particular to facilitation audits. The key question here is whether the conditions have worked that the minister has put in place in order to ensure that government policy is properly implemented by regional, provincial or local authorities. In other words, facilitation audits look at intermediate stages in the policy chain, the stages between the role performed by central government in creating the right conditions for policy and the work performed by local authorities and organisations. In a setting in which policymaking has been largely decentralised, acilitation audits form an excellent complement to audits of central government performance.

The minister in question is directly accountable if the work of his or her ministry is found not to have had any effect upon the possibilities of local authorities. This may explain why certain social effects have not been achieved or have not been achieved to a sufficient degree. 1. 1. 2 Has the target group been reached? In certain practical situations, it may prove difficult to establish whether certain general objectives, such as ‘conserving nature in the Netherlands’, have been achieved.

What you can do in such cases is establish the auditee’s success in achieving certain secondary aims or milestones. One of these is its ability to reach its target group. 2 See also Lulofs and Schuddeboom (1991). Performance audits 10 An audit of the government’s ability to reach its target group may be appropriate where the policy in question is expected to result in either individuals or certain groups displaying a particular type of behaviour. In such cases, policy is effective if it has an impact on the right people.

This means that the predefined target group has got to come into contact with the policy in some way or another. Whilst reaching the target group is not a sufficient condition for also achieving the policy objectives, it is very unlikely that these will be achieved if either the target group or the relevant intermediaries have not been reached. Theoretically, it is possible to include in any audit of the impact of government policy an examination of the extent to which the auditee has been successful in reaching either its target group or the relevant intermediaries.

Effectiveness audits When the Court performs an effectiveness audit, the purpose is to establish whether the government policy in question has had the desired effect. The term ‘effect’ covers both outputs and the outcomes achieved by means of these outputs. Effectiveness audits are complex, partly because the policy chain may be long and partly because there may be external factors at work that can affect the ultimate aims of the policy in question. There are often a whole series of steps between government outputs and the social situation that is ultimately created.

The more steps there are in the chain leading to a given output or outcome, the harder it is to establish a causal link with government policy. In many cases, the outcome is not simply the result of the policy pursued by the minister, but is affected by a range of other factors. For example, whether mature women are successful in returning to the labour market depends not only on measures taken to improve placement services for such women, but also on a variety of socioeconomic factors. 3 Environmental subsidies are a case in point.

The chain from outputs to social impact includes series of intermediate steps such as the payment of subsidies, the target group’s awareness of the existence of the subsidies, the decision taken by the target group to buy the product in question, the use of the product by the target group, and environmental changes. Performance audits 11 1. 1. 4 Efficiency audits Citizens are interested not just in whether certain policy outputs have been achieved, but also in how much they have cost and whether the cost can be reduced. Hence the importance of conducting efficiency audits, in addition to audits of outputs and outcome.

Such audits may either concentrate on the outcomes or on the outputs produced by the implementation of government policy. In the former case, the audit is geared towards establishing the cost-effectiveness of the outcomes of government policy, whilst the aim in the latter case is to measure the efficiency of the outputs produced by the implementation of government policy. The question is whether: • • the same outcomes or outputs could have been achieved with fewer resources; or the same resources could have produced more outcomes or outputs. In practice, the terms ‘policy efficiency’ and ‘operational management efficiency’ are used too.

The former relates to the policy outcomes, whilst the latter relates to policy outputs. Figure 1. 1 shows the relationship between efficiency and effectiveness audits on the one hand and the various aspects of the public-sector production process on the other. The figure does not include audits of the achievement of objectives or the reaching of the target group, as these may be regarded as sub-types of effectiveness audits. Performance audits 12 Efficiency with which outcomes have been achieved cost-effectiveness Efficiency with which outputs have been produced efficiency Low-cost economy Input Throughput Outputs Outcomes

Staff, equipment and money Production process Products and services Impact Effectiveness of policy effectiviteness Figure 1. 1 The public-sector production process (source: Ministry of Finance, 2001) Both the intended and the actual outputs and outcomes play a role in the above figure. 1. 2 Summary of contents Chapter 2 of this manual discusses the legislative framework as this affects efficiency and effectiveness audits. Under the Dutch Constitution, the Netherlands Court of Audit enjoys an independent position in the Dutch constitutional system. The Court’s duties and powers are regulated by the Government Accounts Act.

This chapter explains how the Court’s constitutional position and its duties and powers affect its work in conducting efficiency and effectiveness audits. Chapter 3 discusses the object of the audits: what are the various stages of the production process in the public sector, and what types of outputs and outcomes does the Court examine in its audits? This chapter also explains how to choose between different types of audit: audits of the Performance audits 13 achievement of objectives, audits of the reaching of the target group, effectiveness audits and performance audits.

Chapter 4 is all about collecting and analysing data for the purpose of efficiency and effectiveness audits. Dependent on the nature of the problem in hand and the type of audit chosen, you can use various data collection methods to make pronouncements about outputs and outcomes. Depending on the data collection method you adopt, you can then use a variety of analytical methods to assess possible explanations for discrepancies in outputs and outcomes. The various methods are discussed in this chapter.

Chapter 5 outlines various methods for auditing policy objectives (i. . what outputs and outcomes is the policy intended to produce? ) and explains the standards used for assessing policy information. Chapters 6 to 8 explain how you can use either policy information (provided it is of a sufficiently high standard) or data you have collected yourself to audit outputs and outcomes. Problem definitions, standards, audit designs and possible recommendations are discussed for all the various types of audit. We should like to stress that it is by no means necessary for every audit to keep strictly to the procedure set out in this manual.

Auditors are encouraged to select those items that are relevant to the audit they are planning to perform. Performance audits 14 2 General framework for efficiency and effectiveness audits 2. 1 Introduction The Dutch Constitution guarantees the independence of the Netherlands Court of Audit in the country’s constitutional system. The Court’s duties and powers are regulated in the Government Accounts Act 2001. 4 The Court’s audits are based on the administrative responsibility of ministers, state secretaries and the boards of organisations which have links with central government.