It is becoming increasingly necessary to assess the impact of technical assistance projects. But the standard indicators, most of which are based on quantitative measurements, are not necessarily appropriate given the diversity of the programmes concerned. When resources permit, the conceptualisation of an impact chain and use of a planning matrix help practitioners to develop indicators appropriate to each stage of the project.
These days, impact assessment is in high demand. Donors want to know what happened to their money, and organisations are keen to demonstrate all the many, supposedly positive, developmental impacts their projects generate. This has become more pertinent as development aid has come under criticism from various stakeholders for not helping developing countries.1 Even though technical assistance (TA) is not considered classical development aid, the (perceived) pressure to demonstrate the impacts generated with public funds is high.
Often, the result of this pressure is the production of tedious reports with large charts, numerous tables, long lists of the effects, and other ‘evidence’ of the development impact of the TA project concerned has supposedly generated. The various European Development Finance Institutions (EDFI) have had different approaches to impact assessment and reporting, ranging from modest to more sophisticated procedures. The form and content of these procedures often depends on the specific reporting requirements of the programme sponsor.
The quest for indicators
The trend is toward ‘hard facts’: a project’s impact is normally required to be verified with quantifiable indicators. Ideally, a set of ‘standard’ indicators is applied to projects, allowing for a prompt assessment of developmental surpluses. In addition, one can assign grades or points to the indicators and then rank and compare projects in order to ascertain relative performance. With this information, one can identify, among others, ‘best practices’, lessons learnt and worst-case scenarios.
For a diverse TA programme, such as DEG’s, which includes a range of different projects (in various sectors, relating to various topics and varying widely in size and length), it is not feasible to define a ‘standard’ set of indicators to measure individual project impacts. TA projects are often geared to address and remedy very specific issues: the impacts of a market study on Serbia’s renewable sector cannot be compared to the impacts of asset liability training for bank employees conducted in many different countries. Unlike the assessment of ‘regular’ investment projects, applying a standard set of indicators to every TA project would not do justice to the diversity of the projects. However, what may be feasible is to cluster TA projects into distinct categories and define common indicators for each category. FMO (Entrepreneurial Development Bank of the Netherlands), for example, has been working with the following six categories: ‘Management Strengthening’, ‘Environment and Social’, ‘Product Development’, ‘Seminars, Conferences’, ‘Organisational Strengthening’ and ‘Corporate Governance’.2
The impact chain
To consider the specific nature of a TA project when assessing its impact, it may help to revert to so- called ‘impact chains’. A project’s impact chain describes the connection between the project intervention and possible resultant impacts. It builds on individual steps that exhibit cause and effect relations (based on plausibility considerations).
The Figure 1 depicts a stylised impact chain with a few DEG TA project examples. Consider the first example of an energy efficiency check: a commissioned specialist performs an energy efficiency check at a partner company. The output of this TA project consists of a report with recommended energy savings’ measures for the company. As a consequence of the report, the company implements energy savings’ measures (outcome), which in turn reduces its energy consumption. This could eventually contribute to the broader goal of climate protection (impact). Note that the ‘cause and effect relations’ between the individual steps are based on certain assumptions; for example, the final impact ‘climate protection’ of the project is based on the assumption that the energy used by the company stems from fossil fuels.
Obviously, the energy efficiency example would score badly when applying standard impact indicators such as ‘number of jobs created’ or ‘improved gender equality’. This is because the project pursues a very specific goal. In order to measure the success and impact of this project, it would therefore be more helpful to look at the impact chain, and from there to derive specific indicators. This can be done on different levels of the impact chain: on the output level, for example, the number of proposed energy efficiency measures or the projected amount of energy savings could be indicators to measure the quality of the specialist’s report; on the outcome level, one could measure the change in electricity consumption of the company; an indicator for the project’s impact on climate protection could be the (decreased) CO2 emissions resulting from the company’s energy consumption. The further down the impact chain one moves, the more difficult it is to measure the impact and to assign the impact to the project. It might be quite complex, for example, to go to the company’s utility provider and measure the CO2 emissions at the coal plant. Instead, it might be easier to monitor the company’s electricity (or other energy) consumption.
A tool that can be applied for these purposes is the so-called ‘project planning matrix’: this helps to structure a project and to define suitable goals and indicators. The first step is to define the goal of the project according to the specific bottleneck that is to be remedied. The goal usually relates to the outcome level. The asset/liability training of bank employees, for example, may have the (direct) goal of improving the bank’s risk controlling and thereby increasing the bank’s overall sustainability. The broader impact of such a measure could be a contribution to the stability of the financial system. Next, one could define specific steps that would lead to achieving the goal. These could be translated into the definition of various results on the output level of the project. With such a specification, one is able to derive measurable indicators on the output, outcome and impact levels. In the example of asset/liability training, these could be the number of people participating in the training (output level), the number of people that successfully pass a test (to demonstrate their upgraded skills), and a change in the ratio of non-performing loans.
Thorough impact assessment
When measuring impact, two important issues must be considered. Firstly, a before and after comparison is needed for meaningful results on the changes resulting from the intervention. Secondly, external factors may influence and thereby distort the results. An increase in production, for example, can lead to the consumption of more electricity, even though energy saving measures were applied. To deal with the first issue, it might be necessary to conduct a baseline study to measure the indicators before the implementation of the TA project. This data can be compared to the data collected after the implementation. The second issue requires elimination or ‘discounting’ of the external factors. In the above example, this could be done by comparing the energy consumption for one output level. Since this is not always possible (or feasible), another common practice is to employ a ‘control group’ that has the same features as the target group, but does not receive the TA intervention. Only when comparing the results before and after, as well as the target and control groups, is one able to attain a sound and robust impact assessment.
Effectiveness and efficiency
An efficient impact assessment method, frequently applied with DEG’s TA projects, is ticking a pre-defined list of boxes (of the perceived impacts). This is done by the relevant project manager at the end of each project. The problem with this type of assessment is that it provides an estimation of impacts and may often merely reflect the intrinsic motivation of the project manager to demonstrate the positive results of his/her project. The clear advantage of this approach, however, is that it requires few resources (time and money).
From a practitioner’s point of view, there is a trade-off between effectiveness and efficiency in the conduct of impact assessments. A thorough (more effective) impact assessment with control and target groups as well as a comparison of before and after is often not feasible because of limited resources. Therefore, one has to ask whether thorough assessments are appropriate, taking into account the circumstances, and decide how close one can get to this ideal, taking into account efficiency aspects.
DEG’s TA team is exploring different ways of improving impact assessments. This is an on-going process. One of the main challenges of TA impact assessments is finding an efficient way to attain reliable information on the effects and impacts of a TA project in order to get closer to a thorough assessment. To cope with these challenges, one can apply seven steps that may be useful.
The first step involves a clear definition of the goal of the proposed TA project; the more precise the definition of the goal, the easier the assessment of the success of the project will be. If the goal is clear, the second step would involve describing an impact chain illustrating ways of reaching the goal, as well as the impacts that would be generated by the project. The description should include the assumptions behind the individual steps, if necessary. A third step involves deriving measurable indicators from the impact chain; this can be done on various levels of the impact chain (at output, outcome, or impact levels). A fourth step, if necessary, would involve a baseline study to obtain reference values for the predefined indicators. The fifth step, at the end of the project, involves collecting and comparing data on the indicators to the baseline findings. A sixth step – if the data source is not reliable, or in the case of uncertainties and difficulties obtaining the data – involves having a third party collecting or verifying these on the ground. The seventh step, if appropriate, would involve a second assessment to be conducted at a later point to evaluate the long-term effects of the project.
¹ Compare to i.a. Moyo, Dambisa (2009) “Dead Aid: Why Aid Is Not Working and How There Is a Better Way for Africa”, or Easterly, William (2007) “The White Man’s Burden: Why the West’s Efforts to Aid the Rest Have Done So Much Ill and So Little Good”.
² Presentation by Andrew Shaw at the EDFI-TA Workshop in Köln on 7 April 2011.