How “Practical” or “Useful” are Rapid Evaluation, Really?
There is no doubt that evaluations can play an integral role in decision-making. On the one hand, when the information is accurate, detailed, and insightful; evaluations can help make effective programme decisions, like whether to continue, scale up, discontinue or even, whether to intervene at all. On the other hand, it is well understood that there are significant investment requirements to producing reliable and insightful evaluations.
Besides the financial burden associated with collecting programme data, evaluations can require high levels of technical skills while also commanding long periods of time. However, there are situations when decisions have to be made under time constraints and limited access to technical skills. And indeed, it has been argued that high pressure to deliver development imperatives, limited technical skills, access to financial resources, including weak information systems are a common feature across African public institutions. As such, the idea of producing good evaluation information timeously can be a challenge.
To counter expensive and complicated studies with long turn-around time, rapid evaluations have been strongly lobbied as a more practical alternative. Rapid evaluations are meant to strike a balance between maintenance of high levels of investigative rigour in evaluation, and, the high cost and technical requirements.
“The elephant in the room” though, is the reminder that evaluations are a means to a greater end. Or that an evaluation report itself, IS NOT the end. In other words, the real judgement of rapid evaluations is in whether they truly accommodate key functions of a good evaluation. You see; praxis teaches us that the value of an evaluation must be demonstrated by its role across the programme or policy cycle. And if we are going to rely on rapid evaluations to be in anyway helpful in decision-making – I argue – it is only fair to assess their worth against the same functions. To facilitate my argument, I present the following five key evaluation functions:
1) The Improvement function, referring to enhancement of the efficiency and effectiveness of the chosen programme strategies and how they are implemented. 2) The Coordination function, meaning that the evaluation process assesses the roles of different collaborating actors, allowing partners to know what the others are doing, and how this works or links with their respective roles. 3) The Accountability function, assessing whether identified outcomes are actually being reached, and how wide the deviation is. 4) The Celebration function, celebrating the achievements of the underlying programme or policy, and finally, 5) the Legitimation function, which refers to the idea that an evaluation serves to provide persuasive and credible data to justify the underlying programme or intervention logic. This function is critical in Theory Based evaluation as it is the primary function of evaluation in this context.
The common threats connecting and enabling the attainment of each of the five functions is the ability of rapid evaluations to adhere to basic quality elements such as reliability, credibility, accuracy and relevance. With that said, a key point of contention surrounding rapid evaluation, is their ability to deliver exceptional levels of quality information within a shorter period of time, and in a cost effective manner. A matter in question is whether the ideal that evaluations are meant to be a systematic determination of a subject’s merit, worth or significance (Canadian Evaluation Society, 2015), can sufficiently be met by rapid evaluations. Can the balance between a quick turn-around and comprehensiveness evaluative information actually be achieved in practice? Can evaluators can provide rigorously researched answers within a limited amount of time and budget, and if so, in which situations can this be done?
The idea of reducing evaluation “turn-around time” and budget, while at the same time ensuring high levels of rigour by using multiple methods to collect data from multiple sources, implies very direct assumptions regarding capacity. For one, it assumes existence of adequate technical skills to manage such levels of triangulation. Secondly, it assumes existence of reliable secondary data. And finally, it assumes good programme design with acceptable monitoring frameworks to facilitate the evaluation process.
It is a reality that, in Africa, we operate in a contexts where systems of evidence are not necessarily supportive of evaluations, i.e. we seldom have integrated systems with complete data; or logically designed social programmes with clear monitoring frameworks. In fact, it is almost always the case that evaluations have to include collection of primary data, and hardly rely on monitoring data (if it exists at all). While evaluation budgets are also usually constrained, our commitment to dedicate resources to capacity development is proof that our institutions have glaring gaps in evaluation technical capacity to take on evaluation projects.
Any approach to rapid evaluation we are to employ should therefore take cognisance of our reality. It should be responsive to the levels of evaluation capacity as we experience them. Therefore, my position is that an appropriate guide to rapid evaluation should be one that is based on well-researched elements of what is needed. And not one that is based on a theoretical idea, which might not necessarily fit the context.
What is your position?
By Khotso Tsotsotso
A Twende Mbele Masters Student
Evidence informed decision making amidst competing priorities in the public sector
Background
Basic economic theory notes that scarce resources should be optimally utilized which in turn point to a need for evidence-informed decision-making. The availability of evidence should guide decision-makers on adopting government policies that will best meet development imperatives. Globally there is a clarion call for using existing and new evidence to inform decision-making. In particular, attaining the Sustainable Development Goals, SDGs, now emphasise use of evidence and data driven action (UNDSN, 2018).
There is also ongoing debate as to what interventions strengthen the use of evidence for policy and decision-making. Based on this, several development partners have supported interventions that strengthen monitoring and evaluation (M&E) capacities in use of tools for translation of evidence to make research evidence available for use for decision-making. There is also a growing community of practice around M&E systems for evidence use, including actors such as the African Evidence Network and the Uganda Evaluation Association. These efforts have complimented the establishment of government-wide national monitoring and evaluation systems.
In Uganda, the integrated national monitoring and evaluation system is driven by the national M&E policy for the public sector. The M&E system comprises of both a monitoring and an evaluation component. The monitoring component comprises of government’s self-assessment on development imperatives; community information sharing and accountability forums; as well as local government performance assessments. The evaluation component is managed under the Government Evaluation Facility.
However, the key question is, in the context of competing priorities in budget decision making and policy design, what determines evidence use and how is it influenced by the national evaluation system?
Factors influencing use of evidence in decision-making in the public sector.
Various factors influence the use of monitoring and evaluation evidence, including the levels of institutional effectiveness; political interests; levels of engagement with relevant stakeholders.
Institutional effectiveness: human and financial resources, policy and regulatory frameworks all affect the ability of an institutional to effectively carry out its mandate. Strong institutions have higher chances of yielding results. For example, where the structure allows for knowledgeable experts in M&E and where they actually exist, champions for evidence use should be available to support the process.
The policy and legal framework also plays an enabling role towards evidence use, for example, the M&E policy specifies the role of various stakeholders. It is imperative that the M&E policy is implemented. When for instance, key aspects of the policy are not implemented such as adequate financing for evaluations, the recruitment of M&E staff in Ministries, Departments and Agencies, the operationalisation of management information systems, it becomes difficult for the M&E system to enable M&E evidence use.
Institutions that work through sector-wide approaches tend to demand and apply evidence much more as compared to institutions that work in silos because more than one institution contributes to the results. For example, in Uganda, case disposal (access to justice) under justice, law and order sector consists of contributions from the Police, Prosecutions Office, Judiciary, human rights agencies, as well as legal aid networks. The scaling-up of alternative dispute resolution under The Back-log Reduction Strategy in JLOS sector can be seen as a collective effort informed by evidence.
Political interest: Political influence tends to take precedence in decision-making in situations where the public sector is faced with competing priorities. Politicians (Cabinet and Parliament) continually place pressure in the process of allocating of resources to improve service delivery and are interested in knowing how the resources were used which can be an opportunity for informing and improving decision-making. This is exemplified in a situation where there is evidence that “school feeding significantly increases enrolment rates and reduces absenteeism in developing countries” (Maijo, 2019) and it is in the interest of politicians to increase number of children in school, decisions will be made to increase school enrolment rates.
Engagement or participation of stakeholders in the generation of evidence: Buy-in is needed from stakeholders such as implementers of interventions, funders and policy-makers to ensure uptake of evidence. It is highly likely that recommendations from evaluations that were designed or validated with key stakeholders will be used. Effective stakeholder engagement facilitates evidence use and uptake (3ie, 2018). However, in practice the engagements start late, are not frequent and without champions.
Quality of the evidence: It is important that evaluation is done by competent M&E experts. Credible evidence positively influences the quality of evaluations which in turn will influence the uptake of the evidence by decision-makers (provided they know what credible evidence looks like).
What do we need to do?
It is imperative that consistency in the use of evidence amidst competing priorities is maintained. Decision-makers want “quick evidence” or timely information but the existing M&E structures do not answer these needs.
Governments should have robust and responsive national evaluation systems. A functional M&E system should facilitate evidence generation and use. The M&E system puts in place a platform that will provide politicians with information for decision making. The M&E agenda should be mapped to national strategic intervention priorities for instance; national development plans, ruling party priorities and the SDGs. The plan should be approved by top decision-makers and popularised by key stakeholders in the evidence ecosystem. However, in Uganda, this has been affected by the inadequate implementation of the government M&E system.
Having established national priorities, it is crucial that the generators of evidence provide relevant evidence for decision-making. Decision-makers should also be informed of the available evidence for use. In addition, there is need for tools to be available to facilitate the use of evidence; track actual use; and document changes that result from use of evidence. Experience from Uganda shows that high-level forums where evidence is shared are important avenues for immediate decision-making. For example, the Government Annual Performance retreats, Presidential Investors Round Table, one on one meetings for Permanent Secretaries have supported the use of evidence in budgeting and targeting of interventions in Uganda’s public sector.
Indeed, evidence matters when you are faced with competing priorities but what is important is the understanding of these priorities and using relevant evidence to make decisions that will address development priorities.
By Kachero Benjamin
Monitoring and Evaluation Practitioner, Uganda