For a number of reasons, the agencies responsible for evaluating social programs in both industrialized and developing countries have been much slower in adopting data science approaches than their colleagues who work in research and planning.
York, P. and Bamberger, M. (2020), p.1.
Big Data and Evaluation: a tale of two cities
Beyond the humanitarian sector, across the public policy and development communities, the conversation has been taking place for years. At evaluation conferences from Europe to Asia, from the Americas to Australasia, panels have been discussing the question of how to apply big data to the evaluation of public policy programmes for the best part of a decade.
There are plenty of reasons for evaluation offices to integrate big data tools in their work. Many of these tools promise to speed up the data collection process and increase its reach, thus freeing up the time of the evaluator to focus on the truly evaluative aspects of their work. Likewise, advanced data analytics can help evaluators to increase the depth of their analysis, evaluate complex programmes, and look for patterns that would otherwise not be possible to spot, let alone verify.
Yet, when attending an evaluation meeting on this topic, the potential gains to be made are invariably followed up with questions and concerns. Big data tends to arouse suspicion. It has a reputation as a noisy, messy, dynamic beast; unfit for the slow and careful analysis to which evaluation aspires. How can we ensure the validity of findings drawn from big data sources? What about the population biasing effects of working with digital data collection? How can we ever hope to filter out the “noise” from the “signal” in the big data multiverse?
As York and Bamberger (2020) note, this suspicion is not so widespread among the programme and policy research communities. So why is it any different for evaluators? And how can we learn from programme teams, information management specialists and even in-house MEAL staff who are looking increasingly to adopt the tools provided by the data revolution to improve their work?
What’s holding us back?
York and Bamberger (2020) argue that the problem boils down to something like a cultural divide. Evaluators and data scientists simply aren’t talking enough. Like all cultural divides, this is one enforced by a number of pre-conceived ideas about what life’s like on the other side. These boil down to disagreements about theory (roughly, how does one reliably measure sociological phenomena and their causes?) data quality and validity (what constitutes good data? what constitutes valid reasoning?), and selection bias (how do we sample a population with minimum biasing effect?).
Piciotto (2020) adds that the problems arise equally from both sides of the divide. The evaluators are still so absorbed in “needless methodological conflict” between qualitative and quantitative approaches, that they are unable to look up and see the new kid on the block. Likewise, the big data enthusiasts have been blinded by the sheer power of the numbers available to them; and remain unable to think about causal proof beyond simply correlating observations over vast datasets.
What can we do about it?
How can the divide be overcome? How can evaluators start making use of the potential that big data provides them, whilst mitigating the risks? York and Bamberger (2020) propose a fourfold approach:
- Establish new ways to build bridges between the two communities
- Integrate both evaluation and big data tools in capacity development programmes and learning
- Promote “landscaping research” to map the two ecosystems and look for synergies between them
- Provide seed funding for areas of collaborative research between evaluation and big data specialists
- Improving the ability of humanitarian agencies to measure their results in hard-to-reach communities
- Getting a better understanding of their impact when control trials are either unfeasible or inappropriate
- Focusing budgets on data analysis rather than manual data gathering
Each of these goals already absorbs considerable attention within the evaluation community. So any chance to tackle all three deserves time and investment to make it happen. But it will require engagement from all the major players in the humanitarian evaluation system: from donors and the UN system, to NGOs and the Red Cross / Red Crescent movement, researchers, evaluators and data scientists alike.
References:
- Piciotto, R. (2020). Evaluation and the Big Data Challenge. American Journal of Evaluation, Vol. 41, Issue 2, 2020.
- York, P. and Bamberger, M. (2020). Measuring Results and Impact in the Age of Big Data: the nexus of evaluation, analytics and digital technology. Rockefeller Foundation.