Policy Evaluation in the Era of Covid-19

This is a timely book for policy evaluators, policy makers and politicians. They will want to know how did the policy evaluation process perform during the Covid pandemic? How were evaluation practices and values affected? This blog will give you flavour of the answers. To get a comprehensive view you need to read the full book, ‘Policy Evaluation in the Era of COVID-19’. The book has been complied by practitioners and established academic experts in the field of policy evaluation. They present a sophisticated synthesis of institutional, national, and disciplinary perspectives. The book appears in nine chapters, together with an introduction and an afterword. It has been edited by Pearl Eliadis, Indran A. Naidoo, Ray C. Rist.

Box A shows where copies of the book can be located.

Continuing beyond Covid-19

This book recognises that Covid-19 is not over. As we move on, we need to know what we have done well, what we have done badly and what lessons should we have learned. Much of what has happened will continue, and changed circumstances will continue to morph into a new and very different world. As Ray C. Rist puts it in his afterword  – “… we are still in the process of working through the implication of this vast challenge to our lives and our societies. The virus is not yet through with us” (Page 209). But it is not all bad news. The speed at which new vaccines were developed and approved was truly remarkable. The use of Zoom to allow people to continue to do business and to study during lockdown has been particularly positive. Admittedly the inability of officials to make site-visits did hamper the work of audits, evaluations, and accountability processes.

Evaluators at work

COVID-19 emerged quickly on the world stage. Evaluators had little time to prepare. They had worked previously in a fairly ordered world. Suddenly, they were expected to provide data quickly to enable governments to make big decisions to handle the pandemic. Jan-Eric Furubo puts this point very well in his chapter – “…decision-makers need answers, based on the best possible knowledge, about the causal mechanisms to make it possible to verify or falsify assumptions on which different policy options are based” (Page 39).

As part of the research for this book, the work of 26 evaluation bodies – national and international – was examined. That examination provided an interesting snapshot a what evaluators need to focus on to address COVID-19. The issues that were raised included the need to revise evaluation practises, to recognise that many ethical considerations and human rights issues needed to be handled with greater care, and that many of the tools and methodologies, used pre-Covid, needed to be looked at afresh. And while issues such as these were being examined, decision makers were putting increasing pressure on evaluators to produce quick answers based on the best possible information. In his chapter, Michael Quinn Patton, noted the challenges that evaluators face in order – “…to pivot, adapting evaluation plans and designs, and become capable of responding to complex dynamic systems. This means being prepared for the unknown, for uncertainties, turbulence, lack of control, nonlinearities, and for the emergence of the unexpected”. (Pages 193-194).

Some outcomes

Of a limited number of countries examined, some did better than others. Two countries are referred to in this blog – Canada and the UK. In the case of Canada, Maria Barrados, Steve Montague, and Jim Blian report, in their chapter, that that the work done by the Public Health Agency of Canada was unprecedented. There were notable successes, such as meeting diverse commitments under increasing tight timelines and successful collaboration with partners. Specifically, they pointed out that – “Evaluation responded by implementing innovative methodology to provide timely and informed suggestions for improvement”. (Page 47). By contrast Ray Pawson, in his chapter on evidence from the UK, points out that there were failures of leadership, and that – “COVID-19 generated a policy response that affected every sphere of social and institutional life. The result… was a frenzy of sticking plasters. Each measure, often perfectly valid in its own terms, interacted with other measures, producing emergent effects that were not and could not have been entirely predicted”. (Page 153).

Impact on Sustainable Development Goals

The Sustainable Development Goals (SDGs), adopted by the United Nations in 2015, were a universal call to action to end poverty, protect the planet, and ensure that by 2030 all people enjoy peace and prosperity. However, progress on most SDGs has been hindered by the pandemic, since they have not been priority areas for attention.  As Robert Lahey and Dorothy Lucks put it in their chapter – “The COVID-19 pandemic has, to date, significantly altered the world of evaluation and its use in supporting the SDGs… going forward post-pandemic, evaluators and the practise of evaluation need to adapt accordingly. A number of the changes realised during the period of the pandemic will likely ensure. It will not be “business as usual” for the evaluation community but will require a shift that may lead to a “new normal” (Page 117).

The Afterword

The afterword has been written by Ray C. Rist. He produced five key observations that need to be studied carefully as we move forward. He enters a note of caution, namely that the full impact of Covid-19 has not yet been evaluated. Accordingly, one needs to be aware that even though this crisis is not yet over, another crisis could be around the corner.

With those qualifications in mind, one should now consider this summary of the five key observations, as set out in Box B.

Final observations

One thing clearly emerges from this book. There is no such thing as “business as usual” after Covid-19. But that may be no bad thing.  Indeed, responding to Covid-19 can be seen as an opportunity. An opportunity for improving evaluation practises and procedures. An opportunity to achieve stronger linkages between different disciplines. There is no doubt that different areas of research can be enhanced if results are shared with other disciplines. It is incumbent on every evaluator to study what can be learnt about the way the pandemic was ‘handled’ and what changes need to be made in the way they will ‘do business’ in the future.  But, as the final two sentences of the book, warn us – “We need to stay alert and prepared. Because Covid-19 is an example of what happens when we are neither alert nor prepared”.

Tom Ferris, Consultant Economist.

^{Tom Ferris is a Consultant Economist specialising in Better Regulation. He lectures on a number of PAI courses and blogs regularly for PAI. He was formerly the Senior Economist at the Department of Transport, Ireland.}