Measurement in the time of COVID-19

In the space of weeks, countless community-based organizations have had to shift what they are doing while being faced with reduced staff hours, fewer volunteers, disruptions to revenue, social distancing restrictions and other constraints.  Yet the demand and need for many services that non-profits and social enterprises deliver are high. New needs are emerging, and many programs are being introduced. Across the community – in health care delivery, social services, education, to name a few – we are witnessing a great deal of creativity and experimentation in (re)designing programs. 

---

Impact and performance measurement practices that rely on clearly articulated goals, a plausible theory of change, and attention to performance metrics attached to goals, are not particularly useful in this complex and rapidly changing environment.  What is needed are processes and information gathering that can support organizations to try out different actions that guide the direction of program development and redesign. Fortunately, there are management and evaluation practices that can help guide community-based organization in these unprecedented times. 

Here are a few suggestions:

1. Focus on creating an environment from which good things can emerge, rather than trying to impose a solution to bring about predetermined results.  When it was clear that schools needed to close, the Ministry of Education where I live left considerable latitude to individual schools and educators across the province to use a range tools and approaches to delivering on-line instruction.
2. Create a space to bring people together (even if this is on-line!) for structured reflection about what is being tried, and what people are finding so far about what is working and what isn’t.   Engage not only those who deliver programs and services but the beneficiaries as well as other stakeholders.  It is especially important to capture a diversity of experience.   We may be facing the same storm, but we are all in very different boats!
3. Be alert to the evolving context by staying on top of what is going on and reaching out to others. A shift in the operating environment can open up new opportunities and partnerships.
4. Move ahead by ‘proving, sensing and responding.’   This can be like walking into a forest and moving ahead so that a path forward reveals itself (and changing directions if it doesn’t).  In this context, you have to be ‘awake’ to your surrounding.  Monitoring and measurement is still important, but it has to be flexible.  You can’t miss seeing the forest for trees. 
5. Accept that we are moving ahead in conditions of uncertainty.   The recent re-opening of school this month in British Columbia frustrated many parents and educators.   Teachers and parents alike had to accept considerable uncertainty about what returning to school would look like.   But without actually moving ahead we will not learn about the direction we need to move in, given that it is increasingly clear that we have to find ways to live with the novel corona virus while minimizing its risk for months (or years) until a vaccine or effective treatment is found.
6. Consider using Developmental Evaluation to support program development and redesign.   This form of evaluation helps deliver feedback to support the development of new programming so that it can more accurately address the roots of a community’s complex needs.

At the same time that organizations on the ground have had to adapt and evolve new ways of working, it is critical that governments and funders are able to change performance measurement expectations and support learning by providing resources for evaluation and creating a culture that supports experimentation and risk-taking.

Great resources for learning more about some of the ideas brought out in today’s blog include:

• David J. Snowden and Mary E. Boone, A Leader’s Framework for Decision Making.  Harvard Business Review, December 2007. 
• Better Evaluation. Developmental Evaluation 
• Hallie Preskill, Tanya Beer, Evaluating Social Innovation, FSG.
• Mark Cabaj, Five Good Ideas: Evaluation in disruptive times (Video of Talk) Maytree Foundation