top of page
Search

Why evaluation feels terrible (Evaluation Series: Part 1)

In nonprofit life, we can feel like we’re drowning in numbers.


Funders always want more details on what we’re doing, when, and with whom. The annual report calls for data we’ve got to generate by filling in gaps of what we actually measured. The strategic plan tax comes due quarterly. We’re sending endless surveys to participants. Rarely does it feel like any of this is making our programs better.


Evaluation is a major source of heartburn for almost everyone I work with, in both my classes and my executive coaching, and I live that life too - my Google Drive is full of half-finished spreadsheets, and my inbox always has a request for something more, something else, something different.


How did we get here? Why does evaluation feel so heavy? Why does evaluation so rarely seem to contribute to things getting better? Evaluation feels like it’s been around forever, and surely we know what we’re doing, right?


Well, about that…


I came of age in nonprofits in the 2010s. It always seemed to me that evaluation was something we knew how to do. My whole nonprofit lifetime has had surveys and focus groups and reporting requirements. I’ve never known a nonprofit world where we could do something just because we thought it was right, and not have to check in after to see if it works.


I’ve been bought in - I believe in evaluation. Surely, if we measure things effectively, we will be able to prove what we are doing is the right thing to do. And if we can prove this, the funders of the world will have no choice but to rain dollars down on us. We will be able to afford whatever our program needs, and pay our staff well. Evaluation is the key to prosperous programs that work.


These are the promises of evaluation. These are the reasons we’re spending so much time, effort, money, and relationship capital to gather data, to put it into our trackers, and to condense it into reports. But it isn’t as simple as all that. We’re doing quite a few things we don’t need to do, and doing some things we should be doing incorrectly. So what’s the path forward?


First, let’s explore how we got here. Then, let’s talk about alternatives.

This is the part of the blog where I start sounding like Charlie discovering a plot in the mailroom.



Bear with me. Where we came from helps us to understand where we are.

If you had to guess when we started evaluating nonprofit programs, when would you guess? 


There are many ways we can trace the history of evaluation. It’s an ancient practice - tinkering and experimenting to see if something works and looking for data to prove if we were right - but it’s a relatively new science.


A scholar named R. Lance Hogan outlined a general history of evaluation, which helps to sum up many various origins into a logical progression of events.


Here’s a brief overview:

  • The Age of Reform: 1792-1900s

    • Largely in education spaces, quantitative measures began to be used to reform how teaching happened and to determine if it worked

  • The Age of Efficiency and Testing: 1900-1930

    • Educational evaluation matures, and objective-based tests become the standard for determining if learning is occurring in students

  • The Tylerian Age: 1930-1945

    • Ralph Tyler advocates for educational outcomes to be stated in behavioral terms, setting the stage for Bloom

  • The Age of Innocence: 1946-1957

    • In the aftermath of World War II, major funding is directed towards social good, and there are massive expansions of education funding 

    • Benjamin Bloom builds his “taxonomy” of learning objectives, which you’re likely very familiar with if you’re anywhere in the education space

  • The Age of Development: 1958-1972

    • Due to the Cold War and pressure to be better than the USSR at social services, the United States passes major legislation to boost education, and with it comes requirements for evaluation to ensure this major investment is working (we had to show we were better than the communists!)

  • The Age of Professionalization: 1973-1983

    • Universities get in the mix as the field becomes more professionalized - you can now, in some places, study things like evaluation methodology at school

  • The Age of Expansion and Integration: 1983-Present

    • The field of evaluation rapidly expands into most parts of life, especially those related to education and social services

    • Over the course of this period, grantors integrate evaluation into what they expect their grantees to be doing in order to justify getting those funds


Putting that into context, consider for a moment when the nonprofits you’re familiar with were founded, and where it falls on this timeline. Some examples:


A new picture emerges when we put these timelines next to each other. We’ve been doing nonprofit work for much longer than we’ve been proving that nonprofit work works. There’s a complicated legacy there.


I can’t say what exactly this means for every nonprofit, because there’s many other reasons why evaluation is challenging, but I do feel this historical disharmony between work and outcomes must be in the mix. I think it’s too grandiose to claim that this is why we so often start off on the wrong foot with evaluation, but I do think it’s in the atmosphere. 


  • Have you worked for an organization that predates evaluation as a science?

  • How often have you been a part of a program that started before you knew how it would be evaluated? 

  • How often have you seen programs that began because there was funding (but no strategy)? 

  • How often do you truly feel like you begin with the end in mind?


And, 40-some years of evaluation-as-a-science doesn’t feel like that long of a time to me. It’s not like in 1983 everyone said “ok, let’s make logic models!” and snapped their fingers to align programs to them. In reality, it’s taken a much longer time to integrate evaluation into how we conceptualize nonprofit work, and it hasn’t always been done successfully.


Many of the start-up nonprofits I work with haven’t even begun to think about how they’ll prove why what they’re doing is the right thing to do. Many nonprofits that do decide to implement evaluation measures don’t have the capacity to really research what is best, and instead find themselves shooting from the hip with a Google Forms survey an intern made.

There is a path to evaluation that’s done right, that’s not onerous for us or those we serve, and that encourages our programs and participants to thrive. 

We’re all starting from different places when it comes to evaluation. Some people in the world are scientists with deep expertise (If that is you, hi! Welcome!), but most of us are more near the status of novice. You know that evaluation should happen, and you have some ideas of how it should happen, but there’s still gaps everywhere.


These gaps have consequences. They keep us stuck behind computers instead of out in the field. They make our participants suspicious of why we're asking all these questions. They bog us down and point us in directions we know we don't need to go. So what do we do?


I’ll get into what I’d advise you to do in my next blog - how to make evaluation work for you.


In the meantime, think about where your evaluation legacy comes from? Where did your nonprofit start with evaluation? Who made the surveys you use? Who designed your strategic plan? Who made the spreadsheets, the trackers, the dashboards? If it wasn’t an evaluation scientist, all is not lost, but there is likely work to do. Stay tuned -

P.S. This blog is certified AI-free, and partly inspired by Tiny Experiments by Anne-Laure Le Cunff. Thank you to that book and author for helping me get my butt off the couch and into my office chair to write these ideas I’ve had for ages.

 
 
 

Comments

bottom of page