Charity Evaluation: More of a Luxury Than Many People Care to Admit

To the Editor:

Allison Fine, in her opinion piece “Charity Evaluation: a Necessity, Not a Luxury” (My View, January 28), comments on how the “[evaluation] movement that started as a whisper has built into a crescendo of action, winning the support of charities and foundations.”

For some years now, as a grant maker and observer of the non-profit world, I have watched a growing number of large foundations, United Ways, and, yes, lots of consultants pick up this banner, urging charities to prepare “to prove they are effective” if they are going to succeed in an increasingly competitive not-for-profit world. Ms. Fine urges that “grant makers and grantees dedicate the necessary time, energy, and money so that evaluation becomes commonplace and not a luxury.”

The problem is that for the vast majority of non-profits, evaluation is a luxury. It is precious time, energy, and money used evaluating rather than providing services. The pinch is more extremely felt as more and more funders are demanding evaluation but refusing to pay for it. Why? Perhaps because funders think their money is better spent on services. And more often than not, I think they’d be right.

It is the wealthy few, the large foundations, the intermediary non-profits (various types of institutes, associations, and “think tanks”), and consultants that are driving an effort that, in my view, is fundamentally flawed.

As a field, we have failed to come to grips with two essential dimensions of the evaluation movement. First, we don’t discuss evaluation in the context of its opportunity cost — the dollars that will not be spent on services and programs in order to conduct evaluations. Second, and even more important, we don’t confront the fact that much of the information laboriously gathered is fatally flawed by what I have seen referred to as a “positivity bias” from start to finish.

On the first point, good evaluation is expensive. It needs to be professionally designed, properly executed, and objectively evaluated. Often, only longitudinal information would really “prove” a program’s effectiveness. Further, if a particular “problem” that a program is addressing gets worse while the program operates, how can we know the program wasn’t minimizing the decline?

Most programs are like little toy boats bobbing on an ocean of demographic, economic, and social trends that individual non-profits can do little to control. If we accurately capture the cost of good evaluation, we will (and should) often conclude that it is not justified at the expense of programs and services. On the other hand, we can always continue our current practice of pressuring non-profits to conduct lots of little “bad” evaluations which, of course, are cheaper. But even a little time and money spent on evaluation that is ultimately worthless is still too much.

On the second point, few evaluations are conducted to discover whether an effort had an impact; rather, they are conducted to prove that an effort had a positive impact. This is a fatal flaw. Charities want to believe their hard work is making a difference. Funders want to believe they made a good grant. Between the two, we can usually succeed in seriously undermining the value of any attempt at evaluation. Therefore, even if I accept the theoretical value of evaluation, which I do, the real-life end product seldom measures up.

I want to be careful not to overstate evaluation’s lack of value. I am all for strategically conceived and designed studies that provide different fields of non-profit endeavor with valuable research to inform how programs should be designed and implemented at the local level. We simply must stop pretending, though, that good-quality evaluations can be done by every charity in every case, nor should they.

We must also admit the pressure throughout the process to present and interpret information in the best light. I hope Ms. Fine’s research indicating that most non-profits undertake evaluation not to satisfy funders but to improve their own efforts is true. However, I am confident these same executives would admit pressure from board members, donors, and funders to achieve evaluations that prove that programs are working, which influences the way evaluations are designed and conducted.

Clearly, evaluation is essential to provide any competent non-profit executive, staff, and board with the management information necessary to run the organization and accomplish its mission. This must be integral to the philosophy of the organization’s leadership and culture — not something done at the behest of funders. But I fear that this is not the scale and scope of evaluation that proponents of the “movement” usually have in mind.

As a funder, I can’t find any form of evaluation that competes with the old-fashioned way: Know your grantees. Fund the ones that strongly impress you with their vision, dedication, competence, track record, and inherent organizational goal setting and evaluative practices. Common sense, the value of subjective assessments, and a more-realistic assessment of what can be measured, what should be measured, and what we can afford to measure has to be brought back into this movement if Ms. Fine’s confident assertion that “the rewards far outweigh the costs” can be supported.

Rose Meissner
President
Community Foundation of St. Joseph County
South Bend, Ind.