A flawed effortto evaluate charities

To the Editor:

I find the GiveWell and Clear Fund — its grant-making arm — business plans wonderfully motivated but flawed (“A Quest for the Best,” December 13).

Holden Karnofsky, the co-founder (who apparently has run into trouble), might have been a tad more cautious about his project. (See article on Page 54.)

It is amusing that hedge-fund experience — the least transparent sector in the financial world — is cited as the motivating force for a project promoting transparency through rankings of not-for-profits.

Laughs aside, the scope of Mr. Karnofsky’s potential influence is restricted to the subset of groups that are invited to apply for grants and meet GiveWell’s requirements. Building up the database of program information will be a drawn-out affair and thus limiting. Consider too that things change rapidly in the not-for-profit world and so yesterday’s rankings may say little about today.

Second, one has to wonder whether the outcome criteria utilized to promote cross-program analysis are valid, especially through self-reporting. In formal evaluation methodology this is called measurement error. Holding managers accountable for outcomes they may not reasonably control and redirecting attention away from management’s desire to describe their program outcomes with appropriate contextual and nuanced information represents what evaluators call additional “threats to validity.”

Maybe Holden should have sought more experience before seeking coverage in blogs and the media?

Here at Brandeis University, we teach our graduate students the importance of developing a knowledge-development plan for continuous learning in nonprofit organizations, perhaps under the auspices of a new kind of manager called a chief learning officer. Outcomes evaluation should involve key stakeholders, and cover appropriate and policy-significant outcomes found valuable by both the program and funders. All of this must be tied to genuine internal motivations for learning and growth.

While I applaud Mr. Karnofsky’s efforts to use research to create rankings, his methods may oversimplify a complex process to the detriment of his well-intentioned goals.

Andrew Hahn
Professor and Director
Sillerman Center for the Advancement of Philanthropy
Brandeis University
Waltham, Mass.

***

To the Editor:

We applaud GiveWell’s efforts to encourage charities to think about and report on their effectiveness.

However, we’d like to point out that experts don’t always agree about what terms such as “effectiveness” and “impact” should mean for charities.

In our experience, most of the charities we contact are eager to meet high standards. Sometimes it may simply take a while for a charity doing good work to reach full compliance with our standards. Charities that might not seem perfect at first glance may be making significant efforts to improve their operations.

Many charities want to satisfy donor wishes for information about charity impact. But data are not always easy to collect; once collected, they may not be easy to translate into results that would be meaningful to donors; and the evaluation process may be expensive to the charity. In other words — it’s not always easy, cheap, or quick for a charity to respond to donor requests for charity-impact information.

Of course, that doesn’t mean that charities are off the hook about self-evaluation. Just the opposite: Charity accountability is a very hot topic, and charities need to respond appropriately to donors who want assurance that their gifts will be well spent.

Who should pay for charity-effectiveness evaluations? Should donors have a right to see any and all of a charity’s self-assessment data? Who should determine what kind of impact assessments charities should do? What kind of impact should charities demonstrate to maintain their tax-exempt status?

Those kinds of difficult questions are being debated on Capitol Hill and in the philanthropic sector.

There are no easy answers to those tough questions. But the good news is that the evaluation process helps to strengthen charities. We strongly encourage charities to assess themselves, report on their effectiveness to their own trustees, discuss impact-evaluation issues with grant makers, and be as transparent as possible with donors when publishing their results.

Claire Rosenzweig
Chief Executive
Education and Research Foundation
Better Business Bureau of Metropolitan New York