Measuring Nonprofit Results Is Not A Futile Task

To the Editor:

William Schambra seems to suggest that it is not possible for funders to require grantees to measure their impact in a way that makes sense for organizations (“Measurement Is a Futile Way to Approach Grant Making,” Opinion, February 10).

I strongly agree that a practical, user-friendly system is needed; I strongly disagree with his conclusion that this system does not exist.

Measurement is not the problem; the system of measurement required and the use of the information are the problems.

As Mr. Schambra rightly points out, there are a number of problems with most current systems of measurement. Three that I have also identified in my work with government agencies and nonprofits are:

• Data-reporting systems are often complex and use complicated language that varies from grant maker to grant maker.
• Many funders have not identified an impact that they hope to achieve with their money.
• There is no generally accepted, uniform way for outcomes to be reported.

Mr. Schambra identifies the solution, cautioning that foundation and government officials should not use measurable outcomes as a method for deciding on investments until a “simple, coherent, user-friendly system” can be developed to help nonprofits report on performance.

Throughout the span of my career as a service-agency administrator, nonprofit board member, government funder, and now a consultant with nonprofits, government agencies, and foundations, I have utilized a system that addresses these problems: results-based accountability, as described by Mark Friedman in Trying Hard Is Not Good Enough. The approach solves these problems by helping funders and organizations identify the precise condition of well-being that they hope to impact and to prioritize the most important performance measures that will provide feedback and useful information to improve their programs.

It also provides a simple method for managers to identify the three to five most important measures they will use to assess their programs from three simple categories:

• How much do we do?
• How well do we do it?
• Is anyone better off?

In addition, it uses plain language and a simple tool to help agencies translate their work into the language required by different funders, provides a format for organizations to tell the “story behind” their data, and uses a simple framework of seven questions to identify strategies and action plans to continue to achieve the desired impact of the organization. Working through these seven questions can be done in an hour and the answers written on one page.

The issue is not about measurement; it’s about making measurement useful and not a futile paper exercise.

Karen Finn
Results Leadership Group
Bethesda, Md.

To the Editor:

Mr. Schambra argues that the past 100 years of measuring philanthropy has resulted primarily in increasing the burden on nonprofits.

Surely Mr. Schambra can see that tremendous progress has been made with the help of evaluation.

Early in the 20th century, the Carnegie Corporation of New York funded an overarching evaluation on the state of medical schools nationwide. In response, medical schools improved academic training and rigor.

Measurement was instrumental in helping the Grameen Bank bring millions of families out of extreme poverty through microlending.

Finally, in 2005, a healthy body of data allowed Don Berwick and the Institute for Healthcare Improvement to prevent 100,000 deaths in 18 months. While there are many flaws with how evaluation has been done, calling it futile painfully brushes aside some of our greatest triumphs.

Mr. Schambra goes on to argue that because measurement has not given us a coherent science of grant making, it must be futile. This dangerously confuses research with evaluation. The purpose of evaluation is not to result in comprehensive theories but simply to know whether something is working.

Knowing whether a nonprofit is effective has not given us a body of knowledge on grant making. Alone, measuring the effects of a nonprofit does not tell us how to eradicate poverty or how to prevent homelessness. However, by measuring effects, we are able to help the organizations that are doing the best jobs, which is certainly not futile.

Evaluations, like those completed for Carnegie’s medical-school reports, the Grameen Bank, and the Institute for Healthcare Improvement, provide actionable answers to timely questions.

Meanwhile, research serves the role of acquiring and testing a body of knowledge applicable to a larger audience. What Mr. Schambra bemoans is the lack of a grant-making body of research, not the futility of evaluation.

Catherine Jahnes
Research & Evaluation Associate
Virginia G. Piper Charitable Trust
Phoenix

To the Editor:

As a longtime evaluation consultant, let me be among the first to agree with Mr. Schambra that mindless measurement is a futile way to approach grant making.
I’ve seen many foundations force grantees to spend scarce time and money to develop and report performance indicators that ultimately prove useless. They do this at the behest of board members and executives who are eager to demonstrate impact and quantify results. Only rarely, however, are foundations willing to underwrite the rigorous experimental tests that are needed for scientific proof of effectiveness and which, as Mr. Schambra rightly notes, are often extremely expensive, ethically questionable, and out of phase with decision making.

But more evaluation these days is not primarily about measurement; it’s about learning. Evaluation, when it’s done right, helps grant makers reflect critically on their strategies, ground their assumptions in trust, and receive feedback from a wide range of stakeholders.

While it sometimes demands effort from grantees, it repays that investment by giving grantees a stronger voice in the grant-making process. At its best, it helps break down the ivory tower that can isolate grant makers from the communities they serve, and that leads to better decisions.

Edward W. Wilson
Senior Evaluation Fellow
The Headwaters Group
Chicago