Evaluations Can Be Dangerous

“Sweet are the uses of adversity,” wrote Shakespeare, “which, like the toad, ugly and venomous, wears yet a precious jewel in its head.”

Judging from the interest several foundation leaders have shown in discovering and publicizing their most troubled grants, plenty of bejeweled toads must be lurking in the stacks of evaluations that grant makers have long been commissioning. But the gap between wanting to learn from mistakes and putting their lessons to good use is not easy to bridge.

Just how willing most foundations will be to uncover flaws in their efforts is itself uncertain. Foundation watchers, such as Joel Fleishman of Duke University and Phil Buchanan of the Center for Effective Philanthropy, as well as foundation presidents, such as Paul Brest of Hewlett and James E. Canales of Irvine, have recently become more vocal about the value of doing so.

And this month, the McCormick Tribune Foundation and the Center on Philanthropy at Indiana University — where I work — will hold the 4th National Philanthropy Summit to gather scholars, nonprofit leaders, and business executives to figure out how grant makers and their recipients can best demonstrate their accountability.

But virtually all the publicity about mistakes that were made has come from large grant makers, with professional staffs and ambitious agendas. While those grant makers spend a lot of money and are influential, they are hardly typical of American foundations, most of which are smaller and do not have staffs.

Indeed, for many of the small foundations, promoting social change on a national or global scale is apt to be far less important than consistently supporting a variety of local causes. While they do not want to waste their money, they are not likely to be meticulous about insisting on proof of results. Providing support to worthwhile groups might be all they hope to do, a goal that makes up in immediate usefulness what it might lack in accountability.

Those who favor more rigorous evaluations tend to regard this type of grant making as too modest, especially for foundations, which are supposedly meant to institutionalize — and thereby improve upon — purely charitable giving. Yet, with grander goals come greater challenges.

The well-known problems of measuring change are one kind. Results may take time to show up, especially in fields such as health or education, which often require many years for a program to run its course.

Just like investors in businesses, foundations will have to be patient for signs of progress or failure and willing to make long-term commitments, even if such signs are absent. They must also resist pressure from journalists and others who will want to know why, with all the money that is being spent, malaria (or some other problem) has not yet disappeared.

Evaluations are also costly. Foundations that are serious about learning from their mistakes will have to be willing to spend large sums on consultants and researchers, not to mention data collection and analysis, rather than on the programs that captured their attention in the first place. More-ambitious goals — improving education in American cities or reducing poverty in Africa, for example — require even more money to assess properly.

Even then, the findings are likely to be ambiguous, at best. This is partly because social changes do not occur in a laboratory, but in the real world, where many other changes are going on at the same time. Figuring out which results are due to a foundation’s efforts, and which to other causes, is no easy task.

In all likelihood, even the best-financed foundation programs can usually be expected to make no more than a small impact on the problems they are trying to solve.

In education, for example, countless studies have shown that a child’s upbringing, peer group, and perhaps even genes have more to do with academic success than what happens in the classroom, which is what grant makers mostly have the power to influence. Efforts to reduce AIDS, improve the environment, or bring about world peace face exactly the same problem.

If they are to learn from failures, grant makers must first have a realistic sense of what they can reasonably count as successes, which may often mean limiting their aspirations.

Even when results are clear, making use of them can be controversial. If success has many fathers, failures usually have many explanations, each of which is likely to suggest a different route toward improvement.

Moreover, even poorly performing grant programs have their supporters, who will resist efforts to point out their flaws, let alone change their direction in significant ways. Government agencies are stocked with evaluations of ambitious social programs that have done little, yet keep going year after year.

With no real consequences for failing themselves, grant makers may be just as reluctant as bureaucrats to take on those who view the results of assessments as a threat, not a learning opportunity.

Such a reaction is by no means short-sighted. Evaluations can be dangerous, not only to the programs that are being scrutinized, but also to the grant-making enterprise itself.

Large-scale philanthropy is built on the notion that foundations can act as investors in society, using their resources of money and skill to focus on important problems that others are unable or unwilling to solve. That is why donors get special tax breaks for creating them, face little public oversight, and are required to spend no more than a small share annually on charitable activities.

But if their record turns out to be strewn with highly publicized failures, those advantages may come under renewed examination. Whatever opportunities for improvement they may provide grant makers, a steady stream of mistakes — especially in very visible efforts, like the Annenberg Foundation’s school challenge — may undermine confidence in what foundations can achieve, not only among policy makers, but also among potential donors, who might seek other ways of providing social benefits, such as through less-ambitious programs or investing in for-profit businesses.

On balance, the growing willingness of grant makers to confront their failures, rather than simply proclaim their successes, is probably a useful step toward reducing the exaggerated sense of how much multibillion-dollar endowments can do.

But, to paraphrase the 19th-century humorist Josh Billings, it is not ignorance that hurts, but finding out that what we thought was true just “ain’t” so. As foundations learn more about what “ain’t” so, how they react to the “hurt” will determine how useful such knowledge will be.

Leslie Lenkowsky is professor of public affairs and philanthropic studies at Indiana University and a regular contributor to these pages. His e-mail address is llenkows@iupui.edu.