Don’t Get Caught in the ‘Evaluation Gap’

It is widely conceded that billions (some say trillions) of dollars in international assistance were wasted in the 20th century. Now the same thing could well happen again.

Thanks to the leadership of Jeffrey D. Sachs, director of the Earth Institute at Columbia University (along with Bill Clinton and George W. Bush, the Bill & Melinda Gates Foundation, and Bono, among others), government agencies and private donors from wealthy countries are pouring more and more money into efforts to improve the developing world.

But nobody knows what works and is sustainable over the longer run. That is in large part because few meaningful efforts are under way to evaluate development projects so that “best practices” can be identified and copied on a cost-efficient basis.

Therefore, donors often support attractive-sounding projects proposed by charismatic leaders without insisting on rigorous evaluation of results. And the journalists who report on the success of some of these projects simply do not ask the tough questions.

This “evaluation gap” has been documented by a Washington think tank, the Center for Global Development (with support from the Gates and the William and Flora Hewlett foundations), and while progress is being made in trying to marry serious evaluation efforts to development projects, this effort is in its infancy and is still too small to make an important difference.

Mr. Sachs is certainly the best, and most successful, leader in the effort to alert the world to the continuing crushing poverty in developing nations where more than 1 billion people still subsist on less than $1 a day. And Mr. Sachs has been particularly successful in focusing attention on sub-Saharan Africa, where extreme poverty rates actually increased in the 1990s.

Moreover, his Millennium Villages projects are designed to demonstrate a successful holistic, “bottom up” strategy that is, in the view of most modern development economists, exactly the right approach to solving the world’s poverty problem, since 70 percent of the very poorest people live in rural areas. It is far preferable to the “top down” approach that characterized so much aid in the last century.

Mr. Sachs has attracted substantial donations from the Japanese government and $50-million from the philanthropist George Soros in support of his work. He has, in a single month, received laudatory coverage about the early successes of some of the Millennium Villages in The New York Times, The Economist, and Vanity Fair.

Unfortunately, none of those reporters (one of whom spent extensive time with Mr. Sachs in Africa) apparently ever pressed him on how he will be able to demonstrate that his projects worked and are worthy of spreading worldwide. Nor, apparently, did Mr. Soros insist on third-party evaluation of the effectiveness of the approach to be certain his large investment will have the desired payoff.

One reporter wrote that “the results [of the work in the Millennium Villages] are tested and monitored, [Sachs’s] goal being to prove that the same scientific model can be used on a grand scale to save the lives of hundreds of millions of people trapped by poverty.”

But who is doing the “testing and monitoring”?

Unfortunately, it is Mr. Sachs’s own Earth Institute at Columbia University, i.e., it is self-evaluation.

Those who care about closing the “evaluation gap” of the last century are properly critical of that traditional system, where the project managers report the results of their work without outside validation. The natural tendency to emphasize only the good news is too great for most project leaders. Moreover, most evaluation efforts are too short term to know whether the gains are sustainable after the outside resources are withdrawn.

Development-aid projects, if they truly hope to become models for others to follow, will need to invest meaningful resources in third-party, rigorous, long-term evaluation.

The great tragedy is that Mr. Sachs’s villages may really work — as someone who started working in rural Africa in 1962, I very much hope they do — but we will never be quite sure.

At least two other holistic “bottom up” village-development efforts in Africa are confronting the need for third-party, long-term evaluation — those run by the Hunger Project and the William J. Clinton Foundation. More projects should follow those leads. (In the interest of full disclosure, I consulted with the Robertson Foundation as they made the decision to provide the money for a five-year village-development effort by the Hunger Project in Ghana as well as for an evaluation of 10 years of the group’s impact by experts at the University of California at Berkeley and Yale University.)

Moreover, like the trials of new drug therapies, the “gold standard” of evaluation, designed by the Poverty Action Lab at the Massachusetts Institute of Technology and others, requires that the project sites be randomly selected so that they can be compared with similar villages that are not involved in the project (like placebos in drug trials).

Thus, donors and reporters need to ask three questions about all poverty-reduction projects, both in the United States and in other countries:

• Is the evaluation being conducted by a third party (not the donor or the project manager)?
• Have the beneficiaries been randomly selected?
• Will the evaluation be conducted for a sufficient period of years after the project ends to be certain the improvements are self-sustaining?

If donors and reporters systematically insist on answers to these three questions, they can help save billions of grant dollars in the coming years — both at home and abroad — by identifying projects that truly work and investing follow-up money in the most effective approaches rather than wasting dollars on unproven — but, frequently, “tried before” — projects.

William Cotter is the former president of the Oak Foundation, of Colby College, and of the Africa-America Institute. He has lived and worked in Africa and Latin America and serves as a consultant to foundations on international development.