The Struggle to Show ‘Impact’

To the Editor:

Sean Stannard-Stockton’s recent opinion piece (“More Than Money, a Lack of Research Hampers Nonprofit Innovation,” February 11) sheds light on an important point of distinction between a “promising” and an “impactful” nonprofit.

Impactful nonprofits are those whose efforts have brought about a sustainable result. This proof is determined through some form of independent evaluation—like a randomized controlled trial.

Very few organizations are impactful. Very few organizations have rigorous evidence that can prove that their programs actually work. Until a program is formally evaluated and shown to be the cause of a specific outcome, outcomes claimed are only indicative of actual impact.

As Mr. Stannard-Stockton mentions in his piece, Nurse Family Partnership is one of the few examples of an impactful organization.

When we look at evidence of impact, we are looking at the result of an organization’s past efforts. We are looking at what the organization accomplished yesterday. This is important—but past accomplishments don’t guarantee future success.

When we invest in an organization, we need to be interested in what the organization will do tomorrow.

Although Nurse Family Partnership has had great success its continued success relies on its ability to carefully manage its performance in the future.

Promising organizations—the next Nurse Family Partnership—won’t have the rigorous evidence that proves they work. If this is what the Social Innovation Fund is looking for, then there are just a handful of choices.

And if this is the case, then it might minimize confusion by losing the term “innovation.” Maybe refer to the fund as the “Social Impact Fund?”

However, if the Social Innovation Fund seeks “promising,” “innovative” organizations, then it should seriously consider Mr. Stannard-Stockton’s recommendations.

Look for organizations that base their programs on research about what works, that actively collect information about the results of their programs, that systematically analyze this information, that use information collected to make adjustments to their activities to continuously improve their performance, and that operate with an absolute focus on results.

In essence, look for organizations that have the systems, processes, and people in place to carefully manage their performance.

Organizations that do these things are most likely to evolve into the next Nurse Family Partnership. Those are the characteristics of promising, innovative organizations. Organizations that can’t, or don’t, manage their performance are operating blind and will never intentionally achieve impact.

Jeff Mason
Vice President
Social Solutions
Baltimore

Mr. Mason is chair of the Alliance for Effective Social Investing, a coalition of nonprofit groups and others that focuses on how to help donors identify effective groups to support.

In his most recent column, Sean Stannard-Stockton has this important issue exactly right, as does Katya Fels Smith in her excellent article, “Focusing on What Works: Federal Fund’s Approach Makes a Misstep.” (Opinion, January 14.)

Mr. Stannard-Stockton is saying that the Social Innovation Fund must not let the perfect become the enemy of the good. While it is helpful and appropriate for the fund to support “scaling what works,” the determination of what works can’t be set at an unattainable level of proof.

Mr. Stannard-Stockton’s thoughtful formulation about what the Social Innovation Fund’s grantees should do—“actively collect information about the results of their programs, systematically analyze this information, adjust their activities in response to new information, and operate with an absolute focus on producing results”—echoes that of another thought leader, Mark Kramer of FSG Social Impact Advisors, who noted in a discussion about a nonprofit study:

“Most of the performance measurement and outcome measurement systems do not track long-term outcomes, control for external influences, or use randomized control trials to prove that the outcomes are attributable to a particular organization’s efforts. Even so, they can provide valuable data that enable funders and grantees to improve their performance and increase their impact. Many foundations are expanding their range of evaluation approaches to include more timely, pragmatic, and forward-looking techniques, often without proof of attribution. Such techniques can help them better plan their strategies, implement their initiatives, and track overall progress toward their goals.”

The Social Innovation Fund is overcorrecting for the sins of the past, in which virtually no meaningful performance evaluations were conducted, by making new sins, requiring controlled experiments that simply cannot be undertaken in the real world.

Worse, such studies raise the bar so high that the results are often inconclusive, which helps no one.

Consider this analogous observation from a recent New York Times analysis: “For all the anecdotal evidence, it’s hard to show statistically that money has a large and systematic influence on political outcomes.”

If randomized controlled studies can’t establish a connection between money and political influence, what are the chances that the same techniques will be able to confirm and quantify the cause-and-effect relationships between well-conceived and capably delivered social innovations and longstanding, protracted social problems?

Mr. Stannard-Stockton is right to alert the Obama administration about the dangers of strangling innovation in the crib by attaching unrealistic strings to the Social Innovation Fund’s support.

The fund should encourage feasible and cost-effective performance evaluation, not penalize it. Mr. Kramer, Ms. Smith, and Mr. Stannard-Stockton have all shown the way to a sensible middle course that deserves serious White House consideration.

Steven H. Goldberg
Nonprofit consultant
Boston

Mr. Goldberg is the author of Billions of Drops in Millions of Buckets: Why Philanthropy Doesn’t Advance Social Progress, published last year by John Wiley & Sons.