How Charity Navigator’s New Approach Affected a Social-Services Charity

Charity Navigator, a ratings group that many donors consult before deciding where to give, now awards charities up to four stars based on their financial health and their accountability policies.

But in a new approach it dubs CN 3.0, it has also begun analyzing whether organizations measure the results of their programs and let the public know about those results. Its assessments will not affect a charity’s star ratings until at least 2016, but the findings will be posted online for anyone to see.

The watchdog has so far evaluated more than 200 family- and children’s-services organizations on their results reporting, including Roca, a nonprofit in Chelsea, Mass., that provides educational and career services to young men who have been involved in gangs or criminal activities.

Roca, a four-star charity that gives priority to measuring what works, earned eight positive marks out of 14 questions—much higher than most charities that have been reviewed.

“We’re really committed to driving the outcomes and having an impact,” says Anisha Chablani, Roca’s chief knowledge officer. “When we started down this road internally, our mantra was, Either we should help young people to get these outcomes or we should shut our doors.”

‘Pay for Success’

Roca has been selected to participate in a “pay for success” pilot program operated by the State of Massachusetts, which will pay for the charity’s services only if it meets specific goals. It is now setting up a randomized controlled trial—considered the gold standard of program evaluation—to study how its graduates have fared compared with a group of similar young men who did not receive its services.

However, even Roca got all Xs from Charity Navigator in a section called “constituent voice,” which evaluates whether a charity is getting feedback from the people it serves about its effectiveness. Ms. Chablani says the charity does seek client feedback but has no formal process in place—and is pondering how to strengthen its efforts. She says it is challenging to find the right methods and timing in dealing with young men who often find it difficult to engage with Roca staff, especially at first.

“If we collected it in the first year of engagement, it would be highly skewed because we’re pushing them to change behaviors,” she says. “They might say, ‘I hate Roca, it doesn’t make sense,’ but it might be based on where they are in the program.”

Following are the check marks that Roca won in a CN 3.0 assessment and the documents Charity Navigator cited to explain its results.

☑ Do the mission, solicitations, and resources align?

Roca devotes 100 percent of its expenses to a program designed to help “high-risk” young men ages 17 to 24 move out of violence and poverty. Its fundraising materials accurately reflect that mission and spending breakdown.

☑ Is the causal logic plausible? [In other words, does the charity’s explanation of how it plans to achieve its goals seem possible or likely?]

Roca’s Web site outlines a three-phase “intervention model” to help its graduates get jobs and stay out of jail. In the first phase, the charity recruits young people into the program, offering a range of job-training and educational programs. In the second phase, it builds “transformational relationships” with its clients, offering more intensive programs. In the third phase, it offers formal vocational training and other career help so they can “move themselves out of violence and into jobs.”

☑ Is there some indication of how much of what action is required to produce what effects?

Roca’s 2013-17 strategic business plan fleshes out its timeline for achieving its goals during a two-year intensive program, followed by two years of follow-up services. It specifies how many times a week clients should meet with youth workers or attend various programs, depending on how long they have been enrolled. It also outlines a series of activities that clients should complete during the first two years, including two industry-recognized job-certification trainings, three mock job interviews, 10 work-readiness workshops, and four pregnancy-prevention workshops.

☑ Is there some indication that [its plans to achieve its goals are] based on reasonable evidence?

In the paper “The Evolution of Roca’s Intervention Model,” Roca explains how it has studied, adapted, and put into place elements of behavior-change programs that have proven effective. It says it has aligned its approach with the “Eight Evidence-Based Principles of Effective Intervention” that were identified by the Crime and Justice Institute, a research center at the nonprofit Community Resources for Justice.

☑ Are there specified measures (indicators) to be collected and a plan to do so?

Roca’s 2013-17 strategic business plan says its program is designed to achieve three things that will save the public money: reduce incarcerations of young men, make sure they comply with parole or probation conditions, and prepare them for economic independence through jobs and education. It highlights statistics that it has collected to show progress in those three areas: In 2012, 90 percent of the graduates of Roca’s two-year intensive program had not been rearrested, none had violated probation or parole, and 79 percent were on track to retain jobs for at least six months.

☑ Does the charity publish evaluation reports that cover the results of its program at least every five years?

Roca regularly assesses its performance and seeks outside evaluations. Charity Navigator specifically cites a report published by Chapin Hall, a policy-research center at the University of Chicago that the charity hired to conduct an extensive study of its impact—partly to prepare for participation in a Massachusetts program that will pay for Roca’s services if they produce specified results. The 2012 report outlined a strategic plan for measuring and reporting on the group’s performance.

☑ Are those evaluation reports based on recognized techniques to understand their results?

Chapin Hall’s report, prepared by its Center for State Child Welfare Data, outlines recognized techniques such as collecting data, strengthening continuous-quality-improvement processes, setting up statistical models for assessing impact, and establishing random and control groups to compare outcomes of Roca’s graduates with those of similar young men who did not enroll in the charity’s programs.

☑ Either in the evaluation report or subsequently, does the charity explain what, if anything, it is changing as a result of the findings in the evaluation report?

The Chapin Hall report outlines ways to conduct “formative evaluations,” or collect preliminary data so Roca can adjust its strategy if it is not achieving its desired outcomes.