Rigorous Evaluation Highlights Hits and Misses for Clark Grant Recipient

Citizen Schools started here in 1995, the brainchild of two idealists who wanted to improve the lives of middle-school

students by creating an after-school program that would connect youngsters with adults who were passionate about their careers. The charity’s co-founders, Eric Schwarz and Ned Rimer, said they would measure their success, in part, by how many participants enrolled in college or obtained good jobs after high school.

That measure was one of the few things that officials at the Edna McConnell Clark Foundation didn’t like about Citizen Schools when, in 2000, they awarded the charity one of the first grants from what would later become known as Clark’s Youth Development Fund.

The Clark Foundation, which started the grant program in the ashes of a failed attempt at improving large public systems, had no interest in lofty goals that could not be directly tied to a charity’s own programs.

Through the youth program, the foundation offers big money, long-lasting grants, and plenty of management advice to the select few charities it chooses to support. In return, it expects the charities to set clear goals for achievement and growth — and then hit the mark.

“How could an organization working with kids at the middle-school level be able to ensure that, four years later, things were going to look the way they wanted them to look?” says Michael Bailin, Clark’s former president, who started the Youth Development Fund. “Organizations come at foundations with huge goals and objectives, and most of them get set up for failure when they’re evaluated.”

Real-World Skills

Citizen Schools, whose main offices are on two upper-level floors of the Boston Children’s Museum, serves 2,000 students at 24 schools, including programs in six other cities in Massachusetts, and in California, New Jersey, and Texas.

Its unusual approach includes bringing in local citizens — such as lawyers, medical students, dancers, and investment bankers — to teach rigorous “apprenticeships” that give students real-world skills and greater confidence. Citizen Schools also spends up to 90 minutes per day working with students on homework and organizational and study skills.

In part due to Clark’s influence, Citizen Schools decided to hold itself accountable for influencing more-immediate results than what happens to the students after they graduate from high school. The charity has established 11 ways to measure the accomplishments of each campus, setting goals for students that include daily attendance at the program (Citizen Schools shoots for 90 percent of students per day), and maintaining or improving scores on standardized math and English exams (the goal is that 75 percent of students will maintain their grades or do better).

In September, Citizen Schools published a half-inch-thick evaluation report on the 2004-5 school year — a far cry from the first few years, when the charity measured its accomplishments almost exclusively through satisfaction surveys completed by students, parents, school administrators, and volunteers.

Aside from the surveys, Mr. Rimer, the charity’s managing director, concedes that the only evidence Citizen Schools has that its programs worked in the 1990s are isolated anecdotes about troubled kids who later did well in high school or college.

Yet while Mr. Rimer believes that the charity’s rigorous evaluation of short-term results is a step forward, he also knows that the program will only be successful if it has a lifelong impact on participating students. “I can tell you that in my sleep, I still think out an extra 10 to 20 years,” he says.

Clark has made three grants to Citizen Schools since 2000, totaling $7.75-million. In the past three years, Citizen Schools has received an average of $1.5-million per year from Clark, roughly 20 percent of the charity’s budget.

But both Mr. Rimer and Mr. Schwarz, the charity’s president, say the advice they have received from Mr. Bailin; Nancy Roob, Clark’s current president; and other members of the Clark team has been equally valuable. Mr. Rimer says he has a “development” folder in his e-mail where he keeps messages from foundation officials. But from the very beginning, he has kept Clark messages in a different folder: “strategy.”

Advice from Clark officials has included encouraging Citizen Schools to be patient with expansion. The foundation raised questions about the charity’s initial desire during discussions in 2000 to be in 30 states within five years.

“During business planning, they realized that they needed to better demonstrate the effectiveness of their program in Boston, while experimenting with franchising and affiliates on a limited basis,” Ms. Roob says.

The advice to move slowly was wise: Citizen Schools’ results to date are good, but not extraordinary, and it is tinkering with its approach to add a greater amount of academic support for students. What’s more, the charity recently switched gears on its preferred method of growth to retain greater control of each site.

MTV-Style Video

Here in Boston, students in the program meet three hours per day, four days a week, at 12 schools. On a recent weekday, four volunteers at Washington Irving Middle School, where 80 percent of the students are so poor that they receive federally subsidized lunches, were “pitching” their apprenticeships to Citizen Schools participants. Students get to choose which apprenticeship they want to take, and the early favorite, beating out apprenticeships in newspapers, photography, and financial management, appeared to be an offering from a young woman who promised to show students how to use video cameras and a karaoke machine to create an MTV-style video.

Roobvia Bernadin, a seventh grader at Washington Irving, took an apprenticeship a year ago from employees of Fidelity Investments, where she learned the key to money management:

“You’ve got to take care of your needs before your wants,” she says. She also learned the power of saving money; she has accumulated $50 toward a $300 bicycle she is eyeing.

“We do a lot for kids that is hard to measure, but that helps them be successful,” says John Werner, executive director of Citizen Schools Boston.

But for Citizen Schools, measurement isn’t only a way to improve its operations; statistics are also needed to show donors and school districts that the program is worth embracing.

Citizen Schools is undergoing two long-term assessments paid for by the Clark Foundation — one conducted by its own employees, and another by Policy Study Associates, a research firm in Washington. By 2008, the charity will have spent $1-million on the outside study, which is based on the group’s work since 2001, and roughly the same amount on its own assessment.

The evaluation by the research firm, which focuses on the Boston program, found that Citizen Schools is reaching students who are at risk of failure: 90 percent are from low-income families, and 95 percent are minority students.

Policy Study Associates also found that the charity’s 8th Grade Academy, an after-school and summer program that aids with the transition to high school, had a positive impact in getting students to choose an academically challenging high school and in getting them to 10th grade on time.

During 8th Grade Academy, students go on college visits, work one-on-one with lawyers on writing skills, and conduct research on which high schools in the city are best at preparing students for college. On the down side, Policy Study Associates found that participants in 8th Grade Academy didn’t outperform a peer group in areas like suspension rates and English and math grades.

According to its own research, slightly more than half of all Citizen Schools participants maintained or improved their test scores in math (54 percent) and English (59 percent) — mediocre results at first glance, but not so bad considering that, on the whole, students’ grades tend to drop in middle school.

“The glass half-empty measure is that we need to do better, and we can do better,” Mr. Schwarz says. “The glass half-full measure is that we have one of the most rigorous evaluations that’s ever been done in the after-school space, and we’re getting results that are more positive than almost any other after-school program.”

Issues of Control

After going through business planning with Clark and others in 2000, Citizen Schools originally felt it could best grow by encouraging the creation of affiliates, essentially licensing its program to school districts and community-based organizations, typically a YMCA. But some of the affiliates weren’t as committed to fund raising or growth as the charity would have liked, so Citizen Schools’ new approach is to maintain control of most new sites it creates.

“It’s our raison d’etre,” Mr. Rimer says. “For others, it’s just another program.”

In 2005, Citizen Schools closed two campuses — in Tucson (due to scheduling challenges and low student demand) and in Baytown, Tex. (for similar reasons, plus a leadership transition) — but the charity remains committed to its growth plan.

Citizen Schools plans to add sites in one new state per year through 2010. Discussions with Clark officials helped prompt Citizen Schools to take into account a state’s record of supporting after-school programs when deciding where to expand. This fall, Citizen Schools will open two sites in Charlotte, N.C. — its first campuses in that state — and at least six other new sites in states where it has existing operations.

In each state, Citizen Schools hopes to eventually have at least 10 campuses in two to three cities. Such state hubs will each cost about $2.5-million per year to operate ($250,000 per campus), and the charity believes a sustainable financial structure will require that 40 percent of the budget come from government sources. Federal spending on after-school programs rose sharply from 1999 to 2002, to approximately $1-billion, but has since stalled.

That means other foundations will have to play a large role for Citizen Schools to achieve its growth objectives. That’s a challenge for Citizen Schools and other charities the Clark Foundation supports because money doesn’t necessarily follow performance, as it does in the corporate world.

“In the nonprofit world, we’re trying to get a larger social return — it’s a different kind of bottom line, but a bottom line nonetheless,” Mr. Bailin says. “And there’s no place for these organizations to go to get funds for growth.”

That realization has prompted Clark officials to spend time trying to change the way other grant makers think about their work, although the effort has proven tough going.

In September, Clark sponsored a one-day conference in Seattle, organized by the Coalition of Community Foundations for Youth, that focused on how best to help youth-development organizations.

One session included discussion about how community foundations could support the national growth of Clark’s grantees, including Citizen Schools.

Some community-foundation officials voiced their concerns: How will my local after-school programs react if I provide money to bring in a Boston group like Citizen Schools?

Woodrow McCutchen — a Clark program officer, or “portfolio manager,” as the foundation describes the job — grew visibly frustrated with such arguments.

He suggested that community-foundation leaders tell the local charities that grumble about bringing in outsiders that Citizen Schools has a proven model for improving youngsters’ lives — credentials shared by very few after-school programs. In other words: Put up or shut up.

“The question is back to you,” Mr. McCutchen barked at the crowd. “Do you feel politically empowered enough to say that to your organizations?”

The question hung in the air for 30 seconds, but no one answered.