An Unexpected Event Changes an Experiment

To the Editor:

Inspired in part by a study on matching gifts reported in The Chronicle (“Bigger Matching Gifts Don’t Produce More Donors,” June 15), a charity focusing on women’s social issues conducted a test this summer in its annual appeal. The charity had received a matching grant from a major donor to try to encourage others to make large donations. The terms of the grant were that the donor would match contributions of $500 or more.

The charity decided to test the match in its general appeal, but its experience offers a cautionary tale in how to think about results. Since most gifts in response to the annual appeal are typically well below the match threshold of $500, it would indeed be interesting if the match had a positive impact, especially among donors with no chance of qualifying for the match. It would suggest that donors respond to matching grants for reasons other than to magnify the size of their donations.

The charity conducting the test took great care to randomly divide its list in two. Analysis showed that the two groups were statistically the same in all observable characteristics, such as past giving, geographic location, and gender. It sent an appeal describing the match offer to one group and a similar letter that did not mention the match to the other. They then compared the response rates and overall revenue generated for the two versions.

Surprisingly, the results showed that the match letter significantly underperformed the control letter. The test letter generated 42 percent less revenue per solicitation than the control. Could it be that the high match threshold of $500 actually discouraged people from giving?

Or, was the true explanation for the difference explained by the timing?

The charity took great care to ensure that the test group and control group were statistically the same by randomly assigning people on their list to one of the two groups. The organization handles its direct mail in-house, and this was its first experimentation with randomized testing. Because of limited capacity and lack of experience, the charity mailed the two versions of the letter eight days apart.

The Israeli-Hezbollah war happened to escalate between the times that the two letters were received. The control letter was mailed on July 11. The next day, the war began.

On July 19 the match letter was mailed. On July 25, about the time the match letter was beginning to arrive in the mail, the United Nations made an urgent appeal for humanitarian assistance to Lebanon.

An alternate explanation for why the matching grant letter did so poorly relative to the control letter is that the conflict in the Middle East changed donor behavior after the control letter was received but before most donors received the match letter.

The lesson from this case is that in addition to randomly assigning recipients to either the test group or the control group, all letters should be mailed at the same time. In this example, the explanation was relatively clear, but in other cases it may be less so.

Whether the explanation is known or not, mailing on different dates introduces a second variable, and you may never be able to distinguish between the variation in dates and the variation you intended to test.

The beauty of a well-executed randomized test is that even an event powerful enough to alter donor behavior would not affect the results.

If control and test groups are statistically the same (i.e., randomly assigned) and the direct-marketing piece is sent to both groups at the same time, then external events will affect both groups the same, and any difference will be due to the variation in the two versions that you are trying to test.

In this case, if both versions had been sent at the same time, the charity would now know whether it was the match that made the difference or not.

Doug Parkerson
Fund-Raising Research Director
Innovations for Poverty Action
New Haven, Conn.

Mr. Parkerson also directs the charity’s efforts in Bolivia.