Posted by on Jul 15, 2014 Views: 5584

The Surprising Truth Behind Beneficiary Feedback

Article written By Shuo Xie & Kris Sankaran

New research suggests that the feedback of nonprofit clients is less biased than many think.

In the business world, collecting user feedback has become a rule of thumb. It enables companies to improve products or services, and strengthen relationships with clients. The social sector also recognizes the importance of collecting and responding to client feedback, but critics cast doubt on its validity, citing “courtesy bias”—the tendency for people to give answers they think the asker wants to hear. We at the Stanford Statistics for Good Group (SSFGG) set out to understand whether courtesy bias truly exists among clients in the nonprofit sector and worked with the crowdsourced nonprofit review site GreatNonprofits.org to analyze user feedback via its nearly 200,000 nonprofit reviews. In particular, we wanted to learn whether low-income food bank and homeless shelter clients were more apt to give positive feedback because they are in need of the benefits provided by the nonprofits.

Reviewers on GreatNonprofits indicate which one of seven populations they most identify with: clients, donors, volunteers, board members, the general public, advisors, or professionals with expertise in the field they are reviewing. Using this data, we analyzed the differences between the ratings among nonprofit clients and the six other groups.

The illustration below shows how the ratings of different groups deviate from the overall rating of an organization. The more a curve sways from the 45-degree diagonal line, the more difference there is. For instance, on average, for the nonprofits that reviewers give five stars (out of five) 75 percent of the time, board members give five stars 83 percent of the time and the general public gives five stars 60 percent of the time. In the chart, each group for each nonprofit is represented as a dot in the background. The mass of dots on the upper-right corner indicates that all the groups rate most nonprofits highly.

Proportions of five-star ratings given by different groups, controlled for nonprofits. (All images courtesy of Stanford Statistics for Good Group, Stanford University)The study also revealed that clients who completed the open field question “If I had to make changes to this organization, I would … ” had the same response rate and response length as all other reviewer populations. The chart below compares the response rate of clients and other reviewer groups to this question, and illustrates that clients are not more reluctant to provide suggestions to nonprofits.

Difference in suggestion response ratesWe also compared response length. The plot below compares the mean number of characters in reviewers’ responses, per each nonprofit, between clients and other reviewer groups—similar-sized areas above and blow the x-axis indicate that responses from both groups were about the same length.

Difference in average suggestion lengths.Our group originally hypothesized that clients served by food banks and homeless shelters would be more apt to provide higher reviews, because they need and rely on the immediate services those nonprofits offer; we expected a stronger courtesy bias in this group. But we found that the contrary was true: Compared to clients of other kinds of nonprofits (such as environmental protection and education), clients of food banks and homeless shelters rated the nonprofits that serve them significantly lower than their peer groups. The table below shows homeless shelters and human services versus clusters of other nonprofit types.

Mean ratings within roles across organizations.

Recommendations

So are client evaluations for nonprofits reliable? Our research indicates that they are, and that online reviews can help provide nonprofits with a true and honest picture of its performance. Here are our recommendations for organizations seeking this kind of feedback:

  • Embrace third-party platforms or methods to help avoid courtesy bias.
  • Invite clients, staff, donors, and others to review your services.
  • Don’t be afraid of what you may find out; use the feedback to improve your services.
  • When you receive feedback, respond promptly and publicly so that others know you’ve heard them.

A great example of an organization doing this is Mission in Citrus, a Florida-based nonprofit that offers homeless and shelter services and has successfully collected and leveraged reviews from clients. Executive Director James Sleighter notes, “When we ask clients after a few weeks, they gladly do the reviews. Clients who depart often leave reviews. No one is forced.” When time permits, Mission in Citrus also shares client reviews on its website, on Facebook, and in newsletters to attract others who need help. By actively sharing client reviews publicly, the organization communicates that client opinions matter.

Shuo Xie is a master’s student in statistics at Stanford University and an active member of Statistics for Good. In her spare time, she serves as a nonprofit consultant at iJoin Social Innovation Consulting.

 Kris Sankaran is currently at Stanford University pursuing a Ph.D in statistics. He dreams about the potential for rigorous quantitative thinking and inference from data in the 21st century.