FRSB annual review reveals tiny rates of complaints – so why all the fuss?

Fundraisers must be absolutely delighted with the figures about complaint levels contained in the Fundraising Standards Board’s Annual Review, which was published last week.

The FRSB revealed that during 2007, only 8,434 complaints (any type of complaint) were lodged with the 165,000 charities in England and Wales about their fundraising practices. And, as the FRSB’s review says (though not their press release) this figure has to be put into context of the “billions” of contacts charities have with members of the public each year.

Of the 8,434 complaints, 13.6 per cent – or 1,147, were about face-to-face fundraising. The PFRA says that street fundraisers make about four million contacts a year with the public so the complaint levels represent 0.029 per cent of all contacts with an F2F fundraiser – or to put it another way, you need 3,448 contacts between a member of the public and a ‘chugger’ to generate a single complaint.

There’s even better news. Charities dealt with the vast, vast majority of those complaints themselves. In fact, only four had to be referred to the FRSB. So that means that only 0.048 per cent of all complaints made about fundraising to charities required FRSB attention.

And only one actually went to adjudication. So, out of all complaints made about fundraising in 2007, FRSB had to make a decision on 0.012 per cent of them.

What all this says to me is that charities operate their fundraising to a very high standard, that they know how best to communicate with their donors and supporters, and that fundraisers’ expertise minimises conflict with the public and this results in a relatively low number of complaints.

That’s not how the FRSB would appear to interpret these figures however. The press release heralding the publication of the annual review focused on the “significantly high” numbers of complaints made against direct mail (31.1 per cent), telephone (21.1 per cent) and F2F (13.6 per cent).

The first point is that “significantly” when used in the context of statistics has a very precise meaning: it doesn’t just mean “very big”. To be “statistically significant” means that a variation could not occur by chance but was driven by unexplained variation in one or more dependent variables.

So for instance, if complaints about DM were 31.3 per cent one year and 32.2 per cent the next and 30.9 per cent the year after, then those differences probably wouldn’t be significant because they would likely be due to chance fluctuations in the dataset. But if, in the fourth year, complaints were 44.8 per cent, then that isn’t likely to be due to a sampling error; it would be because something had happened to drive an increase in complaints and would thus be significant.

To assess significance, you need to compare datasets. As we have just one dataset – complaints for 2007 – how on earth does FRSB know that 31.3 per cent is “significantly high”? My guess is that over the years we’ll discover this is a pretty average proportion of complaints against DM. But who knows, we might find that 31.1 per cent is actually significantly low.

So when Jon Scourse comments that the findings show that “the practice of direct mail is causing concern and problems for many charities and supporters alike”, it doesn’t imply anything of the sort. Maybe DM is causing concern, but you can’t draw that inference from these figures.

This brings me to my next point. The annual review says that these complaints should be put in the context of “billions of contacts” charities have with the public, yet the comments in the press release signally fail to do just that.

We’ve already seen that complaints against chuggers run at just 0.029 per cent of all contacts with the public. Gordon Michie at Relationship Marketing estimates there are six million contacts with telephone fundraisers each year and complaint levels here also, coincidentally, run at 0.029 per cent.

But now we come to direct mail.

In 2005, according to the Letterbox Factfile survey conducted annually by the Direct Mail Information Service, charities sent 369 million items of DM. If they sent a similar number in 2007, (figures aren’t yet available past 2005) that means that 369 million items of direct mail generated 2,624 complaints. Or complaints ran at 0.0007 per cent of all DM sent. Or you’d have to send 142,857 items of charity DM to generate just one complaint.

Maybe in absolute terms, 2,624 complaints against charity DM is unacceptably high and does indicate a serious problem (perhaps we shouldn’t be satisfied until with have zero complaints) although you can’t come to that conclusion simply by number crunching. What number crunching can tell you though is that relative complaint levels against charity DM are incredibly low and the reason DM generates the most complaints is, quite simply, because charities do so much more if it than anything else.

So why does the FRSB maintain there is a problem with charity DM, and why did they choose to focus on that message in their press release rather than celebrate the excellent job fundraisers have done in keep complaints against them so low?