Intelligent Giving research is so sexed up it could have been done by Ann Summers

A couple of weeks ago, Adam Rothwell, director of ‘donor advice website’ Intelligent Giving, accused Joe Saxton of “sexing up” nfpSynergy’s research. The delicious irony here is that, while he was making those comments, Adam knew full well that he’d shortly be dressing up his own research in fishnet stockings and a peep-hole bra.

So-called ‘Intelligent’ Giving’s latest piece of mystery shopping ‘research’ into its bête noire face-to-fundraising is so sexed up, one wonders if the fieldwork was outsourced to Ann Summers.

It is truly execrable.

First, there is the embarrassing failure to comprehend when the solicitation statement is required to be made. IG’s researchers never completed a donation transaction so the statement was not required.

This however, did not stop IG coming out with their headline finding that chuggers were breaking the law by not making this disclosure.

That in itself is enough to condemn this piece of work. But there is so much more wrong with it than that.

One of the researchers, in interpreting the research, says on the IG website that the “pressure [to raise money] can lead to a fair level of harassment” and cites the fact that 15 out of the 50 chuggers mystery shopped “would not leave our researchers alone when asked”.

IG does in fact practise what it preaches about transparency and its methodology and raw data are available on their website. This shows that the mystery shoppers were asked to rate chuggers’ “harassment levels” on a score of 1-5.

Even this tiny part of their research methodology is flawed because IG rated their scale as 1 = very respectful and 5 = very harassing. This is a Lickert Scale but Lickert scales such as this are designed to measure a single variable. In this case, the scale should have read 1 = not very harassing through to 5 = very harassing.

IG however tried to measure two variables on the same scale: respectfulness and harassment. But the opposite of harassment is not respect, it is the absence of harassment; and the opposite of respectfulness is not harassment, it is disrespectfulness. It is perfectly possible to be disrespectful to somebody without harassing them (though the converse might not be true).

I make this point simply to illustrate that there were some very fundamental flaws in the way this research was constructed (and anyway, most researchers would use a Lickert scale that ran from 1-7).

But still, I know what IG was trying to get at so let’s apply the principle of charity and treat this 1-5 scale as the measure of chugger harassment levels that IG intended it to be.

So, what do you reckon the average score was for all 50 contacts with chuggers?

1.7.

That’s not very high, is it?

Twenty-seven chuggers (54 per cent) scored 1, the best score they could get for respectfulness. Another 11 (22 per cent) scored 2. So 76 per cent got pretty good scores for respectfulness. In fact, just one chugger scored 4 and one scored 5, indicating high levels of harassment, each one representing two per cent of the sample.

IG also rated chuggers’ politeness on a scale of 1-5 where 1 was least polite and 5 most polite. The average score was 4.72, with 40 face-to-facers (80 per cent) scoring top marks. Another 12 per cent scored 4 and the remaining eight per cent scored 3. No street fundraiser mystery shopped scored 1 or 2 – the lowest – for politeness.

So where does all this talk of harassment come from? From the fact that 15 chuggers “would not leave our researchers alone when asked”. (What I don’t understand though is why some of the chuggers who ‘harassed’ researchers by not terminating the conversation when asked did not score more highly in the harassment levels scale. Some of these 15 harassing chuggers scored 2 in that scale, which was actually quite respectful and not very harassing at all).

After being told by a researcher that he/she was not going to donate, one chugger said: “I haven’t had my first break yet, please sign up.”

I wasn’t there, so I don’t know, but I suspect he/she said it with a smile on his/her face. Perhaps someone other than an IG researcher would have been able to share the joke.

Another said: “We want to catch you now.” Another: “I wish I had a quid for everyone who said they’d donate online.”

I suspect these too were said with a smile or a tongue in the cheek.

However, that is not to exonerate all the fundraisers whom IG flagged as not terminating the contact and thus being in contravention of PFRA codes of practice, and some do seem to be serious breaches (you should take a look at the raw data yourself). It does though bring us on the most fundamental flaw in this whole piece of research.

Do you trust it? Do you trust that the researchers correctly interpreted the conversations? Perhaps they saw a guilt trip when none was there. Perhaps they interpreted something intended as a joke as a serious breach of the code of practice. We can’t know for sure, which is why it comes down to trust.

And I don’t trust the research for the simple reason that it was not independent and had bias built into it.

Having a question that asked researchers to rate a chuggers’ harassment level is simply biased from the start. ‘Harassment’ is a non-neutral, value-laden term. It should not have been on the question sheet because it prompted the researchers to look for harassment (the fact that they didn’t find it, despite being prompted to look for it, speaks volumes about the real lack of harassing behaviour from chuggers).

Worse than this is that the fieldwork was conducted by Intelligent Giving’s own interns, not by an independent market research company (which is what the PFRA does when it mystery shops face-to-facers). Corporately, Intelligent Giving has very negative views about F2F and, possibly subconsciously, they constructed a survey that was biased toward delivering the outcomes they were expecting. IG sent its researchers into the field with preconceived ideas about what they would find. That is astonishingly bad market research practice.

When the research didn’t deliver – through the harassment and politeness scales –what was expected, the analysis had to find other ways to support IG’s status quo ante view of F2F while downplaying the positive results of the politeness and harassment scales. IG appears to have forced the data to fit their predetermined conclusion, rather than draw an unbiased conclusion from the raw data.

In other words, they sexed it up.

As serious research, it is totally unreliable and cannot be trusted. In legalistic terms, it represents an unsafe conviction. It is a disgrace this was ever put into the public domain, much less promoted to the national news media.

IG would no doubt plead that they didn’t have the money to commission independent research. But that doesn’t get them off the hook. If they couldn’t afford to do it properly, they shouldn’t have done it at all.

I’ve been a critic of Intelligent Giving right from the time it began life as anti-chugging website The Charity Sleuths (http://thecharitysleuths.blogspot.com/). But with this ill-conceived, sub-standard, logically-flawed, self-serving, amateurish piece of research, IG has undermined its credibility with the fundraising sector to a far greater degree than anything I could say or write about them would ever do.