Recently some shocking research on actual Arbitron PPM panelists was revealed in this blog and elsewhere. The research was from Broadcast Architecture, and the video I recorded about it with BA President Allen Kepler shocked a whole lot of broadcasters (see it if you haven’t already).
While I didn’t receive any response from Arbitron, I did get one from Canada’s equivalent of Arbitron, BBM. The response is posted at Steve Kowch’s blog, and I welcome you to review it. The response is from BBM Canada President Jim MacLeod.
This is my attempt to analyze that response in the context of Allen’s research.
Mr. MacLeod comments:
These panelists signed a confidentiality agreement (as do BBM’s) so the fact they would even respond to Broadcast Architecture might say something about them. I would say they are far from typical of the people who take part in PPM.
Mr. MacLeod seems obsessed by the breach of confidentiality as if keeping a secret is more important than the integrity of the process itself.
The implication here is that Arbitron’s dirty laundry should be Arbitron’s alone – that the clients who pay vast sums of money to Arbitron are, somehow, not entitled to a comprehensive understanding of PPM dynamics unfiltered by Arbitron’s own point of view. The implication is, further, that PPM panelists who violate a confidentiality agreement that, I’m guessing, nobody even remembers signing somehow makes these folks both bad people and bad respondents.
Mr. MacLeod probably hasn’t seen the actual panelist interviews. If he had I think he would conclude that these are not bad people. These are people playing the game Arbitron has placed before them according to their best sense of the rules. They are doing their best, and they don’t deserve to have their integrity questioned by a stranger.
Meanwhile, if they’re such bad respondents, then perhaps they never should have been recruited for Arbitron’s sample in the first place.
Mr. MacLeod continues:
Thirteen people and seven homes is hardly a representative sample of the 70,000 or so in the Arbitron system and the fact that is all they could find in a major market might say something about the average panelist.
I think I’m speaking for every broadcaster when I applaud Mr. MacLeod for concluding that thirteen panelists are hardly representative of the total. Indeed, we will all remember that when we see critical Arbitron outcomes based on volatile swings caused by one or two PPM panelists. If thirteen are not representative, then how representative are the one or two listening to your station right now? Beware going down the sample size path, Mr. MacLeod. The sharp point of that argument will cut you.
And more:
They of course found the panelists most likely to respond to incentives … they paid $50 to each panelist!
This is an incredibly disingenuous argument from a research professional who should know better.
Every form of research across the globe that requires respondents to leave their homes to meet an interviewer in another place incentivizes respondents for this process. That’s true of your music tests and in every category of research under the sun. Not surprisingly, folks don’t show up for free. This is a generally accepted practice in all professional research circles – period.
Years ago, I did a project similar to BA’s – this one among diarykeepers. Like Allen, I invited diarykeepers to a central location, paid them to attend, and recorded the interviews. I asked them specifically whether the money they received along with their diary had any influence on their completion of the diary task. Most laughed at the question – because the diary incentive was $1.00.
There’s a big difference between incentivizing research respondents – and hiring them.
If Mr. MacLeod is arguing that paying the respondents produces a non-representative sample of radio listeners then he should take up that objection with Arbitron, who pays every PPM panelist and pays them a lot more than BA did.
Says Mr. MacLeod:
A Canadian panelist can earn a maximum of $10 a month from carrying a PPM.
That’s apparently a lot less than panelists in the U.S. are paid, and that alone could alter the U.S.-based conclusions from what one might find to be true among panelists in Canada.
Mr. MacLeod had a lot more to say, especially about differences between the Canadian and U.S. systems, so it’s worth a read.