Radio Ratings are in a Heap of Trouble


If you can’t trust the Nielsen ratings in America’s largest radio market, then where can you trust them?

A few days ago, Nielsen delayed the release of May ratings for Los Angeles when “inconsistencies” resulted in the need for more “quality control reviews.”

Since then, the problem has only deepened. From the LA Times:

Nielsen has been attempting to verify that individuals who participated in its sample audience panel were truly independent and did not have ties to any radio stations or radio personalities that were being measured. The ratings agency said Wednesday that it needed to “remove a household from the panel for not meeting our quality standards.” Radio industry insiders said such actions were unprecedented. One person said the move marked the first time in more than a decade that the release of radio ratings in Los Angeles had been postponed due to such concerns.

One single household has this scale of impact? In LA?!

So what’s really going on here?

The rumor in the radio industry is that this particular household had media ties and the usage recorded was significant enough to skew the overall ratings for the market.

Indeed, how could it be otherwise? Because if one household mattered so little to the overall results, then why would Nielsen bother to hold back the ratings for the entire market in the first place? The very withholding of the data implies that one household has the potential to skew the whole in a significant way.

Ponder that for a moment:

One household in Los Angeles has the potential to meaningfully skew the ratings for America’s largest market!

And if this one household can do that in LA, then can’t any one household in any market do it any time?

What does this say about the validity of the metrics? Why is one goofy household with outlandish listening any more goofy than any other goofy household with outlandish listening simply because it is affiliated with a local media company?

Do we see this same impact for online radio metrics?

I set out to find the answer, and the results will surprise you.

With the permission of Pandora, I gathered their webcast metrics listenership data for April in the LA MSA among all persons M-F 6A – 7P.

Pandora’s total AAS (Average Active Sessions) in LA during this period was 119,111.

What happens when we pull out the single heaviest listener in the audience? AAS declines almost not at all to 119,102.

What happens when we pull out the heaviest 10,000 listeners from the audience? AAS declines to 109,072.

In other words, you barely dent the overall online radio listening averages when you take out the heaviest listening 1, 10, 100, or even 10,000 listeners!

What does it mean when one household corrupts the PPM data for America’s largest market but 10,000 people can’t do it for online radio?

Update: Apparently there are two households involved here. Not that that changes my point in the least. Indeed, if not for the first problem household it’s unlikely Nielsen would have ever found the second. By the way, do you suppose these are all of them? Do you suppose that in all the PPM markets in America there aren’t media-related squatters hiding their PPM devices right this very minute? But even if there aren’t, now the lesson is clear: One or two households in America’s largest market can can quite unfairly shape millions of dollars in buying decisions, all built on a house of cards paid for by broadcasters and brought to you by Nielsen. Well done, Nielsen. Well done, radio. Well done, advertisers. Let’s make your clients proud!

* = required field
  • Dave Paulus

    and this is a household that just got BUSTED. I’ve had three members of my staff approached in the last 6 months to carry a meter..they said no, of course…but it does make you wonder a LOT. Amazing story.

  • paulgoldstein

    Important observations Mark, thanks.

    Pandora, Apple and Spotify R over the moon about this. Not only is broadcast radio’s performance in online listening extremely weak but as advertisers compare audience measurement reliability; bcast radio’s “estimates” via Nielsen Audio vs. online radio’s actual numbers, it’s not difficult to see why the phrase “disrupt or be disrupted” is more true for bcasters than ever before. https://drive.google.com/file/d/0B_pxKkiQlibyVHJqbnZSRUZmZ0E/edit?usp=sharing

  • The point here is that Nielson’s methodology failed them and their clients. It is being corrected to the best of their ability. Why Internet radio would even be mentioned in the same breathe is my question. Why would you so quickly attempt to draw a comparison Mark? It has nothing to do with the news of the day. AAS is a farce. AAS to AQH is an even bigger farce. Not having verifiable data on your audience (because unlike Nielson data it is not audited) is the biggest farce. I’m speaking of Pandora! AAS is devised by taking the total listening hours (TLH) and dividing them by hours against that time period. If you don’t prompt against 80% of your consumer base (mobile) how do you pretend to come up with a number that would be even remotely accurate. Have you asked Triton about this issue? I have and the answer was amusing. “We drop the lows and the highs”, they tell me. Lows were defined as under 3 minutes and highs over 5 hours. Wow! Starting with zero accuracy means your end result can never be accurate. The fact is, both have issues but Pandora because of an original business model that failed desperately needs to find revenue. P is a good consumer product but a bad ad vehicle.

    To give you a clearer picture of Pandora’s issues see slides below……


  • Peter, I know you feel strongly about this topic, but the Triton metrics have been accredited by the Media Rating Council, the same outfit that accredits the ratings of terrestrial players like Nielsen.

    As to your question: Why did I include the reference to online ratings…The answer was in the post. Because 10,000 of the heaviest listening sessions don’t materially change Pandora’s ratings, but TWO households in PPM materially change the radio ratings.

    I don’t think we should be so quick to excuse this faux pas by Nielsen because it is systemic not just to ratings in LA but to PPM ratings everywhere. The idea that they’re cleaning this up to the best of their ability is small consolation to the stations like yours whose lifeblood depends on the presumption of accurate measurement – a presumption which is more of a fantastic dream than anything practical in this PPM environment.

  • thegreatonymariani

    Mark, if this happens in the PPM world, what do you make of the old diary method we still have in some markets in Canada?

  • And in the US…

    You can say that PPM is more accurate in the sense that listening is measured passively, but it’s clearly more “work” to maintain the meter-keeping process, and that “work” distracts from the “accuracy” of PPM. And with a panel setup and smaller effective samples exposed to ever larger media offerings, one really has to ask if we’re getting more accurate or simply more granular.
    Meanwhile, the average diary is strictly a once-a-day recreation of listening from throughout the day.
    So both are abstractions. But only one pretends to be “accurate.” The other only argues that it’s the best measurement money can buy in those markets.

  • Mark- Because Tritons metrics have been MRC accredited unfortunately has nothing to do with the accuracy of the measurement. It meant something in the past but I would question its worth in this day and age. That’s neither here nor there. Again, comparing a product that delivers a music playlist vs. a radio station is apples to oranges. So, that was my question. Why the comparison? The 2 have nothing in common or to do with each other. It’s like comparing the accuracy of an umpire in baseball vs. a referee in football. Not sure why you would have gone there instead of staying on the Nielson issue, which is the issue.
    If you were privy to my conversations with Nielson you would quickly realize that I was/am far from excusing them. I waited for an extra week knowing that my ratings situation was going to change for the better (We already had 3 great weeks reported for May). As far as the metrics go for Pandora they are far from accurate. When you don’t know who your listeners are, where they are, whether they are listening or not, what do you really have as far as accuracy goes. Nothing! Nielson has their issues but please don’t misinterpret what Pandora or Triton are telling you as far as their accuracy goes. Remember, “throw out the highs and the lows”……….big spread there!

  • Peter, I think you’re operating under the assumption that Pandora and radio are in different categories. I am operating under the assumption that they are not.
    I think my assumption is closer to that of the end user of content, if not ratings services.

  • I think you’re right. However, people for years have been exposed to IPods which is an offline version of Pandora (minus the algorithm). The IPod didn’t have to sell commercials (not that they could have delivered them anyway) because they executed their original business plan successfully. Please understand though that an attack on radio by an internet playlist trying to sell commercials will not go unanswered by me. Their metrics are inaccurate from a buying perspective.
    I do think you are right in that everyone is competing for the end user of content. Selling commercials and audience measurement is a completely different story.

  • This is Pandora’s own admission to what some in our business might consider their cume figure- “The number of active users may overstate the number of unique individuals who actively use our service within a month as one individual may register for, and use, multiple accounts.” Source-Pandora’s 2013 annual report. So, what is the number? 75m, 65m or 30m? Not a ton of accuracy here at all. Add that to not knowing if listeners are listening or not due to the lack of prompting and you don’t have a TSL number. Add that to data that is unverifiable and you have nothing to target. Now compare those facts on what Pandora really knows about their audience to what Nielsen knows and that’s your comparison.

  • Peter, two things…

    Peter, you don’t actually believe that many people have multiple Pandora accounts do you? What would be their incentive to do so?
    I honestly feel that throwing stones at Pandora’s methodology when the current headlines reveal just how broken the NIELSEN system is is misplaced focus. It will be hard to get the world to focus on peculiarities with Pandora’s measurement at a time when it is revealed just how extraordinary the impact can be of only TWO households in America’s largest media market.
    The truth is that Nielsen’s problems with PPM are systematic problems with the entire platform in all markets. The sooner we face up to that, the better.

  • Peter, I’m no expert in Pandora/Triton metrics, so I forwarded your questions to someone who is: Triton’s Rob Favre. His reply follows. Rob’s not in this thread, so you should probably reach out to him directly if you want to continue the dialogue…

    Peter – You seem to have a number of misconceptions regarding our measurement service. Let me address a few here:

    1. AAS is an MRC accredited metric that is audited by Ernst and Young annually. You are correct that AAS = total listening hours divided by the number of hours in the day part. It includes ALL listening hours measured in the day part regardless of device (desktop, mobile, embedded).

    2. AQH for Pandora is an exact match to the NLSN metric in that it requires five minutes of listening during a fifteen minute period. Not only is this the textbook definition, but our AQH metric is also MRC accredited and audited annually.

    3. We are a ‘census’ based measurement service. We do not drop the “highs or lows” to arrive at our metrics. They are an exact accounting of listening.

    4. Where you might be confused is what we call our “one minute rule”. Unless a listening session lasts a full minute, we don’t count it. This is one of our quality measures employed to remove non-human based listening such as bots and spiders.

    5. In addition to Pandora, we also measure many publishers (look at our monthly rankers). All of those companies rely on our MRC accredited and audited metrics to support their monetization efforts.

  • Mark- I’m not sure about duplicate accounts and neither is Pandora. That’s the point no one knows, which = inaccurate measurement. They felt that is was a big enough issue that they had to place it prominently in their annual report. By the way, you brought up the comparison between the 2 in your article, not me. As you have seen from me, Pandora’s ratings are exposed as being grossly inaccurate in all areas. We certainly don’t need to cover that again. There were 2 issues in the Nielson May report. 1 was a PD from Univision carrying a meter. He had his wife and 2 kids carrying meters as well. He’s been fired and the ratings on the Univision property have been adjusted. The other issue was another Hispanic household with a number of meters and that household after being interviewed for the 2nd time was non compliant. They have been pulled out of the May report. April will be re-issued tomorrow. The general market went unchanged as none of the listening occurred in that arena. In the end, less than 1% of the meters were tainted. Nielsen’s bad….no question about that.
    I would love to see Nielsen increase their sample. I would also love to see Nielsen figure out the ear bud issue as on line listening growth to radio stations is out pacing that of pure play. It would be nice for stations to get some of that recognition. Like in life, nothing is perfect unfortunately.

  • The important point that I have been making for 3 years now is that Pandora is not radio. Certainly from a end user standpoint (UX) and most certainly from a buying standpoint. The 2 are completely different products and should be treated as such. If Pandora wants to go after radio dollars to offset a failed original business model that’s fine.When they (Pandora) show up at Radio’s direct clients and agencies they will have to explain themselves more and more as word spreads of their inaccurate reporting. I have spoken to Triton Media who was never able to answer many of my questions.

    1. One element of devising an AQH number is Cume. You need the cume figure to be unduplicated in order for it to be a cume figure. Through Pandora’s own admission they don’t have any idea how much duplication there is in their “75m” monthly users. Not my assumption or words but theirs in their annual report as mentioned in an earlier post. So, this first element is completely inaccurate.

    2. The other element is TSL. Again, they have a completely inaccurate way to calculate how long listeners are listening, if at all. In Tritons own words “we throw out the highs and the lows”. Again, like their cume figure totally inaccurate. If you don’t PROMPT you don’t know! Really simple.

    3. I asked Triton in the same conversation how they verify the data input on the registration page. They told me that Pandora deals with that through Ernst and Young. I asked if they comb through all 75m registered user pages to verify that the data is accurate and received no answer outside of Ernst and Young audits them. Maybe since Triton can’t answer that important question Pandora can? Love to know the answer to this question as it has been avoided since i started asking 3 years ago.

    So, again you have no way to know how many listeners are engaged in total and no way to know whether those are lisare listening you have nothing.Important questions that continue to go unanswered. P is a digital playlist not a radio station. Agencies should see it that way. That’s all. Rob can tell me nothing that i already don’t know but please thank him for weighing in.

  • Peter, I have to admit I’m surprised you’re pushing so hard in this venue on this issue – it was a tangent in the original post.
    I will only address your opening point – Pandora and radio are part of a larger audio experience and the two definitely overlay some of the same problems consumers are looking to solve. Thus they ARE part of the same category. That’s true from a consumer standpoint and is increasingly true from the buyer standpoint.
    And at hivio a couple weeks ago I had an agency CEO say exactly that. An AGENCY CEO.
    So I think you’re wrong to view Pandora as “not” radio – and it doesn’t help radio to be viewed as “not” Pandora.
    A former Clear Channel CEO once told me one of his biggest regrets was not buying Pandora when he could.
    That’s how much “radio” Pandora is.

  • We can agree to disagree. My points have been made and I’m satisfied with my stance and attention to detail. I’m speaking from the sales side and you from the consultancy side. 2 different viewpoints. We can leave it at that.

  • No, I’m speaking from the consumer’s side. That’s the side I have always been on. But yes, we disagree. Thanks for taking the time to make your opinions clear.

  • Thank you for presenting the forum and opportunity.

  • Pingback: Can Los Angeles PPM ratings scandal happen in Canada?()

  • Rick Starr

    Over the top reaction. One single household might skew the ratings in a particular demographic cell (which obviously would have a smaller base than the total market) but it is unlikely that it could “skew the whole in a significant way.”

  • Not so, Rick.

    One household – especially in a large MSA Hispanic market – can contain several meters. And that one household with those several meters in that under-sampled and heavily weighted cell can absolutely skew the market. And what happened in LA is proof.

  • Gordon Atkins

    In the St. Louis market you can hold the number one position with as few as 15 meters reporting on you station…that’s 15 out of 1200 in a market of 2,000,000 people. Fuzzy math….

  • Gordon Atkins

    Not over the top….see above

  • Rick Starr

    I think you agreed with me. They might skew “a cell” but they certainly couldn’t skew “a market.” Everything in the article is supposition except the fact that they have found “one family”. If it *your* assertion that it somehow influences the entire market, and you do so without statistical evidence. The fact that they’ve recalled LA only indicates they are scrupulous if they find potential fraud, which I find admirable. (I am not downplaying the important of “a cell”, especially in certain demos, but still…) I dealt with them closely in 30 years in the industry and saw a couple of similar recalls and the variance was tiny, at best.

  • Rick, if they significantly affect the market ranking of any station or stations, then they – by definition – have skewed the market.
    This is the tip of the iceberg. The horror stories in radio ratings, particularly in PPM markets, are legion.

Dive Into The Blog