TORONTO, ONTARIO - Long-time readers of this blog will be aware that it has tried to follow the implementation of the Arbitron Portable People Meter (PPM) as the new technology to measure radio listenership has been rolled out in the United States in Canada. In the last update, the concerns of an influential program director in San Francisco were explored.
Since then, the PPM has hardly seen clear sailing. Of the 33 markets in the United States where Arbitron has attempted to introduce the devices, only three have been certified by a Media Rating Council audit. It has been outright rejected in 18 markets, including New York, Los Angeles, and Chicago. Aribtron has successfully gone to court to force the Spanish Broadcasting System (SBS) to keep doing PPM encoding on its broadcasts, which the SBS had claimed it no longer needed to do because the PPM wasn't providing it with accurate data. In the middle of all this, Arbitron has changed Chief Executive Officers.
While the failed audits indicate a potential technical problem with the devices (or methodology), others have now opined that the PPM has simply highlighted deeper problems with the whole rating process. A well-known rating consultant, Mike Henry, has been particularly outspoken. As described in Michael Hood's BlatherWatch blog, Henry wants the US to go to the Canadian model, in which the Bureau of Broadcast Measurement is a non-profit organization controlled by the broadcasters. Rather than alienating its customers (both radio stations and advertisers), in Canada, they effectively control the process and thus have input at every stage of the ratings process.
For now, Henry's opinion seems to be in the minority. Unless Arbitron gets its act together, though, both technically and in terms of its relations with both radio stations and advertisers, his views may become mainstream, and there may be a lot of radio people visiting Ottawa to figure out how to structure a new system.
Showing posts with label Portable People Meter. Show all posts
Showing posts with label Portable People Meter. Show all posts
Friday, February 26, 2010
Tuesday, September 29, 2009
Media: The PPM Has "Absolutely No Value"?
TORONTO, ONTARIO - Long-time readers of this blog may recall some discussion of the Arbitron Portable People Meter (PPM), the new device that the radio ratings services company has been intending to replace diaries as a way of deriving ratings for radio stations. When I last explored this subject back in March, there were growing concerns that reduced sample size might be creating a problem for Arbitron.
I'm a bit late in reporting this, but it appears that sample size is indeed a big deal. In an open letter to Arbitron last month, KGO-KSFO General Manager Mickey Luckoff, arguably the most successful station boss of this era, specifically decried reduced sample size. In terms of the promised ability to better refine what caused a listener to change the station, Luckoff stated that the PPM-derived data provided "absolutely no value in making any such decision with any degree of accuracy or reliability." (I suspect any accurate mechanism would just tell them that the frequency was changed at a commercial break anyway.)
How bad is the problem? Harker Research's Radio Insights looked at what they expected to be a typical case. They found that during morning drive time, the key time for radio listenership, the median station in a major market had either one or zero PPM listeners for 70% of that time period. That's not a sample, that's a single listener. Luckoff clearly had a point in his open letter.
Morning drive in general has been an issue with the PPM--more people claimed to be listening to the radio in their diaries than the PPM is reporting. In another Radio Insights posting, they showed a 12% decline in listeners in the top 3 US markets during morning drive, according to Arbitron's publicly-released data. For an industry starving for advertising money, every listener counts, and a 12% drop is significant. It's so significant, in fact, that the ratings service in the United Kingdom dropped the PPM over the decline in morning ratings, which it claimed was not real.
As if the sample size and morning listener decline problems were not enough, there has long been a perception that minority listeners were not adequately represented. While I have yet to see any data on this issue, Reuters has reported that the Media Rating Council has found "persistent problems" with minority representation in the PPM data. Frankly, if sample sizes are as low as they seem to be, then it's not surprising that there are minority representation problems--there are majority representation problems as well.
The really interesting thing about the sample size problem is that it is not fundamental to the technology--it's only that Arbitron is trying to save money by having fewer PPM units in the field. The morning drive problem, on the other hand, may be a technological issue--people may not find the devices easy enough to integrate into their lives in the morning.
It has been noted that in the United Kingdom, the decision was made to abandon the PPM in favor of on-line diaries, which were deemed easier to use than the traditional written diaries. If I were an Arbitron competitor, I'd be rushing to present an on-line diary system to radio stations--the cost savings alone (Aribitron is charging about 50% more for PPM-derived data than diary data) would justify a switch, never mind the other issues.
I'm a bit late in reporting this, but it appears that sample size is indeed a big deal. In an open letter to Arbitron last month, KGO-KSFO General Manager Mickey Luckoff, arguably the most successful station boss of this era, specifically decried reduced sample size. In terms of the promised ability to better refine what caused a listener to change the station, Luckoff stated that the PPM-derived data provided "absolutely no value in making any such decision with any degree of accuracy or reliability." (I suspect any accurate mechanism would just tell them that the frequency was changed at a commercial break anyway.)
How bad is the problem? Harker Research's Radio Insights looked at what they expected to be a typical case. They found that during morning drive time, the key time for radio listenership, the median station in a major market had either one or zero PPM listeners for 70% of that time period. That's not a sample, that's a single listener. Luckoff clearly had a point in his open letter.
Morning drive in general has been an issue with the PPM--more people claimed to be listening to the radio in their diaries than the PPM is reporting. In another Radio Insights posting, they showed a 12% decline in listeners in the top 3 US markets during morning drive, according to Arbitron's publicly-released data. For an industry starving for advertising money, every listener counts, and a 12% drop is significant. It's so significant, in fact, that the ratings service in the United Kingdom dropped the PPM over the decline in morning ratings, which it claimed was not real.
As if the sample size and morning listener decline problems were not enough, there has long been a perception that minority listeners were not adequately represented. While I have yet to see any data on this issue, Reuters has reported that the Media Rating Council has found "persistent problems" with minority representation in the PPM data. Frankly, if sample sizes are as low as they seem to be, then it's not surprising that there are minority representation problems--there are majority representation problems as well.
The really interesting thing about the sample size problem is that it is not fundamental to the technology--it's only that Arbitron is trying to save money by having fewer PPM units in the field. The morning drive problem, on the other hand, may be a technological issue--people may not find the devices easy enough to integrate into their lives in the morning.
It has been noted that in the United Kingdom, the decision was made to abandon the PPM in favor of on-line diaries, which were deemed easier to use than the traditional written diaries. If I were an Arbitron competitor, I'd be rushing to present an on-line diary system to radio stations--the cost savings alone (Aribitron is charging about 50% more for PPM-derived data than diary data) would justify a switch, never mind the other issues.
Thursday, March 5, 2009
Media: The First Crack in the PPM
TORONTO, ONTARIO - In an earlier post, a promise was made to monitor the performance of Arbitron's new Portable People Meter (PPM) in rating radio stations across the continent. The PPM, a device to be worn by the people in the listener sample that "hears" what radio station is playing in their environment, is replacing the old practice of having people fill out diaries of what they have listened to over the course of a day.
The first really interesting quirk appears to have been identified by attentive broadcast professionals. As reported on a variety of USENET newsgroups (yes, those pre-World Wide Web things still exist, and not just on Google groups!), an interesting set of data has been published with overnight results from the San Francisco market, which has been converted from diaries to the PPM.
The data shows that, in January 2009, KUIC 95.3 FM has the largest average number of listeners in the Bay Area from midnight to 5 AM at 4,000 people, 11.4% of the total audience (which says something about the estimated number of people listening to the radio overnight around San Francisco). According to Arbitron, KUIC attracts a higher average audience with its adult contemporary format (which we used to call "light rock") than the traditional market leader, KGO Newstalk 810 AM, which provides a local talk show throughout this time period, and KCBS All-News 740 AM and 106.9 FM, which provides live, local news throughout the night.
What's wrong with this picture? I lived within 35 miles of San Francisco for four years and I never listened to 95.3 FM. It wasn't because I disliked the station--it was because I couldn't receive KUIC in Palo Alto. KUIC is licensed to Vacaville, California, has a transmitter located to the northwest of that city, and puts a better signal into Sacramento than San Francisco, where it has only fringe coverage. The station web site makes it clear that it is focused on serving Solano County, something it likely does very well. However, Solano County has just under 400,000 residents--about half that of the city of San Francisco alone.
In the overall ratings for the San Francisco market in February 2009, KUIC placed 26th, with about one-quarter the audience of market-leading KCBS (which, by the way, is pretty impressive for a station aimed at Solano County). While ratings can vary by time slot, there is no way that more people are actually listening to KUIC overnight than KOIT-FM, a San Francisco station with a similar format that puts a signal over many more people, which shows as having about one-third of KUIC's audience. This is clearly a sampling error.
Furthermore, the cumulative statistics in the very same survey seem to show more normal results. KCBS leads the way with nearly 83,000 different people tuning in overnight at some point, followed by Spanish-language KRZZ at about 55,000, KGO at about 43,000, public radio KQED at about 41,000, and KOIT at over 37,000. KUIC is well down the list at 11,700. How can that be? It implies that a good portion of people are spending a much longer time listening to KUIC than any other radio station. Statistically, there's no other way to explain this than the sample size in the overnight period is way too small, and one or two security guards that happen to be tuned in to KUIC all night (or have their dogs wear the PPM and leave their radio on KUIC while they're asleep, or some such anomaly) are skewing the results.
Some would find this quirk to be absolutely unremarkable. The overnight figures are not terribly important to any radio station, and they would say that critics of the PPM must be digging pretty deeply to come up with this example. It may indeed be a very unimportant quirk. However, it may demonstrate that the small sample sizes that come with the PPM will prove a problem in developing meaningful results in other time slots, even when a few more PPM's are hearing radio stations--especially in markets much smaller than San Francisco.
I would say the jury is still out on the PPM, but there is reason to be a bit cautious in evaluating its data.
The first really interesting quirk appears to have been identified by attentive broadcast professionals. As reported on a variety of USENET newsgroups (yes, those pre-World Wide Web things still exist, and not just on Google groups!), an interesting set of data has been published with overnight results from the San Francisco market, which has been converted from diaries to the PPM.
The data shows that, in January 2009, KUIC 95.3 FM has the largest average number of listeners in the Bay Area from midnight to 5 AM at 4,000 people, 11.4% of the total audience (which says something about the estimated number of people listening to the radio overnight around San Francisco). According to Arbitron, KUIC attracts a higher average audience with its adult contemporary format (which we used to call "light rock") than the traditional market leader, KGO Newstalk 810 AM, which provides a local talk show throughout this time period, and KCBS All-News 740 AM and 106.9 FM, which provides live, local news throughout the night.
What's wrong with this picture? I lived within 35 miles of San Francisco for four years and I never listened to 95.3 FM. It wasn't because I disliked the station--it was because I couldn't receive KUIC in Palo Alto. KUIC is licensed to Vacaville, California, has a transmitter located to the northwest of that city, and puts a better signal into Sacramento than San Francisco, where it has only fringe coverage. The station web site makes it clear that it is focused on serving Solano County, something it likely does very well. However, Solano County has just under 400,000 residents--about half that of the city of San Francisco alone.
In the overall ratings for the San Francisco market in February 2009, KUIC placed 26th, with about one-quarter the audience of market-leading KCBS (which, by the way, is pretty impressive for a station aimed at Solano County). While ratings can vary by time slot, there is no way that more people are actually listening to KUIC overnight than KOIT-FM, a San Francisco station with a similar format that puts a signal over many more people, which shows as having about one-third of KUIC's audience. This is clearly a sampling error.
Furthermore, the cumulative statistics in the very same survey seem to show more normal results. KCBS leads the way with nearly 83,000 different people tuning in overnight at some point, followed by Spanish-language KRZZ at about 55,000, KGO at about 43,000, public radio KQED at about 41,000, and KOIT at over 37,000. KUIC is well down the list at 11,700. How can that be? It implies that a good portion of people are spending a much longer time listening to KUIC than any other radio station. Statistically, there's no other way to explain this than the sample size in the overnight period is way too small, and one or two security guards that happen to be tuned in to KUIC all night (or have their dogs wear the PPM and leave their radio on KUIC while they're asleep, or some such anomaly) are skewing the results.
Some would find this quirk to be absolutely unremarkable. The overnight figures are not terribly important to any radio station, and they would say that critics of the PPM must be digging pretty deeply to come up with this example. It may indeed be a very unimportant quirk. However, it may demonstrate that the small sample sizes that come with the PPM will prove a problem in developing meaningful results in other time slots, even when a few more PPM's are hearing radio stations--especially in markets much smaller than San Francisco.
I would say the jury is still out on the PPM, but there is reason to be a bit cautious in evaluating its data.
Thursday, November 20, 2008
Media: The PPM Comes to Canada
TORONTO, ONTARIO - Anyone that has looked seriously at radio station ratings in North America for the past quarter-century, as I have been known to do, knows not to take the exact numbers too seriously. The diary method used by Arbitron Incorporated to measure radio listenership has inherent flaws and has been prone to fluctuating results that did not reflect actual changes in the market. However, Arbitron has been moving to a new technology, the Portable People Meter or PPM, that has different characteristics and may change what we hear on the radio. Furthermore, the PPM will come soon to Canada, as well as the United States--the first Canadian PPM data went to some stations just today.
Commercial radio stations care about their ratings because they effectively determine the advertising rates that they can charge. Armed with the data they have purchased from Arbitron, they can tell advertisers exactly how many people in any given demographic listen to their broadcasts, and hence the advertisements they carry. While Arbitron makes public its "12+" numbers, or ratings for all listeners age 12 or greater, most advertisers care about the "25-54" ratings, or listeners between the ages of 25 and 54, and some care about gender-based ratings as well--and those ratings are often different than the more widely-reported "12+" numbers. While public radio stations do not sell advertising and hence usually do not purchase data from Arbitron (and thus do not appear in the ratings), many of the major stations do use ratings to gauge how well they are serving the public (in the case of CBC Radio in Canada, for example) or to use as fodder in attracting underwriting (especially stations that are highly-ranked in their local markets, like WBUR in Boston and KUOW in Seattle).
Traditionally, these ratings have been determined by Arbitron using diaries. After finding a representative sample of listeners, Arbitron gave them notebooks to record the programs they had heard. The good thing about that system is that the programming had to make a real impression on the listener such that they remembered how to record it in the notebook, which means any advertising in it would likely also have had a chance to be remembered. However, it was obviously prone to human error. When I was in the San Francisco Bay Area, a common tale was that Arbitron had many notebooks that claimed to have listened to the BBC news at the top of the hour during morning drive on KQED-FM. The only problem was that KQED-FM carried NPR news. Rival public station KALW actually ran the BBC--but their call letters were so un-memorable that listeners would think they were listening to the memorable call letters, KQED. The accuracy of the diaries was always in doubt.
To solve this problem, Arbitron has turned to the PPM. Instead of asking its participants to fill out a diary, it is asking them to wear a device that detects special, non-audible encoding in radio audio that identifies a station in the wearer's environment. In addition to recording what the wearer hears in their home and in their car, the PPM will pick up the background noise in a store or dentist's office, assuming it is of listenable volume. Right there, one would expect the ratings to change--stations that serve as background, especially "soft adult contemporary" stations like New York's WLTW "Lite-FM", Boston's WMJX "Magic 106.7" or Seattle's KRWM "Warm 106.9" would be expected to get a bounce.
All reports seem to be that the PPM does pick up the encoding signal quite well, impressing skeptical engineers--though there is some question about picking up distant signals (people that listen to, say, WXLO out of Worcester in Boston or KXXO Olympia in Seattle, both of which I was personally known to do on occasion and both stations currently show up in those market ratings). There seem to be deeper issues with the PPM, however. While an attachment is provided to allow the monitoring of radio listened to on headphones, it has to be inserted between the headphone jack and the headphones with each use of the headphone device, something that I personally can't imagine doing every time I turn on a Walkman-type device. The PPM must detect motion or it assumes it is not being worn and shuts off. Thus, it will miss radio listened to at bedside in the morning or before falling asleep, or during showering. These are key radio listening times for some people.
Blog reader Urban Stiess from Los Angeles wrote that he had seen the PPM device about a year ago at a Society of Broadcast Engineers meeting. Said Stiess, "I was probably the only one present who was NOT impressed. Where others saw less required human cooperation in the process - I saw nothing more than a quick, less labor intensive, method of generating numbers regardless of accuracy."
Of course, Arbitron hasn't really been known for accuracy under the diary system, either. I used to say that it was one of the best-named companies in the world, as its results were entirely arbitrary. (The company name actually came from a device by that name developed when the company was called the American Research Bureau.) While many a radio station made a decision based on a single quarterly ratings book, wiser heads looked for trends over multiple quarters as any given book could have strange results. The PPM was designed to end that--Stiess is not the only one who questions whether it actually will.
The fact of the matter is that we do not know what impact the PPM will have on radio ratings and hence on programming. There simply hasn't been enough published data from the PPM available to compare with older diary data. Early tests in New York seemed to imply that besides the pro-light rock effect described earlier that the new ratings tended to favor active rock stations and show lower listenership to ethnic radio stations.
That early result has resulted in political attention. Just yesterday, the Wall Street Journal reported that a Democrat on the US Federal Communications Commission (FCC), Jonathan Adelstein, was calling for FCC Chairman Kevin Martin to investigate the PPM, saying that the system "constitutes a clear and present danger to media diversity." Arbitron claims the FCC has no jurisdiction--and it's probably correct. New legislation on radio advertising would probably be required for the FCC to have any say in how radio stations use privately-commissioned ratings to sell advertising.
Lest this seem like a problem for the US, Arbitron has the contract for radio ratings in Canada, and it is rolling out the PPM here as well, starting in the Montreal market where the last diary-based ratings will come out on 27-November and the first PPM-based ratings will appear on 10-December and monthly thereafter. At 2:15 pm Eastern on Friday, Astral Media will have a live webcast about the PPM results in Montreal, which interested readers may want to watch. Most other major markets in Canada, including Toronto, will move to PPM measurement in the fall of 2009.
It's too early to know the impact of the PPM on radio in the United States and Canada. We may have to put up with more "light rock, less talk." We may lose ethnic programming. Until we have numbers, we won't have an intelligent way to evaluate the situation. I eagerly await some data.
Commercial radio stations care about their ratings because they effectively determine the advertising rates that they can charge. Armed with the data they have purchased from Arbitron, they can tell advertisers exactly how many people in any given demographic listen to their broadcasts, and hence the advertisements they carry. While Arbitron makes public its "12+" numbers, or ratings for all listeners age 12 or greater, most advertisers care about the "25-54" ratings, or listeners between the ages of 25 and 54, and some care about gender-based ratings as well--and those ratings are often different than the more widely-reported "12+" numbers. While public radio stations do not sell advertising and hence usually do not purchase data from Arbitron (and thus do not appear in the ratings), many of the major stations do use ratings to gauge how well they are serving the public (in the case of CBC Radio in Canada, for example) or to use as fodder in attracting underwriting (especially stations that are highly-ranked in their local markets, like WBUR in Boston and KUOW in Seattle).
Traditionally, these ratings have been determined by Arbitron using diaries. After finding a representative sample of listeners, Arbitron gave them notebooks to record the programs they had heard. The good thing about that system is that the programming had to make a real impression on the listener such that they remembered how to record it in the notebook, which means any advertising in it would likely also have had a chance to be remembered. However, it was obviously prone to human error. When I was in the San Francisco Bay Area, a common tale was that Arbitron had many notebooks that claimed to have listened to the BBC news at the top of the hour during morning drive on KQED-FM. The only problem was that KQED-FM carried NPR news. Rival public station KALW actually ran the BBC--but their call letters were so un-memorable that listeners would think they were listening to the memorable call letters, KQED. The accuracy of the diaries was always in doubt.
To solve this problem, Arbitron has turned to the PPM. Instead of asking its participants to fill out a diary, it is asking them to wear a device that detects special, non-audible encoding in radio audio that identifies a station in the wearer's environment. In addition to recording what the wearer hears in their home and in their car, the PPM will pick up the background noise in a store or dentist's office, assuming it is of listenable volume. Right there, one would expect the ratings to change--stations that serve as background, especially "soft adult contemporary" stations like New York's WLTW "Lite-FM", Boston's WMJX "Magic 106.7" or Seattle's KRWM "Warm 106.9" would be expected to get a bounce.
All reports seem to be that the PPM does pick up the encoding signal quite well, impressing skeptical engineers--though there is some question about picking up distant signals (people that listen to, say, WXLO out of Worcester in Boston or KXXO Olympia in Seattle, both of which I was personally known to do on occasion and both stations currently show up in those market ratings). There seem to be deeper issues with the PPM, however. While an attachment is provided to allow the monitoring of radio listened to on headphones, it has to be inserted between the headphone jack and the headphones with each use of the headphone device, something that I personally can't imagine doing every time I turn on a Walkman-type device. The PPM must detect motion or it assumes it is not being worn and shuts off. Thus, it will miss radio listened to at bedside in the morning or before falling asleep, or during showering. These are key radio listening times for some people.
Blog reader Urban Stiess from Los Angeles wrote that he had seen the PPM device about a year ago at a Society of Broadcast Engineers meeting. Said Stiess, "I was probably the only one present who was NOT impressed. Where others saw less required human cooperation in the process - I saw nothing more than a quick, less labor intensive, method of generating numbers regardless of accuracy."
Of course, Arbitron hasn't really been known for accuracy under the diary system, either. I used to say that it was one of the best-named companies in the world, as its results were entirely arbitrary. (The company name actually came from a device by that name developed when the company was called the American Research Bureau.) While many a radio station made a decision based on a single quarterly ratings book, wiser heads looked for trends over multiple quarters as any given book could have strange results. The PPM was designed to end that--Stiess is not the only one who questions whether it actually will.
The fact of the matter is that we do not know what impact the PPM will have on radio ratings and hence on programming. There simply hasn't been enough published data from the PPM available to compare with older diary data. Early tests in New York seemed to imply that besides the pro-light rock effect described earlier that the new ratings tended to favor active rock stations and show lower listenership to ethnic radio stations.
That early result has resulted in political attention. Just yesterday, the Wall Street Journal reported that a Democrat on the US Federal Communications Commission (FCC), Jonathan Adelstein, was calling for FCC Chairman Kevin Martin to investigate the PPM, saying that the system "constitutes a clear and present danger to media diversity." Arbitron claims the FCC has no jurisdiction--and it's probably correct. New legislation on radio advertising would probably be required for the FCC to have any say in how radio stations use privately-commissioned ratings to sell advertising.
Lest this seem like a problem for the US, Arbitron has the contract for radio ratings in Canada, and it is rolling out the PPM here as well, starting in the Montreal market where the last diary-based ratings will come out on 27-November and the first PPM-based ratings will appear on 10-December and monthly thereafter. At 2:15 pm Eastern on Friday, Astral Media will have a live webcast about the PPM results in Montreal, which interested readers may want to watch. Most other major markets in Canada, including Toronto, will move to PPM measurement in the fall of 2009.
It's too early to know the impact of the PPM on radio in the United States and Canada. We may have to put up with more "light rock, less talk." We may lose ethnic programming. Until we have numbers, we won't have an intelligent way to evaluate the situation. I eagerly await some data.
Subscribe to:
Posts (Atom)