Media: The First Crack in the PPM

TORONTO, ONTARIO - In an earlier post, a promise was made to monitor the performance of Arbitron's new Portable People Meter (PPM) in rating radio stations across the continent. The PPM, a device to be worn by the people in the listener sample that "hears" what radio station is playing in their environment, is replacing the old practice of having people fill out diaries of what they have listened to over the course of a day.

The first really interesting quirk appears to have been identified by attentive broadcast professionals. As reported on a variety of USENET newsgroups (yes, those pre-World Wide Web things still exist, and not just on Google groups!), an interesting set of data has been published with overnight results from the San Francisco market, which has been converted from diaries to the PPM.

The data shows that, in January 2009, KUIC 95.3 FM has the largest average number of listeners in the Bay Area from midnight to 5 AM at 4,000 people, 11.4% of the total audience (which says something about the estimated number of people listening to the radio overnight around San Francisco). According to Arbitron, KUIC attracts a higher average audience with its adult contemporary format (which we used to call "light rock") than the traditional market leader, KGO Newstalk 810 AM, which provides a local talk show throughout this time period, and KCBS All-News 740 AM and 106.9 FM, which provides live, local news throughout the night.

What's wrong with this picture? I lived within 35 miles of San Francisco for four years and I never listened to 95.3 FM. It wasn't because I disliked the station--it was because I couldn't receive KUIC in Palo Alto. KUIC is licensed to Vacaville, California, has a transmitter located to the northwest of that city, and puts a better signal into Sacramento than San Francisco, where it has only fringe coverage. The station web site makes it clear that it is focused on serving Solano County, something it likely does very well. However, Solano County has just under 400,000 residents--about half that of the city of San Francisco alone.

In the overall ratings for the San Francisco market in February 2009, KUIC placed 26th, with about one-quarter the audience of market-leading KCBS (which, by the way, is pretty impressive for a station aimed at Solano County). While ratings can vary by time slot, there is no way that more people are actually listening to KUIC overnight than KOIT-FM, a San Francisco station with a similar format that puts a signal over many more people, which shows as having about one-third of KUIC's audience. This is clearly a sampling error.

Furthermore, the cumulative statistics in the very same survey seem to show more normal results. KCBS leads the way with nearly 83,000 different people tuning in overnight at some point, followed by Spanish-language KRZZ at about 55,000, KGO at about 43,000, public radio KQED at about 41,000, and KOIT at over 37,000. KUIC is well down the list at 11,700. How can that be? It implies that a good portion of people are spending a much longer time listening to KUIC than any other radio station. Statistically, there's no other way to explain this than the sample size in the overnight period is way too small, and one or two security guards that happen to be tuned in to KUIC all night (or have their dogs wear the PPM and leave their radio on KUIC while they're asleep, or some such anomaly) are skewing the results.

Some would find this quirk to be absolutely unremarkable. The overnight figures are not terribly important to any radio station, and they would say that critics of the PPM must be digging pretty deeply to come up with this example. It may indeed be a very unimportant quirk. However, it may demonstrate that the small sample sizes that come with the PPM will prove a problem in developing meaningful results in other time slots, even when a few more PPM's are hearing radio stations--especially in markets much smaller than San Francisco.

I would say the jury is still out on the PPM, but there is reason to be a bit cautious in evaluating its data.