Monday, October 22, 2007

New York Time Misses Real News: Measurement Discrepancies Will Be Resolved

As a former newspaperman, I’m sensitive to complaints that “you missed the story.” But today’s New York Times article on audience measurement discrepancies in interactive advertising did skirt by some of the more important developments in our industry during the past year.

The most significant development: The major audience measurement companies are now working with the IAB to get to the bottom of discrepancies, and we are committed to working together to identify, implement, and educate the marketing and media industries on best and emerging practices in audience measurement.

I’m not a Polyanna – these matters can be very complex, and (let’s face it) disputes between media companies and audience measurement firms are as old as audience measurement itself. But I am confident that, as an industry, we are on our way to reducing measurement discrepancies as an issue.

Discrepancies Matter

I don’t want to dump on The Times story – if you think it’s so easy to summarize a history of the world in 750 words, try it someday. (Then try it five days a week, as some of us had to do in our youth.) But today’s piece did read like a rehash of the complaints and counter-complaints we’ve been hearing and voicing for years: The counts offered by comScore and Nielsen//Netratings differ from publishers’ counts by 30-40-50%... The comScore and Nielsen counts differ from each other… Cogent explanations for the differences can’t be easily had… People at work are undersampled… High-income Web users are undersampled

For my taste, there was too much emphasis in the piece on at-work populations, and not enough notice of other populations that are chronically undersampled in audience research, notably ethnic and racial minorities and young men. And there was no notice of one of the ways current audience measurement techniques subvert a primary promise of the Web: the ability to match niche buyers and niche sellers through the perfect media vehicle. Hundreds of thousands, perhaps millions, of small media sites – the Mom & Pop grocery stores of the Web – likely don’t show up in samples because of their audience size or traffic patterns, thereby robbing advertisers and agencies of potentially valuable matchmaking knowledge (but building a business case for the ad networks and search engines which do deliver ads to and eyeballs from such sites).

IAB’s Historic Summit

These complaints had been festering in the interactive industry for years, but they had resisted resolution. Why? My guess is that publishers, measurement firms, and interactive agencies were too busy inventing a new civilization to take the time to perfect that civilization’s culture. When you’re spending all your time building mud huts and foraging for food, it’s a bit hard to worry about table manners.

But as the funds flow into interactive media began to increase strikingly, the impact of these discrepancies on publishers’ sales, pricing, and competitiveness was taking an ever-larger toll. The IAB’s management and board determined it was time to act, and to act aggressively. Last April, we issued an open letter to comScore and Nielsen//Netratings. We sought two things: a “summit meeting” at which we could agree together on a process by which we could resolve discrepancies; and agreement by these vitally important companies to undergo third-party audits and accreditation of their processes.

That story has been well-covered elsewhere (although alas, not by The Times) so I won’t repeat it. (For a full history and continued updates, go here on the IAB Web site.) What never got covered were the opening remarks I made in kicking of that historic summit in May. To the assembled comScore and Nielsen executives, I said, “Your companies are among the heroes of the interactive marketing and media ecosystem. Starting years ago, and continuing through some very dark days, you took risks based on strong beliefs about where our industries are heading. Because you took those risks, you have built vital, important companies. You are important because you are excellent. You are excellent because you practice superior science, lead superior operations, and deliver superior service. And your strength is our strength. We thank you for the work you’ve done to build interactive media.”

But, noting the persistence of the measurement discrepancies amid the transformation of interactive media into major components in marketing plans, I added, “This transformation requires us to find new ways to collaborate. Yes, there will always be tensions among buyers, seller, and intermediaries. And yes, let us be fierce competitors within our individual industries for revenues and share. But let us also do what our various industries have long done, to greater or lesser degrees: work together to serve our customers and delight our consumers at the same time.”

Audits Are Happening

The important points are these: comScore and Nielsen DID join the summit meeting, along with representatives of the AAAA, ANA, MPA, OPA, ARF, and executives from numerous agencies, publishers, and marketers. And the two companies DID agree to undergo audits and accreditation supervised by the Media Rating Council. And the two companies ARE now undergoing these audits – indeed, they will increasingly compete against each other on the basis of the quality of their numbers, and of the insights that can be derived from these numbers.

The audience measurement companies, in turn, had their requests of the IAB. They need agreed-upon standards against which they can measure, particularly definitions of unique visitors, page views and time spent. We agreed to intensify our Audience Measurement Committees’ efforts to derive those standards. We also agreed that we would work to understand more concretely the impact of cookie deletion, international traffic, spiders and bots and other potential factors in audience-size discrepancies.

(By the way, one of our summit meeting conclusions appears to have gotten lost in the shuffle. While IAB and the measurement firms disagree on the significance of cookie deletion as a systematic factor in measurement discrepancies, we all agreed that more rapid and frequent reporting on site traffic would certainly minimize whatever significance it has. Thus we concluded that a system of “overnight” ratings, similar to television’s overnights, would buttress the current system that emphasizes average monthly visitors – a somewhat jejune construct, anyway, in the interactive space. I still would like to see movement in this direction.)

We also agreed that IAB would lead a market-education campaign on the frontiers of audience research. I’m happy to report that our launch is only a month away. On November 29, IAB inaugurates its Audience Measurement Leadership Forum. comScore and Nielsen will be presenting – as will Quantcast, M:Metrics, Visible Measures, Millward Brown, CNET, Conde Nast, CNN, and many other thought leaders and practice leaders from the research, media, and marketing industries. (Might I suggest booking soon? There are fewer than 300 spaces, and as you might expect, it’s been filling up fast.)

Quest for Insights

Without minimizing the painful financial impact measurement discrepancies have today on many interactive publishers, in a very real sense audience measurement is a side show. The main tent is called Insights, and in it are three rings: Transparency, Accountability, and Predictability -- what’s happening, what did happen, and what will happen. Our goal must be – as we said in our May summit meeting – to achieve “convergent validity.”

“Convergent validity” is a phrase coined by Peter Daboll, Yahoo!’s chief of insights and head of global market research, and a former president of comScore’s Media Metrix division. It emphasizes finding the best data sources for different ends, and bringing them together to create a mosaic of the audience, its needs and desires, and its consumption patterns that is richer and more useful than the thousand snapshots we currently assay separately.

Put another way, server logs and panel-based measurement are far more valuable together than they are apart. Server logs – a census counting technique – may be exact, and they should help agency media planners assemble a crank, but they cannot provide us and our advertisers crucial information we need about audience composition. Panels may never, by definition, be exact, but they can provide textural detail of immense value. Bringing these and other techniques together will get us closer to the real insights our customers want.

This is the path we’re now on. I wish The Times had that story. But who needs The Times when you’ve got the Clog?


  1. This is a great article Randy.

    I've been watching all this for some time, and I think the real answer to this problem needs to be simple and elegant. I have two points to make here...

    1. Publishers who are MRC Audited should have a simple way to share their numbers with the audience companies. Perhaps the audience measurement folks should come up with definitive methodologies for counting unique visitors when publishers are processing their log files (this may be impossible technically), and even find a way to provide unique ids to the publishers so that their panels can be called out at log processing time (this should not be). The numbers provided by MRC audited publishers should be used as the baseline by the panel companies - and the panel companies should go off those baselines with their further calculations. Some of the visitors will be 'disqualified' by the measurement companies - but the overall numbers should be much higher. And that doesn't solve the problem of grossly misattributing counts to the wrong demographic because of lack of coverage by panels.

    2. I'm really tired of the intermixing of concepts in this debate.
    a. we have measurement data that is used for planning purposes (Comscore and NNR)
    b. we have measurement data that is used for billing (Publisher server data)
    c. we have measurement data used by agencies to measure across publishers - which they frequently try to force as billing data, but was never the intent (Atlas, DFA)

    This is of course a complex topic to cover - but the industry makes the topic more complex every day be interweaving the stories. Of course, that may be the answer - to use one set of numbers - period. But in the meantime, we should not be promoting the use of billing discrepancies with discussions of why the measurement companies are innacurate.

    Eric Picard, Director
    Advertising Technology Strategy

  2. On behalf of one of the little guys ( to be a serious traffic contender someday, having a uniform set of measurement standards is essential.
    The "overnights" idea is a strong step in that direction.