Archive for February, 2007|Monthly archive page

Measuring Internet traffic: where are the biases?

There have been quite a few discussions on traffic measurement lately. The general consensus is that all of them have some sort of problems. It would be an interesting exercise to see where are the biases and how we may be able to compensate for them.

ComScore and Hitwise are two leading paid services. They use two different approaches: ComScore is “Panel based” and Hitwise is “ISP based”.

1. ComScore

ComScore has over 2 million users who have installed ComSore’s data collection software on their computers (although their US panel sample is 120K in the US and global panel is 500K outside the US). Their users are randomly selected. ComScore recruits them over the web by offering virus protection scanning, web acceleration or sweepstakes prizes under a number of channels (e.g., PermissionResearch, OpinionSquare and Marketscore).

ComScore’s demographic tends to be skewed toward naive Internet users as more sophisticated users are less likely to install ComScore’s toolbar. Serious security issues have been raised with their software. If you are interested in the details of how ComScore collects user data and the security implications, I would recommend you to read the articles by Stanford, Cornell and Forbes.

2. Hitwise

Hitwise gets its user data from ISPs that it has partnered with. According to Hitwise it have over 10 million US and 25 million worldwide users.

While Hitwise has a much larger and diverse pool of sample users, its data partners are mostly small ISPs and has much more dial-up users in the data set.

In general, Hitiwse’s data tends to be more skewed towards home use and underestimates broadband or work use.

3. Alexa

Alexa offers a free traffic data service and is a subsidiary of Amazon.com. Alexa collects information from over 20 million users who have installed the “Alexa Toolbar”. The Alexa toolbar is available on Internet Explorer and an extension (Status Bar) can be used for Firefox.

Alexa toolbar is offered as a webmaster tool and its user panel is biased towards techies/geeks and webmasters in particular. Alexa’s number can not be used to compare two sites with very different demographic.

Another problem with Alexa is that its numbers are relative shares (percentage of the total population). Because Alexa’s international base is growing much more rapidly, US websites’ Alexa numbers tend to increase slower than their internal stats or even show some decline.

4. Compete and Quantcast

Compete and Quantcast are two smaller free services. Compete.com tries to combine toolbar panel and ISP data whereas Quantcast requires websites to install a tracking pixel. For some websites, they offer good numbers whiles for others, their numbers can be way off. It is still unproven that their approaches offer better results. You can read detailed discussions from Venturebeat, Matt Cutts and Traffick.