Let me tell an Old Timey Story. When I joined in 2000, Google was a scrappy underdog search engine. Back then, Altavista was vastly more popular and reported 50 million searches a day. Google was popular among savvy webmasters and at many universities, and usage was growing quickly by word-of-mouth, but the smart folks at Google were eager for the company to be more well-known. At the time, the metrics services of the day vastly underrated the number of searches done on Google every day. Month after month, every report seems to show that Google had a tiny share of the market.
At some point, one of the metrics services (which shall remain nameless) came to Google so that we could try to reconcile our data with their claims. I wasn’t in the meeting, so afterwards I caught an engineer and asked what happened; why did our numbers differ by so much? “They solicit people to install an application for them” was the answer. “But that’s a horrible methodology!” I said. “That would get you a ton more novice users; expert users wouldn’t see the value and probably wouldn’t install the application as much.” The other engineer agreed.
That was an eyeopener for me. At the time, Google was much more popular with highly-technical users, who were less likely to show up in that metric. So while Google gained market share, that particular methodology always lagged in showing Google’s growth. In a way, it was a blessing in disguise: if competitors took the metrics at face value, they would underestimate Google and how fast it was growing. Ever since, I’ve taken every metric with a grain of salt–you have to think about underlying assumptions and limitations in the data.
Let’s do a simple exercise to see if you’ve been paying attention. Suppose someone calls you up on the phone to ask you to record what you’ve been watching on TV. “How did you choose me?” you ask. The caller says, “Oh, we go by the last four digits of your phone number.” Now, what limitations will there be in the data? People without phones will be left out in the cold. People who have two phone lines are more likely to get a call. And someone who ditched their landline for a cell phone might not get a call. That will absolutely skew the selection of people unless the group doing the survey makes special efforts.
Ah, writing down what TV you watch isn’t accurate anyway, you say. Let’s buy metrics data from TiVo–they can pinpoint exactly what their users watch! Well, where’s the flaw in that? Does everyone have a TiVo? No way! TiVo viewers skew toward the hip and smart (and moneyed). Plus some providers (Cox? Comcast? DirecTV? Dish?) may not use TiVo as much because they offer their own DVR. So TiVo’s data is biased too.
Now that you’re appropriately jaded and cynical, let’s look at something out there right now. Here’s a recent post that appeared on Podcasting News:
Nielsen: Podcasts More Popular than Blogging
July 12, 2006
Nielsen//NetRatings announced today that 6.6 percent of the U.S. adult online population, or 9.2 million Web users, have recently downloaded an audio podcast. 4.0 percent, or 5.6 million Web users, have recently downloaded a video podcast.
These figures put the podcasting population on a par with those who publish blogs, 4.8 percent …
Okay, if you think more people podcast than blog, raise your hand. Anyone? No? The thing to notice is that Podcasting News contrasted downloaders of podcasts with producers of blogs. The headline might have been technically correct; it would probably not be correct if the headline were “Podcasting More Popular than Blogging” (notice how I turned “podcasts” into a verb?). Yet that article was at the top of Techmeme, and your average reader could easily miss the distinction.
The story has a happy ending. I went back this morning to check if it was still on Techmeme, and Scoble and another podcasting site are calling people on it. In the instance above, the Nielsen numbers may have been completely accurate, but you still have to analyze how someone takes those numbers and think critically about what claims they make (or imply).
This is long enough and I haven’t even *begun* to talk about Alexa or Hitwise, so let’s split it up. Today is Meeting Galore Day, but there will be at least a part II.