Which Search Engine is Really #1? Metrics Agencies Close in on Reality

by

So how are the rating agencies doing when it comes to tracking down which search engine is really most popular with users? A fresh article by Danny Sullivan notes that Jupiter Media Metrix is finally releasing search-specific traffic figures. This is a real step forward. JMM always released for public consumption a list of the top 50 web properties, but this never told us much about search specific traffic. Yahoo, AOL, MSN, Lycos, and Excite would always show up at the top of the leader board because they were portals with a variety of features and content. Search usage was not separated out. JMM did *also* regularly release themed reports measuring the top “search engine properties,” but here they vacuumed out any portal sites and any site not PRIMARILY identifying itself as a “search engine,” thus leaving Ask Jeeves to duke it out with AltaVista, but not AOL. What we always wanted to know, and still do, is how does Yahoo Search, and only search, stack up against Google? How does AOL Search, and only search, stack up against MSN Search (and only search)?

We’re getting closer to the answer, but there is still a fair bit of room for interpretation (See http://searchenginewatch.com/reports/mediametrix.html for a summary of the numbers). Measured by “audience reach,” we can see that MSN claims top spot with 36.3%, with Yahoo close behind at 33.3%, Google at 24.5% and growing, and AOL at 22.2%. Netscape is also owned by AOL, keep in mind, so it’s fair to add some of Netscape’s 7.9% reach to AOL’s total, although it’s also worth noting that standard Netscape.com searches are powered by Google. (Yes, my head is starting to hurt, too.)

Perhaps most shockingly, AltaVista usage has fallen off a cliff, coming in at 5.7%. Any webmaster who has looked at their server stats will have already noticed this. Good night, nurse. :- (

For now, 4th-biggest portal Terra Lycos is not asking JMM to separate out its search-specific numbers; it is likely a few points behind AOL.

Sullivan notes a methodological problem – MSN’s numbers are inflated by things like accidental searches in the Internet Explorer address bar, a claim disputed by MSN’s Bill Bliss. If you ask around, you’ll hear few site owners who can corroborate this apparent MSN lead. Most will mention the ascendancy of Google and the continued strength of Yahoo in terms of search referrals, though it obviously varies from site to site.

The MSN measurement problem was first brought to my attention, and alternative measurements put forward, by a fledgling data mining outfit called Plurimus. Unfortunately, it appears that Plurimus has gone bankrupt in spite of raising $38 million in venture capital since its founding in 1999. That’s too bad; the current state of Internet research could have been improved by their probing, questioning approach to Internet metrics and study design.

Another problem with JMM’s top ten search engine standings is that measuring “audience reach” is a poor substitute for measuring the number of search queries processed. It’s one thing to have used MSN Search (accidentally in some cases) once in a month, and quite another to perform a couple dozen Google searches every day. Search volume is obviously a better determinant of what search engines are most used as research tools, as opposed to those which happen to snare a lot of casual Internet users occasionally. The search volume numbers as reported by JMM are currently hazy because most major portals don’t report them, and self-reporting is less than objective.

If a trend can be projected from sheer depth of feeling, it won’t be long before Google is the undisputed global search leader. Never have so many been such devotees of a particular search engine, though we’ve certainly seen passion for search destinations like Yahoo, AltaVista, HotBot, Infoseek, Metacrawler, etc. wax and wane in an earlier phase of the game when fewer people were online.

Google’s star is still rising, and now that the cat is out of the bag that they’re ahead of AOL as a search destination, influential observers will sit up and take more notice of what the early adopters have been saying all along. Whether they’ll see much additional growth with the broader consumer market is another question, but I tend to think that the secular trend in the use of the Internet as a research tool, and the likelihood that younger people will use Google (rather than, say, MSN Search) at school, will work in Google’s favor. And maybe someday more of those AOL users will taste Google for the first time, and get hooked.

We wish that the leading stats measurement firms – and with the impending merger of JMM and Nielsen/Netratings, the number of such firms is getting perilously close to one – weren’t quite so emphatic about their rankings of “leading web properties,” and that they did a better job of putting more granular data to use in a way that makes sense. For example: at one point, data were interpreted in such a way as to claim that more “total time spent” on a search engine was proof that it was “less satisfying” (yes, less satisfying) to users because they obviously weren’t finding what they were looking for. Or of course, maybe just the opposite is the case – you use what works, every day of the week!

We’ve been through some paper-thin fads in the measurement of Internet user behavior. Is “stickiness” good? Bad? Indifferent? Is revenue or profitability the only web site stat worth paying attention to? Aren’t most user metrics a distracting anachronism that ought to have died the day the banner died? Could it be that such determinations depend on what kind of results might show the highest-paying clients of the consulting firms in a positive light?

Most polling companies and stat firms need to maintain an air of certainty about their findings. They need for their chosen metrics to be perceived as vitally important, since this is what they have to sell. With the number of credible online metrics agencies shrinking, those left standing will have fewer pesky critics questioning their methodologies. Maybe the key lesson is, if you want the really good data, you have to pay to access it. The public only gets a snapshot of a complex reality.

You may also like