A study of search results displayed at the four major search engines shows that each engine is increasingly producing results that differ from the other engines. In other words, searchers can be reasonably confident that Google, Yahoo, MSN or Ask Jeeves will, more often than not, return results unique to the engine being used.

The study, “Different Engines, Different Results”, which was conducted by the meta-search engine Dogpile and researchers from Penn State and the University of Pittsburgh, examined search results from over 12,500 random, user-entered search queries on Google, Yahoo, MSN and Ask Jeeves. Of the 12,500 inquiries, which produced a total of 485,460 results, only 5,301 or 1.1% produced the same first page results across all engines. The significance of this change over the past year and a half magnified when compared with a similar study released by Dogpile in May, which showed a 3% overlap in total search results.

In previous years search engines tended to draw results from two common databases, Google and Inktomi, or were fed by established directories such as the ODP or LookSmart. Before Yahoo began producing its own results in February 2004, searchers routinely reported frustration with the repetitive nature of results everywhere they searched, leading many users to simply turn to the largest primary source, Google.

While it has only been a year and a half since Yahoo introduced its own proprietary search engine, 18-months can represent an entire generation in the IT world. Since then, MSN introduced its own algorithm search engine, Ask Jeeves maneuvered itself back into the big leagues and a host of newer search tools have emerged. The effect on search results has been enormous, leading proponents of meta-search tools such as Dogpile to suggest that brand-loyal search engine users are limiting their own options by using only one or two engines.

The study says that people are already meta-searching when using more than one search engine to find information. “While Web searchers who use Google, Yahoo!, MSN and Ask Jeeves may not consciously recognize a problem, the fact is that searchers use, on average 2.8 searches per month. Couple this with the fact that a significant percentage of searches fail to elicit a click on the first page of search results, and we can infer that people are not necessarily finding what they are looking for with one search engine.”

The research team set out to examine the degree to which results on the Big4 differed and/or overlapped, how placements differed across each engine, and how meta-search tools such as Dogpile could do a better job of presenting results amalgamated from all four engines. The study focused on only first page results noting that 89.8% of all search result click activity originates from first page results.

In its methodology the study shows the number of references, both organic and paid, returned by each search engine. By default, each of the Big4 display up to ten organic placements for any given search term, along with paid or sponsored listings. Interestingly, MSN and Google both tended to show slightly fewer results (paid or unpaid) than did Yahoo or Ask Jeeves. At the time of the study, Ask Jeeves was seen to be displaying more paid advertising than its competitors with an overall average of 3.3 ads per (first) page of results. Yahoo came in second with 2.9 ads per page of results, Google third with 2.4 and MSN a distant fourth with only 1.9.

Perhaps the most interesting part of the study shows that users of the major search engines might be missing the majority of the web. By taking the results of the 485,460 unique references generated using 12,500 queries and comparing them with each other, the researchers concluded that, on average, users of each of the Big4 search engines “miss” approximately 70% of materials referenced by rival search engines, as illustrated by this grid diagram from the research document:

Missed 1st Page Web Search Results

% of Web’s 1st Page Results Missed
Google
343,700
70.8%
Yahoo!
337,144
69.4%
MSN
349,561
72.0%
Ask Jeeves
329,761
67.9%

Clearly, search engines are starting to show their own personalities as their algorithms increasingly present different search results for the same keyword queries. How these findings impact search engine usage remains to be seen however it is a rather safe assumption that details from this and subsequent studies will be used to market alternatives to search engine users in the future. For search engine marketers, this study shows the importance of learning about each of the search engines. As Chris Sherman noted in his SEW coverage, each of the search firms are speaking with their own unique voices. It’s only a matter of time before the searchers hear them clearly.