As searchers become more sophisticated, the search engines are forced to evolve to keep up with the changing times.
During a session at WebmasterWorld’s Publishers Conference VI, Tim Mayer from Yahoo! Search explained how Yahoo! has improved its abstracts, the presentation of its results pages, and how it has developed a new algorithm designed specifically to focus on longer query searches using more terms.
Yahoo! now offers a larger index and additional changes have been made to its page cache. Yahoo’s daily crawler discovers and crawls millions of new documents each day. The company is also experimenting with submission quality and the new Yahoo! Slurp bot will crawl links from the URLs that are submitted.
Additionally, Yahoo! Search RSS will be arriving on April 15th.
Yahoo! Slurp. Mayer says Yahoo! Slurp adheres more strictly to robot exclusion standards than any other search bot. The best way to block the bot is through .txt files, as other methods won’t necessarily work. The Yahoo! Slurp team currently prefers sites that are navigable by HREF’s, not by forms, JavaScript or Flash. The site should not require cookies for entry. Avoid session IDs in the URL. Sitemap links should be included on the home page. Use REAL 404s for error pages.
New slurp commands: Slurp now recognizes the NOARCHIVE meta tag command and crawl-delay, although Yahoo! admits it is still struggling with the way Slurp handles redirects.
Yahoo’s Crusade Against Spam. Yahoo! takes the fight against spam very seriously, using both automated and human filters for fighting spam. Yahoo! Search will be combining intelligences with Yahoo! Mail to offer better anti-spam technology.
Have you been banned? If your site has been banned from Yahoo! for spamming, Slurp will visit your robot.txt but will not go deeper into your site. Paid inclusion members might notice their URLs appearing lower on keyword query pages.
What can you do? If your site has been banned and you’ve cleaned up your act you can submit your page to the paid inclusion program for review in April or send an email to feedback@yahoo.com and include what method got you in trouble and how you’ve corrected yourself.