Thread: Yahoo Slurp Spider

well we have them on nights like tonight when bugger all members are available to search the forums

Originally Posted by Phil633

Showing my ignorance here, But what is a Yahoo Slurp Spider

something to do with yahoo collecting info for it's search engines I think

its one reason NOT to run the yahoo toolbar---

Originally Posted by Pedro_The_Swift

its one reason NOT to run the yahoo toolbar---

If one isn't watchful, a lot of software will install the toolbar - then you have the hassle of removing it. I've removed it several times.

Ron

OK, this I know about. Slurp is the name of Yahoo's search engine spider software. Slurp goes out to all the websites it can find (millions), reads every page (up to around 200 off each web site, and only to 3 "jumps" from the home page), and sends them to Yahoo's search engine for indexing.

So Slurp is like an automated browser. Instead of you or me sitting on front of a computer, firing up IE and clicking on every link we can find, it does this automatically.

Google's equivalent to Slurp is called "Googlebot".

If you're running a website and check the logs, you should see Slurp and Googlebot hitting your website hundreds of times each month. This is a good thing, because it means some or all of your pages will end up in Yahoo or Google.

Slurp started off as part of Inktomi's search engine (Inktomi was the search behind AOL, Yahoo, MSN, Anzwers, and about 20 other search engines). After the dotcom crash in 2002, Inktomi got bought by Yahoo and they still use the Slurp spider. I remember it well, 'cuz I was Inktomi's ANZ MD when it all came tumbling down

Thanks, Hunchy Hangover you were right too

Now I know

One other point... Slurp has nothing to do with the Yahoo toolbar. It's just a piece of software that crawls websites. looking for content to index and stick into the Yahoo search engine. No more.

You should only ever see Slurp mentioned in your website log files, when it comes visiting. If you see Slurp in ANY other context, then it ain't the Slurp I know about.

If you don't want Slurp to index your site you can tell it not to in your robots.txt file (put this in your website's root directory). For all the gory details on how to do this check out http://www.robotstxt.org.

Sorry for the boring techo rant, I do search engines for a living.

No probs Phil :-)