Yahoo Slurp Spider

[Phil633]

Showing my ignorance here, But what is a Yahoo Slurp Spider:eek: :confused:

[rangieman]

well we have them on nights like tonight when bugger all members are available to search the forums:p

[HangOver]

Showing my ignorance here, But what is a Yahoo Slurp Spider:eek: :confused:

something to do with yahoo collecting info for it's search engines I think

[Pedro_The_Swift]

its one reason NOT to run the yahoo toolbar---:mad:

[p38arover]

its one reason NOT to run the yahoo toolbar---:mad:

If one isn't watchful, a lot of software will install the toolbar - then you have the hassle of removing it. I've removed it several times.

Ron

[Hunchy]

OK, this I know about. Slurp is the name of Yahoo's search engine spider software. Slurp goes out to all the websites it can find (millions), reads every page (up to around 200 off each web site, and only to 3 "jumps" from the home page), and sends them to Yahoo's search engine for indexing.

So Slurp is like an automated browser. Instead of you or me sitting on front of a computer, firing up IE and clicking on every link we can find, it does this automatically.

Google's equivalent to Slurp is called "Googlebot".

If you're running a website and check the logs, you should see Slurp and Googlebot hitting your website hundreds of times each month. This is a good thing, because it means some or all of your pages will end up in Yahoo or Google.

Slurp started off as part of Inktomi's search engine (Inktomi was the search behind AOL, Yahoo, MSN, Anzwers, and about 20 other search engines). After the dotcom crash in 2002, Inktomi got bought by Yahoo and they still use the Slurp spider. I remember it well, 'cuz I was Inktomi's ANZ MD when it all came tumbling down :(

[Phil633]

Thanks, Hunchy:D Hangover you were right too:D

Now I know:) :) :)

[Hunchy]

One other point... Slurp has nothing to do with the Yahoo toolbar. It's just a piece of software that crawls websites. looking for content to index and stick into the Yahoo search engine. No more.

You should only ever see Slurp mentioned in your website log files, when it comes visiting. If you see Slurp in ANY other context, then it ain't the Slurp I know about.

If you don't want Slurp to index your site you can tell it not to in your robots.txt file (put this in your website's root directory). For all the gory details on how to do this check out http://www.robotstxt.org.

Sorry for the boring techo rant, I do search engines for a living.

[Hunchy]

No probs Phil :-)

[incisor]

One other point... Slurp has nothing to do with the Yahoo toolbar. It's just a piece of software that crawls websites. looking for content to index and stick into the Yahoo search engine. No more.

You should only ever see Slurp mentioned in your website log files, when it comes visiting. If you see Slurp in ANY other context, then it ain't the Slurp I know about.

If you don't want Slurp to index your site you can tell it not to in your robots.txt file (put this in your website's root directory). For all the gory details on how to do this check out http://www.robotstxt.org.

Sorry for the boring techo rant, I do search engines for a living.

lets elaborate a little shall we...

yahoo openly admits the toolbar triggers visits by slurp.

yahoo and google both use toolbar info to trigger visits to websites as well as for other things, have a look at the yahoo web rank feature as just 1 case in point.

slurp itself may just be a crawler but it certainly gets fed info returned from toolbar data... i have seen it demonstrated on new site addresses that couldnt have been targeted any other way.

you can turn the function off in yahoo's toolbar and other products if you use the link in their terms and conditions on the web site to turn the function off, or used to be able to...

or

you can block access to the feature by putting a deny on http://cpn.yahoo-webrank.net/ in your firewall or proxy.

or

if a webmaster you can use robots.txt if you wish to stop or throttle slurp connections.

i dont mind the visits, but there are negative issues as well.

[Hunchy]

slurp itself may just be a crawler but it certainly gets fed info returned from toolbar data... i have seen it demonstrated on new site addresses that couldnt have been targeted any other way.

Yeah fair comment incisor. The toolbar is one of a few ways that slurp gets its spider food, and it's also used to figure out whether a page is regarded as "authoritative" by the number of hits it gets. On-site links are another, and third party links are another again.

I should've remembered that :)

[Vandermorph]

Showing my ignorance here, But what is a Yahoo Slurp Spider:eek: :confused:

If you are ignorant so am i i asked that ages ago