Yahoo! Lies about Obeying Robots.txt Directives

15 responses to “Yahoo! Lies about Obeying Robots.txt Directives”

1. 

   Sue 2008/11/16 10:21 am

   Great. And Slurp is also the biggest draw on resources. I’ve added directives to my robots.txt with crawl delays to try and combat this, to stay out of certain parts of my site, and more. So it’s all been for naught if Slurp is so easily confused. *Sigh*

2. 

   Shadow Caster 2008/11/16 12:08 pm

   Hi. To give Yahoo! the benefit of the doubt you could say that the robots rule is to not have that directory put into the Yahoo! index. It might crawl it for statistical analysis purposes or spying (paranoid yet?) but it won’t add it to its index.

3. 

   Jeff Starr 2008/11/16 1:31 pm • Post Author

   @Shadow Caster: the Robots Exclusion Protocol is “is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable..” according to the relevant entry at Wikipedia. So, to “give Yahoo! the benefit of the doubt” you would have to change the meaning, purpose, and functionality of the entire Robots Exclusion Protocol. Perhaps you were thinking of the meta noindex attribute?

4. 

   Donace 2008/11/17 8:32 am

   well we always knew yahoo’s bot had issues :p…wouldnt adding a .htaccess in the folders blocking all access barring exceptions be a more ‘secure’ alternative?

   also I noticed feedburner throws an email after every 3 posts instead of one :p fix it! I want to read them asap!!!

   Sideline issue the trackback on this post have a look at the site… it seems to be genrated using ‘ping crawl’ from bluehatseo.com … just amke sure the site is ‘kosher’ if you want to allow it! :p

5. 

   Shadow Casters 2008/11/17 8:10 am

   OK Jeff, I agree with you now. By any chance do you have any files in that folder that Yahoo! accessed?

6. 

   Donace 2008/11/17 9:29 am

   @ Jeff no imean the subscribe function, normally it sends me an email every time you publish a post etc…ths time it sent one email with three posts i.e.:

   Fruit Loop: Separate any Number of Odd and Even Posts from any Category in WordPress

   Yahoo! Lies about Obeying Robots.txt Directives

   Three Years and Counting

   Maybe they were published at the same date…to lazy to check :p

7. 

   Donace 2008/11/17 9:46 am

   @Jeff Email….easier to read on the move….if they were all published yesterday that explains it then! I’m used you your one every few days posting schedule :p

8. 

   Jeff Starr 2008/11/17 9:22 am • Post Author

   @Shadow Casters: yes, but only an index.php file that tells the visitor that they have “fallen into a trap” and displays their IP and other server information. Yet this file is completely isolated and accessible from only the pages on this site..

   @Donace: indeed, htaccess could easily prevent Slurp from treading into the forbidden realms, but there is nothing there to protect, really. The whole setup is just a big “spider trap” to catch bad bots — strictly for my own amusement! ;)

   About your second point, I am not sure what you mean about Feedburner throwing emails.. Perhaps you refer to the “Subscribe to Comments” feature available for post comments? If so, I hope everything is working correctly, the plugin is supposed to send an email for every comment left on a post.

   Also, thanks for the catch on the ping/crawl trackback — flushed! ;)

9. 

   Jeff Starr 2008/11/17 9:37 am • Post Author

   @Donace: Ah, I see what you mean.. yes, all three of those posts were published on the same day (yesterday). Are you getting updates via Feed or email?

10. 

    Jeff Starr 2008/11/17 10:08 am • Post Author

    @Donace: ah, cool — I won’t worry about it then.. I guess I shouldn’t get so ambitious and stick to only a couple posts per week! ;)

11. 

    allie 2009/11/15 5:53 pm

    I came across this page because I was wondering why this isp address is checking up on me 67.195.111.221 All I know is that it comes from sunnyvale CA. I have a zaba search account, and they inform me whenever someone searches for me online. I have a very rare name, so I know it’s me their looking at. I have no clue why they keep doing this. I know nothing about this stuff. Any help would be greatly appreciated. Also, how can I get a hold of them to find out what they want? I kinda wanna give them a piece of my mind. It’s making me a little nervous that their looking me up all the time. Thanks

12. 

    Christina 2009/12/01 11:34 am

    Me too, they have been constantly checking on me since last August. I think it is really creepy and I do not know anything about any of this stuff either, any help is appreciated.

Comments are closed for this post. Something to add? Let me know.