Articles tagged with “robots”
- Unexplained Crawl Behavior Involving Tagged Query Strings
- I need your help! I am losing my mind trying to solve another baffling mystery. For the past three or four months, I have been recording many 404 Errors generated from msnbot, Yahoo-Slurp, and other spider crawls. These errors result from invalid requests for URLs containing query strings such as the following:
http://perishablepress.com/press/page/2/?tag=spam
http://perishablepress.com/press/page/3/?tag=code
http://perishablepress.com/press/page/2/?tag=email
http://perishablepress.com/press/page/2/?tag=xhtml
http://perishablepress.com/press/page/4/?tag=notes
http://perishablepress.com/press/page/2/?tag=flash
http://perishablepress.com/press/page/2/?tag=links
http://perishablepress.com/press/page/3/?tag=theme
http://perishablepress.com/press/page/2/?tag=press
..plus hundreds and hundreds ...
- Taking Advantage of the X-Robots Tag
- Controlling the spidering, indexing and caching of your (X)HTML-based web pages is possible with meta robots directives such as these:
I use these directives here at Perishable Press and they continue to serve me well for controlling how the “big bots” 1 crawl and represent my (X)HTML-based content in search results.
For other, non-(X)HTML ...
- Yahoo! Slurp in My Blackhole (Yet Again)
- Yup, ‘ol Slurp is at it again, flagrantly disobeying specific robots.txt rules forbidding access to my bad-bot trap, lovingly dubbed the “blackhole.” As many readers know, this is not the first time Yahoo has been caught behaving badly. This time, Yahoo was caught trespassing five different times via three different IPs over the course of four different days. Here is the data recorded ...
- Yahoo! in my Blackhole
- Okay, I realize that the title sounds a bit odd, but nowhere near as odd as my recent discovery of Slurp ignoring explicit robots.txt rules and digging around in my highly specialized bot trap, which I have lovingly dubbed “the blackhole”. What is up with that, Yahoo!? — does your Slurp spider obey robots.txt directives or not? I have never seen Google crawling around that side of town, neither has MSN nor even Ask ventured into the ...
- Comprehensive Reference for WordPress NoNofollow/Dofollow Plugins
- Recently, while deliberating an optimal method for eliminating nofollow link attributes from Perishable Press, I collected, installed, tested and reviewed every WordPress no-nofollow/dofollow plugin that I could find. As of the writing of this post, I have evaluated 12 dofollow plugins, all of which are freely available on the Internet.
In this article, I present a concise, current, and comprehensive reference for WordPress no-nofollow and dofollow plugins. ...
- Stop WordPress from Leaking PageRank to Admin Pages
- [ Keywords: wordpress, pagerank, pr, admin, pages, login, register, meta, robots, search, google ]
During the most recent Perishable Press redesign, I noticed that several of my WordPress admin pages had been assigned significant levels of PageRank. Not good. After some investigation, I realized that my ancient robots.txt rules were insufficient in preventing Google from indexing various WordPress admin pages. Specifically, the following pages have been indexed and subsequently assigned PageRank:
WP Admin Login Page
...
- Eliminate 404 Errors for PHP Functions
- Recently, I discussed the suspicious behavior recently observed by the Yahoo! Slurp crawler. As revealed by the site’s closely watched 404-error logs, Yahoo! had been requesting a series of nonexistent resources. Although a majority of the 404 errors were exclusive to the Slurp crawler, there were several instances of requests that were also coming from Google, Live, and even Ask. Initially, these distinct errors were misdiagnosed as existing ...
- Suspicious Behavior from Yahoo! Slurp Crawler
- [ Keywords: yahoo, slurp, crawl, crawling, spider, url, 404, errors, suspicious, behavior ]
Most of the time, when I catch scumbags attempting to spam, scrape, leech, or otherwise hack my site, I stitch up a new voodoo doll and let the cursing begin. No, seriously, I just blacklist the idiots. I don’t ...
- Invite Only: Visitor Exclusivity via the Opt-In Method
- Web developers trying to control comment-spam, bandwidth-theft, and content-scraping must choose between two fundamentally different approaches: selectively deny target offenders (the "blacklist" method) or selectively allow desirable agents (the "opt-in", or "whitelist" method).
Currently popular according to various online forums and discussion boards is the blacklist method. The blacklist method requires the webmaster to create and maintain a working list of undesirable agents, usually blocking their access via htaccess or php. The downside of "blacklisting" is that ...
- Disobedient Robots and Company
- In our never-ending battle against spammers, leeches, scrapers, and other online undesirables, we have implemented several powerful security measures to improve the operational integrity of our perpetual virtual existence. Here is a rundown of the new behind-the-scenes security features of Perishable Press:
Automated spambot trap, designed to identify bots (and/or stupid people) that disobey rules specified in the site’s robots.txt file.
Automated disobedient-robot identification (via reverse IP lookup), admin-notification (via email) and blacklist inclusion (via htaccess).
Automated inclusion of ...
- Stop Bitacle from Stealing Content
- If you have yet to encounter the content-scraping site, bitacle.org, consider yourself lucky. The scum-sucking worm-holes at bitacle.org are well-known for literally, blatantly, and piggishly stealing blog content and using it for financial gains through advertising. While I am not here to discuss the legal, philosophical, or technical ramifications of illegal bitacle behavior, I am here to provide a few critical tools that will help ...
- Robots Notes Plus
- About the Robots Exclusion Standard1:
The robots exclusion standard or robots.txt protocol is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website. The information specifying the parts that should not be accessed is specified in a file called robots.txt in the top-level directory of the website.
Notes on the robots.txt Rules:
Rules of specificity apply, not inheritance. Always include a blank line between rules. Note also that not all robots ...