One of my favorite security measures here at Perishable Press is the site’s virtual Blackhole trap for bad bots. The concept is simple: include a hidden link to a
robots.txt-forbidden directory somewhere on your pages. Bots that ignore or disobey your robots rules will crawl the link and fall into the honeypot trap, which then performs a WHOIS Lookup and records the event in the blackhole data file. Once added to the blacklist data file, bad bots immediately are denied access to your site. I call it the “one-strike” rule: bots have one chance to follow the robots.txt protocol, check the site’s
robots.txt file, and obey its directives. Failure to comply results in immediate banishment. The best part is that the Blackhole only affects bad bots: normal users never see the hidden link, and good bots obey the robots rules in the first place.
In five easy steps, you can set up your own Blackhole to trap bad bots and protect your site from evil scripts, bandwidth thieves, content scrapers, spammers, and other malicious behavior.
The Blackhole is built with PHP, and uses a bit of
htaccess to protect the blackhole directory. The blackhole script combines heavily modified versions of the Kloth.net script (for the bot trap) and the Network Query Tool (for the whois lookups) (404 link removed 2012/07/08). Refined over the years and completely revamped for this tutorial, the Blackhole consists of a single plug-&-play directory that contains the following four files:
.htaccess– basic directory protection
blackhole.dat– server-writable log file (serves as the blacklist)
blackhole.php– checks requests against blacklist and blocks bad bots
index.php– generates blackhole page, performs whois lookup, sends email, and logs data
These four files are all contained in a single directory named “
Note: By default, .htaccess files are hidden on Windows and OS X, so to view them you need to enable “Show hidden files” on your machine, or use any FTP or code-editing software that is capable of displaying them (it’s a common feature).
I set things up to make implementation as easy as possible. Here are the five basic steps:
- Upload the
/blackhole/directory to your site
- Ensure writable server permissions for the
- Add a single line to the top of your pages to include the
- Add a hidden link to the
/blackhole/directory in the footer of your pages
- Prohibit crawling of the
/blackhole/by adding a line to your
It’s that easy to install on your own site, but there are many ways to customize functionality. For complete instructions, jump ahead to Implementation and Configuration. For now, I think a good way to understand how it works is to check out a demo..
One-time Live Demo
I have set up a working demo of the Blackhole for this tutorial. It works exactly like the download version, but it’s configured to block you only from the demo, not from the entire site. Here’s how it works:
- First visit to the Blackhole demo loads the trap page, runs the whois lookup, and adds your IP address to the blacklist data file
- Once you’re added to the blacklist, all subsequent requests for the Blackhole demo will be denied access
So you get one chance to see how it works. Once you visit, your IP will be blocked from the demo only – you will still have full access to this tutorial (and everything else). That said, here is the demo link: Blackhole Demo. Visit once to see the Blackhole trap, and then again to observe that you’ve been blocked. If I were to include the
blackhole.php in the header of my theme files, you would be banned from pretty much the entire site.
Implementation and Configuration
Here are complete instructions for implementing and configuring the Perishable Press Blackhole. Note that these steps are for implementation on Apache servers. For Nginx servers, check out Stephane’s modified tutorial.
Step 1: Download the Blackhole zip file, unzip and upload the entire directory to your site’s root directory. This location is not required, but it enables everything to work out of the box. To use a different location, edit the
include path in Step 3.
Step 2: Change file permissions for
blackhole.dat to make it writable by the server. The permission settings may vary depending on server configuration. If you are unsure about this, ask your host. Note that the blackhole script needs to be able to read, write, and execute the
Step 3: Include the bot-check script by adding the following line to the top of your pages:
<?php include(realpath(getenv('DOCUMENT_ROOT')) .'/blackhole/blackhole.php'); ?>
blackhole.php script checks the request IP against the blacklist data file. If a match is found, the request is blocked with a customizable message. See the source code for more information.
Step 4: Include a hidden link to the
/blackhole/ directory in the footer of your pages:
<a rel="nofollow" style="display:none;" href="http://example.com/blackhole/">Do NOT follow this link or you will be banned from the site!</a>
This is the hidden link that bad bots will follow. It’s currently hidden with CSS, so 99.9% of visitors won’t ever see it. Alternately, to hide the link from users without relying on CSS, replace the anchor text with a transparent 1-pixel GIF image. For example:
<a rel="nofollow" style="display:none;" href="http://example.com/blackhole/" title="Do NOT follow this link or you will be banned from the site!"><img src="/images/1px.gif" alt="" /></a>
Remember to edit the image path to match the correct location on your server.
Step 5: Finally, add a
Disallow directive to your site’s
User-agent: * Disallow: /blackhole/
This step is pretty important. Without the proper robots directives, all bots would fall into the Blackhole because they wouldn’t know any better. If a bot wants to crawl your site, it must obey the rules! The robots rule that we are using basically says, “All bots DO NOT visit the
/blackhole/ directory or anything inside of it.” More on this in the next section..
Further customization: The previous five steps will get the Blackhole working, but there are some details that you’ll want to customize:
index.php(lines 27/28): Edit to/from email addresses with your own
index.php(lines 206/229): Check/replace path to your contact form
blackhole.php(line 56): Check/replace path to your contact form
These are the recommended changes, but the PHP is clean and generates valid HTML5, so feel free to modify the source code as needed. Note that beyond these three items, no other edits need made.
Here is a short list of other things that may be customized as desired:
index.php(line 52): Set specific timezone
index.php(line 226): Comment out this line to disable email notifications
index.php(line 157) and
blackhole.php(line 29): Edit list of whitelisted bots
index.php(line 158) and
blackhole.php(line 30): customize header redirect location for whitelisted bots
Note about file paths:
If you get an error letting you know that a file cannot be found, it could be an issue with how the script indicates the absolute path, using
getenv('DOCUMENT_ROOT'). That function works on a majority of servers, but if it’s not working on your server for whatever reason, you can simply replace it with the actual path. For example, here are the paths as specified in the script:
- Line to include the script (step 3 above):
Notice the difference between
.php in these lines. So, after changing these lines to specify the explicit absolute paths, they would look something like this:
- Line to include the script (step 3 above):
And that’s gonna fix any “file can’t be found” type errors you may be experiencing. If in doubt about the actual full absolute path, consult your web host or use a PHP function or constant such as
__DIR__ to obtain the correct infos.
Blocking bots is serious business. Good bots obey
robots.txt rules, but there may be potentially useful bots that do not. Yahoo is the perfect example: it’s a valid search engine that sends some traffic, but sadly the Yahoo Slurp bot is too stupid to follow the rules. Since setting up the Blackhole several years ago, I’ve seen Slurp disobey robots rules hundreds of times.
Bottom line: the Blackhole will block any bot that disobeys the Update: By default, the Blackhole no longer blocks any of the popular search engines. See the next section for more information.
robots.txt directives. Proceed accordingly.
Whitelisting Search Bots
Initially, the Blackhole blocked any bot that disobeyed the
robots.txt directives. Unfortunately, as discussed in the comments, Googlebot, Yahoo, and other major search bots do not always obey robots rules. And while blocking Yahoo! Slurp is debatable, blocking Google, MSN/Bing, et al would just be dumb. Thus, the Blackhole now whitelists all bots related to any of the following search engines:
More specifically, here is the list of regex strings that are checked for each request:
aolbuild, baidu, bingbot, bingpreview, msnbot, duckduckgo, adsbot-google, googlebot, mediapartners-google, teoma, slurp, yandex
So any bot that reports a user agent that contains any of these strings will NOT be blocked and will have full access to your site under all conditions.
Whitelisting these user agents ensures that anything claiming to be a major search engine is allowed open access. The downside is that user-agent strings are easily spoofed, so a bad bot could crawl along and say, “hey look, I’m teh Googlebot!” and the whitelist would grant access. It is possible to verify the true identity of each bot, but as X3M explains in the comments, doing so consumes significant resources and could overload the server. Avoiding that scenario, the Blackhole errs on the side of caution: it’s better to allow a few spoofs than to block any of the major search engines.
License and Disclaimer
The Perishable Press Blackhole is released under GNU General Public License. Check the Creative Commons for a summary and/or see the Blackhole source code for additional information. Also note that by downloading the Blackhole, you agree to accept full responsibility for its use. In no way shall the author be held accountable for anything that happens after the file has been downloaded.
Questions & Feedback
Questions? Comments? Send ’em via my contact form. Thanks!
Here you can download the current version of the Blackhole:
Blackhole for Bad Bots – Version 3.1 ( zip) — updated 2016/02/17