Protect Your Site with a Blackhole for Bad Bots
One of my favorite security measures here at Perishable Press is the site’s virtual Blackhole trap for bad bots. The concept is simple: include a hidden link to a robots.txt
-forbidden directory somewhere on your pages. Bots that ignore or disobey your robots rules will crawl the link and fall into the honeypot trap, which then performs a WHOIS Lookup and records the event in the blackhole data file. Once added to the blacklist data file, bad bots immediately are denied access to your site.
Contents
- Intro
- Overview
- Live Demo
- How to Install
- Testing
- Customize
- Troubleshoot
- Caveat Emptor
- Whitelist Good Bots
- License & Disclaimer
- Questions & Feedback
- Download
Intro
I call it the “one-strike” rule: bots have one chance to follow the robots.txt protocol, check the site’s robots.txt file, and obey its directives. Failure to comply results in immediate banishment. The best part is that the Blackhole only affects bad bots: normal users never see the hidden link, and good bots obey the robots rules in the first place. So the percentage of false positives is extremely low to non-existent. It’s an ideal way to protect your site against bad bots silently, efficiently, and effectively.
With a few easy steps, you can set up your own Blackhole to trap bad bots and protect your site from evil scripts, bandwidth thieves, content scrapers, spammers, and other malicious behavior.
The Blackhole is built with PHP, and uses a bit of .
htaccess
to protect the blackhole directory. Refined over the years and completely revamped for this tutorial, the Blackhole consists of a plug-&-play /blackhole/
directory that contains the following three files:
.htaccess
– protects the log fileblackhole.dat
– log fileindex.php
– blackhole script
These three files work together to create the Blackhole for Bad Bots. If you are running WordPress, the Blackhole plugin is recommended instead of this standalone PHP version.
Overview
The Blackhole is developed to make implementation as easy as possible. Here is an overview of the steps:
- Upload the
/blackhole/
directory to your site - Edit the four variables in the “EDIT HERE” section in
index.php
. - Ensure writable server permissions for the
blackhole.dat
file - Add a single line to the top of your pages to include the
index.php
file - Add a hidden link to the
/blackhole/
directory in the footer - Forbid crawling of
/blackhole/
by adding a line to your robots.txt
So installation is straightforward, but there are many ways to customize functionality. For complete instructions, jump ahead to the installation steps. For now, I think a good way to understand how it works is to check out a demo..
Live Demo
I have set up a working demo of the Blackhole for this tutorial. It works exactly like the download version, but it’s set up as a sandbox, so when you trigger the trap, it blocks you only from the demo itself. Here’s how it works:
- First visit to the Blackhole demo loads the trap page, runs the whois lookup, and adds your IP address to the blacklist data file
- Once your IP is added to the blacklist, all future requests for the Blackhole demo will be denied access
So you get one chance (per IP address) to see how it works. Once you visit the demo, your IP address will be blocked from the demo only — you will still have full access to this tutorial (and everything else at Perishable Press). So with that in mind, here is the demo link (opens new tab):
Visit once to see the Blackhole trap, and then again to observe that you’ve been blocked. Again, even if you are blocked from the demo page, you will continue to have access to everything else here at Perishable Press.
How to Install
Here are complete instructions for implementing the PHP/standalone of Blackhole for Bad Bots. Note that these steps are written for Apache servers running PHP. The steps are the same for other PHP-enabled servers (e.g., Nginx, IIS), but you will need to replace the .htaccess file and rules with whatever works for particular server environment. Note: for a concise summary of these steps, check out this tutorial.
Step 1: Download the Blackhole zip file, unzip and upload to your site’s root directory. This location is not required, but it enables everything to work out of the box. To use a different location, edit the include
path in Step 4.
Step 2: Edit the four variables in the “EDIT HERE” section in index.php
.
Step 3: Change file permissions for blackhole.dat
to make it writable by the server. The permission settings may vary depending on server configuration. If you are unsure about this, ask your host. Note that the blackhole script needs to be able to read, write, and execute the blackhole.dat
file.
Step 4: Include the Blackhole script by adding the following line to the top of your pages (e.g., header.php
):
<?php include(realpath(getenv('DOCUMENT_ROOT')) . '/blackhole/index.php'); ?>
The Blackhole script checks the bot’s IP address against the blacklist data file. If a match is found, the request is blocked with a customizable message. View the source code for more information.
Step 5: Add a hidden link to the /blackhole/
directory in the footer of your site’s web pages (replace “Your Site Name” with the name of your site):
<a rel="nofollow" style="display:none" href="https://example.com/blackhole/" title="Do NOT follow this link or you will be banned from the site!">Your Site Name</a>
This is the hidden trigger link that bad bots will follow. It’s currently hidden with CSS, so 99.999% of visitors won’t ever see it. Alternately, to hide the link from users without relying on CSS, replace the anchor text with a transparent 1-pixel GIF image. For example:
<a rel="nofollow" style="display:none" href="http://example.com/blackhole/" title="Do NOT follow this link or you will be banned from the site!"><img src="/images/1px.gif" alt=""></a>
Remember to edit the link href
value and the image src
to match the correct locations on your server.
Step 6: Finally, add a Disallow
directive to your site’s robots.txt
file:
User-agent: *
Disallow: /blackhole/
This step is pretty important. Without the proper robots directives, all bots would fall into the Blackhole because they wouldn’t know any better. If a bot wants to crawl your site, it must obey the rules! The robots rule that we are using basically says, “All bots DO NOT visit the /blackhole/
directory or anything inside of it.” So it is important to get your robots rules correct.
Step 7: Done! Remember to test thoroughly before going live. Also check out the section on customizing for more ideas.
Testing
You can verify that the script is working by visiting the hidden trigger link (added in step 5). That should take you to the Blackhole warning page for your first visit, and then block you from further access on subsequent visits. To verify that you’ve been blocked entirely, try visiting any other page on your site. To restore site access at any time, you can clear the contents of the blackhole.dat
log file.
Important: Make sure that all of the rules in your robots.txt file are correct and have proper syntax. For example, you can use the free robots.txt validator in Google Webmaster Tools (requires Google account).
chrome
from the whitelist.blackhole.dat
file.Customize
The previous steps will get the Blackhole set up with default configuration, but there are some details that you may want to customize:
index.php
(lines 25–28): Edit the four variables as neededindex.php
(lines 140–164): Customize markup of the warning pageindex.php
(line 180): Customize the list of whitelisted bots
These are the recommended changes, but the PHP is clean and generates valid HTML, so feel free to modify the markup or anything else as needed.
Troubleshoot
If you get an error letting you know that a file cannot be found, it could be an issue with how the script specifies the absolute path, using getenv('DOCUMENT_ROOT')
. That function works on a majority of servers, but if it fails on your server for whatever reason, you can simply replace it with the actual path. From Step 4, the include script looks like this:
<?php include(realpath(getenv('DOCUMENT_ROOT')) . '/blackhole/index.php'); ?>
So if you are getting not-found or similar errors, try this instead:
/var/www/httpdocs/blackhole/index.php
So that would be the actual absolute path to the blackhole index.php
file on your server. As long as you get the path correct, it’s gonna fix any “file can’t be found” type errors you may be experiencing.
If in doubt about the actual full absolute path, consult your web host or use a PHP function or constant such as __DIR__
to obtain the correct infos. And check out my tutorial over at WP-Mix for more information about including files with PHP and WordPress.
Caveat Emptor
Blocking bots is serious business. Good bots obey robots.txt
rules, but there may be potentially useful bots that do not. Yahoo is the perfect example: it’s a valid search engine that sends some traffic, but sadly the Yahoo Slurp bot is too stupid to follow the rules. Since setting up the Blackhole several years ago, I’ve seen Slurp disobey robots rules hundreds of times.
By default, the Blackhole DOES NOT BLOCK any of the big search engines. So Google, Bing, and company always will be allowed access to your site, even if they disobey your robots.txt
rules. See the next section for more details.
Whitelist Good Bots
In order to ensure that all of the major search engines always have access to your site, Blackhole whitelists the following bots:
- AOL.com
- Baidu
- Bing/MSN
- DuckDuckGo
- Teoma
- Yahoo!
- Yandex
Additionally, popular social media services are whitelisted, as well as some other known “good” bots. To whitelist these bots, the Blackhole script uses regular expressions to ensure that all possible name variations are allowed access. For each request made to your site, Blackhole checks the User Agent and always allows anything that contains any of the following strings:
a6-indexer, adsbot-google, ahrefsbot, aolbuild, apis-google, baidu, bingbot, bingpreview, butterfly, chrome, cloudflare, duckduckgo, embedly, facebookexternalhit, facebot, googlebot, google page speed, ia_archiver, linkedinbot, mediapartners-google, msnbot, netcraftsurvey, outbrain, pinterest, quora, rogerbot, showyoubot, slackbot, slurp, sogou, teoma, tweetmemebot, twitterbot, uptimerobot, urlresolver, vkshare, w3c_validator, wordpress, wp rocket, yandex
So any bot that reports a user agent that contains any of these strings will NOT be blocked and always will have full access to your site. To customize the list of whitelisted bots, open index.php
and locate the function blackhole_whitelist()
, where you will find the list of allowed bots.
The upside of whitelisting these user agents ensures that anything claiming to be a major search engine is allowed open access. The downside is that user-agent strings are easily spoofed, so a bad bot could crawl along and say, “Hey look, I’m teh Googlebot!” and the whitelist would grant access. It is your decision where to draw the line.
With PHP, it is possible to verify the true identity of each bot, but doing so consumes significant resources and could overload the server. Avoiding that scenario, the Blackhole errs on the side of caution: it’s better to allow a few spoofs than to block any of the major search engines and other major web services.
License & Disclaimer
Terms of Use: Blackhole for Bad Bots is released under GNU General Public License. By downloading the Blackhole, you agree to accept full responsibility for its use. In no way shall the author be held accountable for anything that happens after the file has been downloaded.
Questions & Feedback
Questions? Comments? Send ’em via my contact form. Thanks!
Download
Here you can download the latest version of Blackhole for Bad Bots. By downloading, you agree to the terms.
240 responses to “Protect Your Site with a Blackhole for Bad Bots”
Update: Thanks to help from X3M, the Blackhole now whitelists the major search engines: Googlebot, Slurp, msnbot, Teoma, Yandex. Please see this section in the article for more information. If you are using a version less than 1.2, it is recommended to update.
I’m not an expert like you and I can’t contribute anything, just I can say thank you. I will test it.
I’m using a rather similar system to block access to some parts of my site, esp. to the download section for my project history. That specifically is this way because I do not want to get my complete contact data indexed by some spam bot, Google or anything else.
A short suggestion to improve your system: Rename the DAT-Logfile to .dat.php to avoid getting it read from the outside, because there are lots of scenarios where you simply CANNOT put this someplace under the web root and/or not being able to properly set the access rights.
cu, w0lf.
Hi Jeff, that’s a interesting idea, but I do foresee 2 problems:
– Some browsers/addons prefetch links on the page (f.e. the Firefox: fasterfox addon)
– Competitors could make their visitors visit your blackbox; for example by including an image pointed to ‘http://site/blackbox/’ in their HTML, thereby banning them from your site.
I think the best solution would be to split the trap into two pages. The /blackbox/ page doesn’t ban the user but links to another page that does. The URL of that page could depend on the IP of the user, for example “http://site/blackbox/?key=”+MD5(ip+”secret”). That way, there would be no way of hotlinking your ban page and prefetching is allowed 1-page in advance. :)
Another interesting solution might be to automatically unban anyone with Javascript enabled, since most bots don’t use Javascript.
Thanks for the script!
Very useful ! I was thinking about something to block bad robots and stumble on it ! Wunderbar !
Thanks
In version 1.2 why are there already 55 lines of IP addresses and other details in the blackhole.dat file?
I recommend renaming the blackhole.dat file to be .htblackhole.dat. Most apache severs will not allow anyone to download any file that starts with .ht. After renaming the file, you need to edit line 37 in blackhole.php and line 127 in index.php.
You should also add the exclusion command to your robots.txt file. Then activate the actual blackhole several days alter. Search engines do not check the robots.txt file every time they visit your site. Most cache the instructions for anywhere from 1 to 7 days.
Final recommendation is to change the name of the directory to something innocuous. Bad bots may try to avoid directory names like “blackhole.”
And you laos need to tweak the footer of index.php to report the current version number, it still says 1.1.
And you also need to tweak the footer of index.php to report the current version number, it still says 1.1.
Thanks Jeff for this awesome trap. I used to block bad bots manually (via .htaccess) and when I hooked up Blackhole on a test server it worked like nuts. I got to implement it on several websites under my belt, soon.
I don’t know much about this, but I’m curious: does publishing a technique like this make it more vulnerable to being beaten/worked around by spammers? I was wondering recently about honeypots on contact forms, which I guess can now be beaten by spammers. Was too much published about honeypots, or was it too simple a technique to keep spammers at bay for long?
I do not think that it is safe to hide text as you recommend using this code:
<a href="http://example.com/blackhole/" rel="nofollow">Do NOT follow this link or you will be banned from the site!</a>
The risk is that “display:none” for text can trigger spam filters especially with Google. Even if you did not have any evil intentions.
I would rather use javascript in combination with a “noscript” tag, or instead of using an anchor text, I would use a linked transparent image.
Another thing I would like to mention is, that using the “nofollow” attribute for internal links is not advisable. With such practices you can dilute PageRank.
So for this case, I would recommend implementing in the index.php file the robots meta tag directives “noindex,nofollow,noarchive”.
In that case the major search engines will still access the page, crawl it but will not index it.
But! The PageRank will still flow to the pages which are linked from that page. If they are external site links, there you can block passing PageRank with the rel=”nofollow” attribute. But you must make sure that you have at least one link that the PR must pass to, i.e to the homepage of your site, otherwise you will have again a PageRank dilution, because you have created a so known as dangling or dead end page.
I hope you will update above and if not, I will take care and modify all that before I implement.
By the way great job Jeff!
HI Jeff, thank you very much for this great little plugin!
I’ve just installed it on my site and found out that my site was craveling by “Baiduspider ” – It appears to be a search engine from China and it’s disregarding the nofollow rule and the robots.txt.
Have anyone of you any experience with this spider?
Do you thhing this might by harmless bot?
I just do not like the fact it’s disregarding the rulles.
Here are some info I’ve got:
IP Address: 123.125.66.22
User Agent: Baiduspider+(+http://www.baidu.com/search/spider.htm)