5G Blacklist 2012
The 5G Blacklist helps reduce the number of malicious URL requests that hit your website. It’s one of many ways to improve the security of your site and protect against evil exploits, bad requests, and other nefarious garbage. If you’re tired of all the slow, bloated security plugins and expensive 3rd-party services, the 5G Blacklist is a solid solution to help protect your Apache-powered site.
Evolution
After extensive beta testing, the 5G Blacklist/Firewall is solid and ready to help secure sites hosted on Apache servers. In addition to beta testing for the 5G, this is the 5th major update of my “G”-series blacklists. Here is a quick overview of its evolution.
- Ultimate htaccess Blacklist (Compressed Version)
- 2G Blacklist: Closing the Door on Malicious Attacks
- Perishable Press 3G Blacklist
- The Perishable Press 4G Blacklist
- 5G Firewall (Beta)
Along the way, I’ve explored a wide variety of different blacklist techniques. The 5G is the culmination of all these efforts, and will eventually be replaced by the imminent 6G Blacklist/Firewall.
What it does
The 5G Blacklist is a simple, flexible blacklist that checks all URI requests against a series of carefully constructed HTAccess directives. This happens quietly behind the scenes at the server level, saving resources for stuff like PHP and MySQL for all blocked requests.
How it works
Blacklists can block just about any part of a request: IP, user agent, request string, query string, referrer, and everything in between. But IP addresses change constantly, and user agents and referrers are easily spoofed. As discussed, request strings yield the best results: greater protection with fewer false positives.
The 5G works beautifully with WordPress, and should help any site conserve bandwidth and server resources while protecting against malicious activity.
How to use
To install the 5G Firewall, append the following code to your site’s root .htaccess
:
# 5G BLACKLIST/FIREWALL
# @ https://perishablepress.com/5g-blacklist-2012/
# 5G:[QUERY STRINGS]
<ifModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{QUERY_STRING} (environ|localhost|mosconfig|scanner) [NC,OR]
RewriteCond %{QUERY_STRING} (menu|mod|path|tag)\=\.?/? [NC,OR]
RewriteCond %{QUERY_STRING} boot\.ini [NC,OR]
RewriteCond %{QUERY_STRING} echo.*kae [NC,OR]
RewriteCond %{QUERY_STRING} etc/passwd [NC,OR]
RewriteCond %{QUERY_STRING} \=\\%27$ [NC,OR]
RewriteCond %{QUERY_STRING} \=\\\'$ [NC,OR]
RewriteCond %{QUERY_STRING} \.\./ [NC,OR]
RewriteCond %{QUERY_STRING} \? [NC,OR]
RewriteCond %{QUERY_STRING} \: [NC,OR]
RewriteCond %{QUERY_STRING} \[ [NC,OR]
RewriteCond %{QUERY_STRING} \] [NC]
RewriteRule .* - [F]
</ifModule>
# 5G:[USER AGENTS]
<ifModule mod_setenvif.c>
SetEnvIfNoCase User-Agent ^$ keep_out
SetEnvIfNoCase User-Agent (casper|cmsworldmap|diavol|dotbot) keep_out
SetEnvIfNoCase User-Agent (flicky|ia_archiver|jakarta|kmccrew) keep_out
SetEnvIfNoCase User-Agent (libwww|planetwork|pycurl|skygrid) keep_out
SetEnvIfNoCase User-Agent (purebot|comodo|feedfinder|turnit) keep_out
SetEnvIfNoCase User-Agent (zmeu|nutch|vikspider|binlar|sucker) keep_out
<limit GET POST PUT>
Order Allow,Deny
Allow from all
Deny from env=keep_out
</limit>
</ifModule>
# 5G:[REQUEST STRINGS]
<ifModule mod_alias.c>
RedirectMatch 403 (https?|ftp|php)\://
RedirectMatch 403 /(cgi|https?|ima|ucp)/
RedirectMatch 403 /(Permanent|Better)$
RedirectMatch 403 (\=\\\'|\=\\%27|/\\\'/?|\)\.css\()$
RedirectMatch 403 (\,|//|\)\+|/\,/|\{0\}|\(/\(|\.\.\.|\+\+\+|\||\\\"\\\")
RedirectMatch 403 \.(cgi|asp|aspx|cfg|dll|exe|jsp|mdb|sql|ini|rar)$
RedirectMatch 403 /(contac|fpw|install|pingserver|register)\.php$
RedirectMatch 403 (base64|crossdomain|localhost|wwwroot|e107\_)
RedirectMatch 403 (eval\(|\_vti\_|\(null\)|echo.*kae|config\.xml)
RedirectMatch 403 \.well\-known/host\-meta
RedirectMatch 403 /function\.array\-rand
RedirectMatch 403 \)\;\$\(this\)\.html\(
RedirectMatch 403 proc/self/environ
RedirectMatch 403 msnbot\.htm\)\.\_
RedirectMatch 403 /ref\.outcontrol
RedirectMatch 403 com\_cropimage
RedirectMatch 403 indonesia\.htm
RedirectMatch 403 \{\$itemURL\}
RedirectMatch 403 function\(\)
RedirectMatch 403 labels\.rdf
RedirectMatch 403 /playing.php
RedirectMatch 403 muieblackcat
</ifModule>
# 5G:[BAD IPS]
<limit GET POST PUT>
Order Allow,Deny
Allow from all
# uncomment/edit/repeat next line to block IPs
# Deny from 123.456.789
</limit>
That’s the golden ticket right there. The 5G Firewall is serious protection for your website: extensively tested, plug-n-play, and completely free. “Grab, gulp, n go” as they say. For more information, see the beta article (and comments).
Troubleshooting
Remember, test thoroughly. If something stops working when the 5G is installed, try removing the 5G. If things start working normally again, you can either pass on the 5G or investigate further. Investigating further is straightforward using something like the halving method, where you remove chunks of the 5G until isolating and identifying the issue. Here is a quick example:
- I’ve installed the 5G, thanks Jeff.
- Uh-oh, the page at
http://example.com/indonesia.html
stopped loading - Hmm, the URL contains the phrase “indonesia”, so let’s check the 5G for it
- Yep, there’s a rule that blocks
indonesia\.htm
- Removing that line resolves the issue, thanks me.
Is it okay to remove rules that are blocking your own pages? Yes, the only downside is that malicious requests that would have otherwise been blocked will now get through. The 5G will continue to block a massive volume of malicious requests — it’ll just be a bit less effective. The protective effect is cumulative, not dependent on any one rule. So customization is encouraged. Once you dial it in, you’re all set.
Disclaimer
The 5G Firewall is provided “as-is”, with the intention of helping site administrators protect their sites against bad requests and other malicious activity. The code is open and free to use and modify as long as the first two credit lines remain intact. By using this code you assume all risk & responsibility for anything that happens, whether good or bad. In short, use wisely, test thoroughly, don’t sue me.
Learn more..
To learn more about the theory and development of the 5G Firewall, check out my articles on building the 3G, 4G and 5G Blacklist. A search for “blacklist” in the sidebar should also yield many results.
Happy securing!
223 responses to “5G Blacklist 2012”
Thanks for this, I am about to try this now on one of my blog sites. I will monitor it for at least a week.
Is there a way we can block content scraper bots?
Absolutely there is. Let’s say the scraper bot’s name is “evil scraper” (or whatever). We can block using the same directives/syntax as in the
[USER AGENTS]
section of the 5G:<ifModule mod_setenvif.c>
SetEnvIfNoCase User-Agent (evil.scraper) keep_out
<limit GET POST PUT>
Order Allow,Deny
Allow from all
Deny from env=keep_out
</limit>
</ifModule>
If you’re using the 5G, it can just be included along with the other bot-names in one of the
SetEnvIfNoCase
directives:SetEnvIfNoCase User-Agent (zmeu|nutch|vikspider|binlar|sucker|evil.scraper) keep_out
Note the dot “
.
” stands for any character.thanks Jeff!
By the way Jeff, do you have a list of bot scrapers?
Hi Guys
I found a problem I wanted to share. I use Piwik web analytics and my Google Adwords have urls like
damien.co/page-name/??pk-campaign
Visits to these pages were being blocked because of the rule
RewriteCond %{QUERY_STRING} ? [NC,OR]
Thought you’d like to know. Can you suggest a modification so I can allow sitename
/page-name/
??Cheers
Damien
Hi Damien,
Remove or comment-out that line should do the trick.. otherwise there could be other factors involved..
Hi Jeff,
First off, thank you for providing this incredible resource to website owners.
I’ve been using parts of the 5G and 6G setups for my install.
I’m curious about one thing – the request method sections. I’ve noticed that in your stupid htaccess tricks post and maybe one other place that you have a line like this:
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK|OPTIONS|HEAD)
RewriteRule .* - [F]
I’ve seen HEAD requests in my logs from various sources (grepnetbot.com (appears to be Amazon – maybe it’s Alexa?) and the LinkedIn bot).
Is there a reason one should deny HEAD requests at all? I can’t seem to find information about this on the web.
Thank you!
Great question. Blocking HEAD requests further decreases the liability on your server. Every potential point of entry is a potential attack vector, so closing as many of those doors as possible is what securing your site is all about. There is nothing inherently risky about HEAD requests, but most websites only need GET and PUT, so request-methods beyond that aren’t needed.
Feel free to allow HEAD requests by removing that string from the rule.
Hey there, I noticed that this rule:
RewriteCond %{QUERY_STRING} (menu|mod|path|tag)=.?/? [NC,OR]
causes default WP menu editor to not work, FYI. REmoving menu fixes it.
Also by the way I really love the resources you provide, thanks so much.
Interesting, I use 5G and have no problem with the WP Menu editor.
Curious… What theme are you using?
That was exactly my problem and Thomas Bacon comment was the solution (thx mate!).
Ken Daves, I’m using a WooTheme. Haven’t tested with TwentyTwelve, but why do you think this’d be an issue?
This looks very good to me. I have just gotten finished with dealing with clean up from rr.nu site hackers leaving “eval code 64” type crap in all the .php files. Easy to clean up once you know what to do, but still a pain in the “bottom”.
I am currently using a deny field in my .htaccess file with a list of those sites I have seen have gotten access. This is keeping them out so far, although WP Firewall is sending me regular reports of deflected attacks.
Would this 5G cover attacks from those type of hackers, or would I need to add it to my .htaccess along with?
Seriously good blog and info you are sharing. And you write in a way that makes it very simple to understand for us non-techies too, which is the biggest thing. Thanks heaps!
This works great for me on my servers, saves me a load of unwanted traffic, and unnecessary bandwidth. Thanks for all your hard work on this…..and everyone else who contributed to it!
I have some clients using Nginx, and I’m wondering if something similar to this can be implemented….anyone know a way to accomplish this?
Blocking blank user agents…
SetEnvIfNoCase User-Agent ^$ keep_out
…can have an unwelcome side effect if your site uses PayPal. PayPal IPN responses have a blank user agent and blocking it may break ecommerce functionality on your site.
Thanks Jeff, the blank user-agent will indeed be removed from the 6G Blacklist. For now, anyone who needs PayPal can simply remove that directive.
All your time spent on these blacklists and our strange questions is greatly appreciated. I seem to have trouble blocking one particular visitor by IP using a Deny from… line in the [BAD IPS] section. He continues to successfully POST login attempts to my WordPress installation. I’m not actually worried, he’ll never get in the way he’s going.
I confirmed this behavior by adding my own IP to the same section. I was unable to GET any page, as expected. Thus, syntax is correct. I then loaded the wp-login.php page from history and was able to successfully POST a login. The login page then successfully redirected a GET to an admin panel page. Server logs indicate an initial unsuccessful GET, then a successful POST and GET, all from my IP I placed in a Deny from… line.
I will probably use mod_rewrite.c to block POSTs by IP, so it’s no big deal. Just odd. Do you have any insight to explain this? My server is running Apache 2.2.
Thanks for the great article and resource set!
I am running Apache2 on Ubuntu with multiple virtual hosts. Am a green system administrator and this seems like a great way to secure the system. Since I have access to the apache2 main server config file(s), I suppose it would be most efficient to add G5 to one of the server config files (which one?) with hopes that the rules will be applied for all virtual sites. Is that a reasonable idea?
Tim, this is to be included in your .htaccess file. You have to have this file in the root of every website/virtualhost you want to use it with.
More than likely you already have this file under some sites, if not just create one and paste this into it.
Google will help you understand .htaccess files a little better as well.
Thank you for the reply and I understand your advice but am also considering the following prescription from the Apache HTTP tutorial:
“You should avoid using .htaccess files completely if you have access to httpd main server config file. Using .htaccess files slows down your Apache http server. Any directive that you can include in a .htaccess file is better set in a Directory block, as it will have the same effect with better performance.”
Correct, use the httpd.conf file if you have access to it. .htaccess is for per-directory configuring and for broader changes when httpd.conf isn’t available (eg shared hosting). Also keep in mind that “slows down” is relative and potentially insignificant in most situations. For example, add the 5G and clock the difference between .htaccess and httpd.conf.
Hi Jeff
Great article. I’m currently banging my head against a wall and freaking out about all the possible security issues I don’t actually know about. This helps a lot.
Just curious though, what, if any, plugins would you recommend to use in combination with these rules for some really solid security? OR, will these rules be considerably secure as is?
Hi Phil,
I don’t have any plugin recommendations because honestly I don’t use any security plugins in most cases. For example, Perishable Press is a highly visible target and the 5G/6G rules are all that’s needed to keep it secure. It all depends on the server setup and software, as to how much “locking down” is required. I know there’s a gazillion articles out there listing some top WP security plugins if needed.
Thanks Jeff
I might just see how the 5G/6G rules run side by side with the Bulletproof htaccess rules. I don’t suppose you have had any feedback regarding the combination of the two?
Hi Phil,
Provided you’re talking about WordPress have a look at the WordPress Security Checklist which you can get for free from http://www.wpsecuritychecklist.com
It gives you a good all round information about WordPress Security… not just plugins, but also stuff like using SFTP, renaming table prefixes etc etc…
Hi Anders
I came across that yesterday and used it as a guide to get a better understanding of what I need to do.
It was very helpful, but some of the plugins recommended haven’t been updated in some time and produce error notifications when wp_debug is set to true. It makes me wonder how many more WP updates the plugin can survive, and tends to discourage my usage of it in order to avoid the inevitable frustration of my clients when things break.
That is the main reason why I would love to find a solution such as the 5G/6G and feel confident that it is a consolidated approach without having to load up on plugins.
It is a handy guide though and helped me gain a clearer understanding of WP security requirements so I appreciate the work that went into it’s creation. I just wonder if there are more appropriate plugins that could take the place of a few of the recommended? Wordfence seems to be pretty decent and if Bulletproof was to get a cleaner, more intuitive interface, it also seems like a great option.
Disclaimer: I may not actually know what I am talking about… :D
Thoughts?
Hi Phil,
The update frequency of a plugin is indeed very important.
We’ve built the list recently and work with the recommended list on our own sites with good results. However software is a moving target, and as you say it is impossible to say when a WordPress Update will break something.
Have had a look at WordFence and it does look promising. However it is still in an early version, so it changes frequently. Therefore we’ve decided to wait till the dust settles a bit before we review it. It does have a lot of potential.
Things like Login Lockdown (either using WordFence or the Login Lockdown plugin) you will not be able to achieve with .htaccess rules, so you will unfortunately need a combination…
Hi Anders
Thanks for the feedback. I am already subscribed to your list and look forward to the WordFence review. I’m pretty impressed with it so far, mostly for it’s consolidated and easy to manage approach.
Thanks again for the hard work on the WP Security Checklist. I’m sure there are plenty of people sweating a little less with information like that freely available.
Cheers
Hi Phil,
I arrived here from the Block Bad Queries plugin, which I found about two years ago and installed on all the WP sites I maintain. I believe your plugin has kept the baddies from infiltrating my client databases — thank you very much!
I was redirected to this blacklist, and can’t seem to find the Block Bad Queries plugin anywhere, so my question is: does the 5G Blacklist replace the old plugin? Should I use them both, or will they conflict?
Thanks for your help!
My apologies, was addressing the above to *Jeff*, not Phil. Was reading the comment threads and got confused. Sorry!
Just an FYI –
On my WordPress site, I have the WP Super Cache plugin installed, and the “Delete Cache” button it puts on the admin bar seems to be broken due to the following query string filter:
RewriteCond %{QUERY_STRING} (menu|mod|path|tag)=.?/? [NC,OR]
Taking “|path” out of that line fixed it.