5G Blacklist 2012
![[ 5G (2012) ]](https://perishablepress.com/wp/wp-content/images/2012/5G-Blacklist-2012.png)
The 5G Blacklist helps reduce the number of malicious URL requests that hit your website. It’s one of many ways to improve the security of your site and protect against evil exploits, bad requests, and other nefarious garbage. If you’re tired of all the slow, bloated security plugins and expensive 3rd-party services, the 5G Blacklist is a solid solution to help protect your Apache-powered site.
Evolution
After extensive beta testing, the 5G Blacklist/Firewall is solid and ready to help secure sites hosted on Apache servers. In addition to beta testing for the 5G, this is the 5th major update of my “G”-series blacklists. Here is a quick overview of its evolution.
- Ultimate htaccess Blacklist (Compressed Version)
- 2G Blacklist: Closing the Door on Malicious Attacks
- Perishable Press 3G Blacklist
- The Perishable Press 4G Blacklist
- 5G Firewall (Beta)
Along the way, I’ve explored a wide variety of different blacklist techniques. The 5G is the culmination of all these efforts, and will eventually be replaced by the imminent 6G Blacklist/Firewall.
What it does
The 5G Blacklist is a simple, flexible blacklist that checks all URI requests against a series of carefully constructed HTAccess directives. This happens quietly behind the scenes at the server level, saving resources for stuff like PHP and MySQL for all blocked requests.
How it works
Blacklists can block just about any part of a request: IP, user agent, request string, query string, referrer, and everything in between. But IP addresses change constantly, and user agents and referrers are easily spoofed. As discussed, request strings yield the best results: greater protection with fewer false positives.
The 5G works beautifully with WordPress, and should help any site conserve bandwidth and server resources while protecting against malicious activity.
How to use
To install the 5G Firewall, append the following code to your site’s root .htaccess:
# 5G BLACKLIST/FIREWALL
# @ https://perishablepress.com/5g-blacklist-2012/
# 5G:[QUERY STRINGS]
<ifModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{QUERY_STRING} (environ|localhost|mosconfig|scanner) [NC,OR]
RewriteCond %{QUERY_STRING} (menu|mod|path|tag)\=\.?/? [NC,OR]
RewriteCond %{QUERY_STRING} boot\.ini [NC,OR]
RewriteCond %{QUERY_STRING} echo.*kae [NC,OR]
RewriteCond %{QUERY_STRING} etc/passwd [NC,OR]
RewriteCond %{QUERY_STRING} \=\\%27$ [NC,OR]
RewriteCond %{QUERY_STRING} \=\\\'$ [NC,OR]
RewriteCond %{QUERY_STRING} \.\./ [NC,OR]
RewriteCond %{QUERY_STRING} \? [NC,OR]
RewriteCond %{QUERY_STRING} \: [NC,OR]
RewriteCond %{QUERY_STRING} \[ [NC,OR]
RewriteCond %{QUERY_STRING} \] [NC]
RewriteRule .* - [F]
</ifModule>
# 5G:[USER AGENTS]
<ifModule mod_setenvif.c>
SetEnvIfNoCase User-Agent ^$ keep_out
SetEnvIfNoCase User-Agent (casper|cmsworldmap|diavol|dotbot) keep_out
SetEnvIfNoCase User-Agent (flicky|ia_archiver|jakarta|kmccrew) keep_out
SetEnvIfNoCase User-Agent (libwww|planetwork|pycurl|skygrid) keep_out
SetEnvIfNoCase User-Agent (purebot|comodo|feedfinder|turnit) keep_out
SetEnvIfNoCase User-Agent (zmeu|nutch|vikspider|binlar|sucker) keep_out
<limit GET POST PUT>
Order Allow,Deny
Allow from all
Deny from env=keep_out
</limit>
</ifModule>
# 5G:[REQUEST STRINGS]
<ifModule mod_alias.c>
RedirectMatch 403 (https?|ftp|php)\://
RedirectMatch 403 /(cgi|https?|ima|ucp)/
RedirectMatch 403 /(Permanent|Better)$
RedirectMatch 403 (\=\\\'|\=\\%27|/\\\'/?|\)\.css\()$
RedirectMatch 403 (\,|//|\)\+|/\,/|\{0\}|\(/\(|\.\.\.|\+\+\+|\||\\\"\\\")
RedirectMatch 403 \.(cgi|asp|aspx|cfg|dll|exe|jsp|mdb|sql|ini|rar)$
RedirectMatch 403 /(contac|fpw|install|pingserver|register)\.php$
RedirectMatch 403 (base64|crossdomain|localhost|wwwroot|e107\_)
RedirectMatch 403 (eval\(|\_vti\_|\(null\)|echo.*kae|config\.xml)
RedirectMatch 403 \.well\-known/host\-meta
RedirectMatch 403 /function\.array\-rand
RedirectMatch 403 \)\;\$\(this\)\.html\(
RedirectMatch 403 proc/self/environ
RedirectMatch 403 msnbot\.htm\)\.\_
RedirectMatch 403 /ref\.outcontrol
RedirectMatch 403 com\_cropimage
RedirectMatch 403 indonesia\.htm
RedirectMatch 403 \{\$itemURL\}
RedirectMatch 403 function\(\)
RedirectMatch 403 labels\.rdf
RedirectMatch 403 /playing.php
RedirectMatch 403 muieblackcat
</ifModule>
# 5G:[BAD IPS]
<limit GET POST PUT>
Order Allow,Deny
Allow from all
# uncomment/edit/repeat next line to block IPs
# Deny from 123.456.789
</limit>That’s the golden ticket right there. The 5G Firewall is serious protection for your website: extensively tested, plug-n-play, and completely free. “Grab, gulp, n go” as they say. For more information, see the beta article (and comments).
Troubleshooting
Remember, test thoroughly. If something stops working when the 5G is installed, try removing the 5G. If things start working normally again, you can either pass on the 5G or investigate further. Investigating further is straightforward using something like the halving method, where you remove chunks of the 5G until isolating and identifying the issue. Here is a quick example:
- I’ve installed the 5G, thanks Jeff.
- Uh-oh, the page at
http://example.com/indonesia.htmlstopped loading - Hmm, the URL contains the phrase “indonesia”, so let’s check the 5G for it
- Yep, there’s a rule that blocks
indonesia\.htm - Removing that line resolves the issue, thanks me.
Is it okay to remove rules that are blocking your own pages? Yes, the only downside is that malicious requests that would have otherwise been blocked will now get through. The 5G will continue to block a massive volume of malicious requests — it’ll just be a bit less effective. The protective effect is cumulative, not dependent on any one rule. So customization is encouraged. Once you dial it in, you’re all set.
Disclaimer
The 5G Firewall is provided “as-is”, with the intention of helping site administrators protect their sites against bad requests and other malicious activity. The code is open and free to use and modify as long as the first two credit lines remain intact. By using this code you assume all risk & responsibility for anything that happens, whether good or bad. In short, use wisely, test thoroughly, don’t sue me.
Learn more..
To learn more about the theory and development of the 5G Firewall, check out my articles on building the 3G, 4G and 5G Blacklist. A search for “blacklist” in the sidebar should also yield many results.
Happy securing!
223 responses to “5G Blacklist 2012”
You can’t switch or add a new menu, because the “
menu=” RewriteCondFixed in the upcoming 6G, thanks.
Brilliant stuff…
Just wanna add my two cents…
Got 403 Permission denied when I tried to performance test my WordPress site using Pingdoms Full Page Test and gtmetrix.com.
Had to change:
RedirectMatch 403 (,|//|)+|/,/|{0}|(/(|...|+++|||\"\")To:
RedirectMatch 403 (,|)+|/,/|{0}|(/(|...|+++|||\"\")I.e. removing the second test (“//”)…
Thanks for publishing and maintaining!!
Thanks for the heads up, Anders!
Little new problem with 5G :)
I’ve a download section, managed by Worpdress Download Monitor. One of most donwloaded files has
.rarextension (http://www.skyflash.it/downloads/?did=4) and when visitors try to download it, gets the 403 error because a rar rule in 5GI’ve some bad crawlers that try to download specific rar files (that are not present on my site)
61.160.194.112 - - [07/May/2012:19:46:58 +0200] "GET /dmtamirserver.rar HTTP/1.1" 403 1809 "-" "Mozilla/4.0"61.160.194.112 - - [07/May/2012:19:46:59 +0200] "GET /kjfrmirserver1.rar HTTP/1.1" 403 1809 "-" "Mozilla/4.0"61.160.194.112 - - [07/May/2012:19:47:00 +0200] "GET /cpromirserverxd0xc2xc7xf8.rar HTTP/1.1" 403 1809 "-" "Mozilla/4.0"61.160.194.112 - - [07/May/2012:19:47:01 +0200] "GET /ajulmirserverxc0xcfxc7xf8.rar HTTP/1.1" 403 1809 "-" "Mozilla/4.0"61.160.194.112 - - [07/May/2012:19:47:02 +0200] "GET /lokjmirserver.zip HTTP/1.1" 500 54576 "-" "Mozilla/4.0"61.160.194.112 - - [07/May/2012:19:47:03 +0200] "GET /mirserver.zip HTTP/1.1" 500 54576 "-" "Mozilla/4.0"...so i’d like to block this crawlers with the rar rule, but i have to “free” my rar downloads
Any advice, pls?
My advice is simply to remove the
rarmatch from the list. The remainder of the 5G is still going to be effective at blocking a bunch of other bad stuff, but your rar URLs will continue to work properly.Otherwise, to block
rarrequests only in other directories, some more precise pattern-matching is required, based on existing file names, and so on.I’ve removed the rar rule
The rest of the blocking protection is demanded to Better WP Protection and a custom ip blocking list obtained reading the access.log :D
Yup, I ran into the “//” issue like Anders Vinther, which was solved by removing the test.
Also, if you plan on sharing your website/pages, I noticed that Google+ doesn’t display the thumbnail correctly unless you comment out this line:
# SetEnvIfNoCase User-Agent ^$ keep_out # Google+ snippetAh, good catch with the Google+.. at one time blocking the empty user-agent string was a no-brainer, but these days lots of apps and social-media stuff use it. And now with Google+ in the mix, I’m thinking it’s best to remove that rule for the 6G. Thanks for the heads up.
No problem, and thank you for all the hard work. It took me weeks to figure out why Google+ wasn’t displaying the thumbnail.
The 5th line in the request strings section interferes with entering taxonomy terms when creating/editing pages in Drupal. I don’t know why, exactly, so I had to comment it out.
No other Drupal-related issues found yet, but I will gladly keep you posted, Jeff.
I was getting a 403 Forbidden error when clicking links that I had inserted in a newsletter. The error message: “You don’t have permission to access / on this server.”
Changing:
RewriteRule .* - [F]toRewriteRule .*!/ - [F]solved the problem.Hi again…
I just wrote a little blog post detailing the process of the halving method for problem determination…
It can be found here: http://www.wpsecuritychecklist.com/the-perishable-press-halving-method/
Cheers…
Awesome Anders, Thanks for writing and sharing :)
I’ve mentioned the article w/link at the original article:
https://perishablepress.com/the-halving-method-of-identifying-problematic-code/
Cheers!
Hi. you’ve done perfect job,I enjoy reading and implementing all the tricks with .htaccess. Currently I am using snipets from your firewall lists to make something different for me,
and I am wondering is it possible to redirect all user agents (snipet # 5G:[USER AGENTS]) to one specific page on my site, on that page will implement some codes and links from www projecthoneypot org . I figured out how to do it with IP’s and IP ranges but can’t find code for user agents, SetEnvIfNoCase User-Agent and similar. Can yo give me idea ?
Just wanted to let you know that
SetEnvIfNoCase User-Agent ^$ keep_outdoesn't work well with WP-PIWIK, as reported here
http://wordpress.org/support/topic/plugin-wp-piwik-not-working-with-5g-blacklist?replies=1
I’ve had to remove a few things to make the 5G Blacklist work with my multitude of WordPress sites:
RewriteCond %{QUERY_STRING} (localhost|mosconfig|scanner) [NC,OR] #removed environ#too many pieces of legitimate content with the word "environment" in the URLRewriteCond %{QUERY_STRING} (mod|path|tag)=.?/? [NC,OR] #removed menu#menu is needed by the WP appearance admin#I often must comment out this one:RewriteCond %{QUERY_STRING} : [NC,OR]#non-URL encoded argument URLs are common in the wild.#RedirectMatch 403 (,|//|)+|/,/|{0}|(/(|...|+++|||\"\")#http://wordpress.org/support/topic/plugin-wp-super-cache-double-forward-slashOtherwise, thank you for this wonderful tool!
Just wanted to post a quick thank you for creating this for the community, it’s helped us already and has certainly toughened up our WP sites.