5G Blacklist 2012

♦ Posted by Jeff Starr in .htaccess, Security

Updated July 19, 2024 • 223 comments

[ 5G (2012) ] The 5G Blacklist helps reduce the number of malicious URL requests that hit your website. It’s one of many ways to improve the security of your site and protect against evil exploits, bad requests, and other nefarious garbage. If you’re tired of all the slow, bloated security plugins and expensive 3rd-party services, the 5G Blacklist is a solid solution to help protect your Apache-powered site.

Update: Check out the new and improved 6G Firewall »

Evolution

After extensive beta testing, the 5G Blacklist/Firewall is solid and ready to help secure sites hosted on Apache servers. In addition to beta testing for the 5G, this is the 5th major update of my “G”-series blacklists. Here is a quick overview of its evolution.

Along the way, I’ve explored a wide variety of different blacklist techniques. The 5G is the culmination of all these efforts, and will eventually be replaced by the imminent 6G Blacklist/Firewall.

What it does

The 5G Blacklist is a simple, flexible blacklist that checks all URI requests against a series of carefully constructed HTAccess directives. This happens quietly behind the scenes at the server level, saving resources for stuff like PHP and MySQL for all blocked requests.

How it works

Blacklists can block just about any part of a request: IP, user agent, request string, query string, referrer, and everything in between. But IP addresses change constantly, and user agents and referrers are easily spoofed. As discussed, request strings yield the best results: greater protection with fewer false positives.

The 5G works beautifully with WordPress, and should help any site conserve bandwidth and server resources while protecting against malicious activity.

How to use

To install the 5G Firewall, append the following code to your site’s root .htaccess:

# 5G BLACKLIST/FIREWALL
# @ https://perishablepress.com/5g-blacklist-2012/

# 5G:[QUERY STRINGS]
<ifModule mod_rewrite.c>
 RewriteEngine On
 RewriteBase /
 RewriteCond %{QUERY_STRING} (environ|localhost|mosconfig|scanner) [NC,OR]
 RewriteCond %{QUERY_STRING} (menu|mod|path|tag)\=\.?/? [NC,OR]
 RewriteCond %{QUERY_STRING} boot\.ini  [NC,OR]
 RewriteCond %{QUERY_STRING} echo.*kae  [NC,OR]
 RewriteCond %{QUERY_STRING} etc/passwd [NC,OR]
 RewriteCond %{QUERY_STRING} \=\\%27$   [NC,OR]
 RewriteCond %{QUERY_STRING} \=\\\'$    [NC,OR]
 RewriteCond %{QUERY_STRING} \.\./      [NC,OR]
 RewriteCond %{QUERY_STRING} \?         [NC,OR]
 RewriteCond %{QUERY_STRING} \:         [NC,OR]
 RewriteCond %{QUERY_STRING} \[         [NC,OR]
 RewriteCond %{QUERY_STRING} \]         [NC]
 RewriteRule .* - [F]
</ifModule>

# 5G:[USER AGENTS]
<ifModule mod_setenvif.c>
 SetEnvIfNoCase User-Agent ^$ keep_out
 SetEnvIfNoCase User-Agent (casper|cmsworldmap|diavol|dotbot)   keep_out
 SetEnvIfNoCase User-Agent (flicky|ia_archiver|jakarta|kmccrew) keep_out
 SetEnvIfNoCase User-Agent (libwww|planetwork|pycurl|skygrid)   keep_out
 SetEnvIfNoCase User-Agent (purebot|comodo|feedfinder|turnit)   keep_out
 SetEnvIfNoCase User-Agent (zmeu|nutch|vikspider|binlar|sucker) keep_out
 <limit GET POST PUT>
  Order Allow,Deny
  Allow from all
  Deny from env=keep_out
 </limit>
</ifModule>

# 5G:[REQUEST STRINGS]
<ifModule mod_alias.c>
 RedirectMatch 403 (https?|ftp|php)\://
 RedirectMatch 403 /(cgi|https?|ima|ucp)/
 RedirectMatch 403 /(Permanent|Better)$
 RedirectMatch 403 (\=\\\'|\=\\%27|/\\\'/?|\)\.css\()$
 RedirectMatch 403 (\,|//|\)\+|/\,/|\{0\}|\(/\(|\.\.\.|\+\+\+|\||\\\"\\\")
 RedirectMatch 403 \.(cgi|asp|aspx|cfg|dll|exe|jsp|mdb|sql|ini|rar)$
 RedirectMatch 403 /(contac|fpw|install|pingserver|register)\.php$
 RedirectMatch 403 (base64|crossdomain|localhost|wwwroot|e107\_)
 RedirectMatch 403 (eval\(|\_vti\_|\(null\)|echo.*kae|config\.xml)
 RedirectMatch 403 \.well\-known/host\-meta
 RedirectMatch 403 /function\.array\-rand
 RedirectMatch 403 \)\;\$\(this\)\.html\(
 RedirectMatch 403 proc/self/environ
 RedirectMatch 403 msnbot\.htm\)\.\_
 RedirectMatch 403 /ref\.outcontrol
 RedirectMatch 403 com\_cropimage
 RedirectMatch 403 indonesia\.htm
 RedirectMatch 403 \{\$itemURL\}
 RedirectMatch 403 function\(\)
 RedirectMatch 403 labels\.rdf
 RedirectMatch 403 /playing.php
 RedirectMatch 403 muieblackcat
</ifModule>

# 5G:[BAD IPS]
<limit GET POST PUT>
 Order Allow,Deny
 Allow from all
 # uncomment/edit/repeat next line to block IPs
 # Deny from 123.456.789
</limit>

That’s the golden ticket right there. The 5G Firewall is serious protection for your website: extensively tested, plug-n-play, and completely free. “Grab, gulp, n go” as they say. For more information, see the beta article (and comments).

Troubleshooting

Remember, test thoroughly. If something stops working when the 5G is installed, try removing the 5G. If things start working normally again, you can either pass on the 5G or investigate further. Investigating further is straightforward using something like the halving method, where you remove chunks of the 5G until isolating and identifying the issue. Here is a quick example:

I’ve installed the 5G, thanks Jeff.
Uh-oh, the page at http://example.com/indonesia.html stopped loading
Hmm, the URL contains the phrase “indonesia”, so let’s check the 5G for it
Yep, there’s a rule that blocks indonesia\.htm
Removing that line resolves the issue, thanks me.

Is it okay to remove rules that are blocking your own pages? Yes, the only downside is that malicious requests that would have otherwise been blocked will now get through. The 5G will continue to block a massive volume of malicious requests — it’ll just be a bit less effective. The protective effect is cumulative, not dependent on any one rule. So customization is encouraged. Once you dial it in, you’re all set.

Disclaimer

The 5G Firewall is provided “as-is”, with the intention of helping site administrators protect their sites against bad requests and other malicious activity. The code is open and free to use and modify as long as the first two credit lines remain intact. By using this code you assume all risk & responsibility for anything that happens, whether good or bad. In short, use wisely, test thoroughly, don’t sue me.

Learn more..

To learn more about the theory and development of the 5G Firewall, check out my articles on building the 3G, 4G and 5G Blacklist. A search for “blacklist” in the sidebar should also yield many results.

Happy securing!

About the Author

Jeff Starr = Web Developer. Book Author. Secretly Important.

223 responses to “5G Blacklist 2012”

Xiaoyu 2012/11/16 12:46 pm

5G on my site block Google+1 button and throw me a google +1 button error. I delete the 5G on htaccess and then the Google+1 work. When I enable the 5G firewall, the Google+1 keep working on the page I already +1. But when I create a new +1 button on an different page when 5G still enable, same error again. Do anyone know a known problem in 5G that cause the Google +1 button error?

Jeff Starr 2012/11/19 3:00 am • Post Author

Hi Xiaoyu, can you post some example URLs that are getting blocked or not working? If so, please wrap each URL with <code> tags. Thanks!

Xiaoyu 2012/11/16 1:02 pm

Further investigation, it is this line in 5G block Google+1 (spent good 2 hours to test can comment out rules):

#RedirectMatch 403 (,|//|)+|/,/|{0}|(/(|...|+++|||\"\")

For now, I comment it out. But I wonder if this will cost a big security hole for my site. Any idea what I missing if I comment this line out? Any work around for letting Google+1 to work and still keep this line for tight security? Thanks you!

Jeff Starr 2012/11/19 2:59 am • Post Author

Hi Xiaoyu, thanks for the feedback and for narrowing it down to that specific rule, which itself blocks each of the strings that are separated by a vertical bar |. More than likely, it’s only one of these patterns that’s blocking the URL. So, if you can post the URL(s) here (plz wrap each with code tags), we can narrow it down even further.
- Xiaoyu 2012/11/19 1:24 pm
  
  It is a Google+1 button. I have no idea how I can get the URL. I embed the standard Google+1 code. And then click the button, it has a JavaScript type of window popup with sharing info from Google. Then the window went away with the Google +1 button become a warning button. I click it again, it say Google +1 button error. I assume this is a common problem right? Since I used the standard Google+1 embed code and everything.

Ken 2012/11/20 4:48 pm

Hi Jeff,
I came across something I’ve not seen before re using Here’s the url
http://cd34.com/blog/web-security/hackers-bypass-htaccess-security-by-using-gets-rather-than-get/

Is this someting to address in 5G/6G?

Thanks for all your hard work!

Jeff Starr 2012/11/21 1:33 am • Post Author

Hi Ken, that’s very interesting.. I was unable to replicate similar GETS results on a similar setup, so honestly no idea if it’s still an issue or not. The post is two years old, doesn’t contain any references, and doesn’t mention the version of Apache, Zend, or anything else, so difficult to investigate. If you discover any related info plz post (I searched, but the apparently “get” is the same as “gets” to google).

Kristofer 2012/11/28 12:20 pm

Will this g5 list block wget and curl bad bot?

Best regards
Kristofer

Jeff Starr 2012/12/01 3:53 pm • Post Author

Nope. But they’re easy to block if needed, just add this line to the 5G User Agent rules:

SetEnvIfNoCase User-Agent (curl|wget) keep_out

frances 2012/12/06 6:22 am

Hi Jeff,

these two lines seem to be a problem with WordPress 3.5 RC-4:

RewriteCond %{QUERY_STRING} \[ [NC,OR]
RewriteCond %{QUERY_STRING} \] [NC]

They cause load-scripts.php (in wp-admin) to throw a 404.

I’ve only briefly tested this, and it could be caused by a combination of other things, but commenting these two lines out fixes the problem. Possibly this will carry over to wp3.5.

Frances

Jeff Starr 2012/12/06 12:32 pm • Post Author

Hi frances, thanks for reporting this.. do you happen to know the URLs that are blocked because of these rules? Thanks.
- frances 2012/12/08 9:28 am
  
  Hi Jeff, I’ve tested this a bit more. My version of the 5G was slightly different from the current one above, which doesn’t cause the problem, so call this solved.
  
  Frances
- Jeff Starr 2012/12/08 2:06 pm • Post Author
  
  Thanks for the update, frances!

RJ 2012/12/11 10:18 am

Thank you for this. If anyone is interested, I found two lines that break applications that I use.

This line breaks Gallery Scraper:
RewriteCond %{QUERY_STRING} (menu|mod|path|tag)\=\.?/? [NC,OR]

This line breaks OpenX:
RewriteCond %{QUERY_STRING} \: [NC,OR]

Right now they are simply commented out since I don’t know what they’re supposed to do.

ale 2012/12/12 11:14 am

Hi!

Besides the “menu” removal from the query strings to be able to edit multiple wp menus, after the 3.5 update is necessary to remove these lines as well:

RewriteCond %{QUERY_STRING} [ [NC,OR]
RewriteCond %{QUERY_STRING} ] [NC]

I see this was reported before but I want to add that my 5g version is not modified. The url causing the conflict is this one:

http://www.example.com/wp-admin/load-scripts.php?c=1&load[]=swfobject,jquery,utils&ver=3.5

This is anywhere on the wp-admin section.

Hope it hels!

Thanks for sharing this great work!

Jeff Starr 2012/12/12 1:33 pm • Post Author

Thanks for the heads up on this ale. Removing the square brackets from 5G/6G is only a workaround. The issue itself is with WP’s use of unencoded/unsafe characters (ie, square brackets) in URLs. For more info see: http://www.blooberry.com/indexdot/html/topics/urlencoding.htm
MickeyRoush 2012/12/13 3:01 am

Instead of removing the query strings, you should consider whitelisting (using !) whatever is requiring their usage. At least that way you’ll have some protection against malicious strings. Either whitelist wp-admin (broad), or the file load-scripts.php (narrower), or by IP (most narrow) of whoever will need access to that url.
- Jeff Starr 2012/12/13 12:24 pm • Post Author
  
  Excellent suggestion. Probably will apply this strategy for the 6G for dealing with unsafe characters in WP.

Micha 2012/12/12 3:19 pm

Removing those two lines broke even more – in fact I lost access to the WP admin after removing them (403 Forbidden)

To “fix” it, I had to remove RewriteRule .* - [F] as well.

Jeff Starr 2012/12/12 3:25 pm • Post Author

Actually removing the RewriteRule disables the entire block of code. Instead of doing that, after removing (or commenting out) the two bracket directives, replace this line:

RewriteCond %{QUERY_STRING} \: [NC, OR]

..with this:

RewriteCond %{QUERY_STRING} \: [NC]

..and you should be good to go.

Micha 2012/12/12 3:34 pm

Ah – I see (well, I don’t really “see”, as you can tell, but it works)

Thanks

Mike 2012/12/13 1:06 pm

Hi Jeff,
Are you going to update 5G for us easily confused users :) with this new information and recommended changes?
Happy trails, Mike

Jeff Starr 2012/12/31 6:33 pm • Post Author

Yes, here it is: Stop Using Unsafe Characters in URLs , which includes the fix for 5G near the end of the post. Cheers :)

hucste 2012/12/14 3:58 am

Hello,

In case, you use 5G blacklist with Munin, those block when you consult an url as :
http://server.kimsufi.com/munin/localdomain/localhost.localdomain/index.html#disk

I’ve modified this line on [Request String]:

AndyL 2012/12/16 5:53 am

On my site, the two rules for square brackets caused the Widgets page to stop working and the buttons to disappear in New Posts page. Removing them and the preceeding OR fixed the problem.

I suggest you add a comment on this as a top post as a lot of people are hitting this issue, and at least some of them will have followed advice here.

Some further info which might help track down what is happening:
I have five plugins: Askimet, BackWPUp, Better WP Security, Fast Secure Contact Form and Subscribe2 HTML. Adding the following to wp-config.php provided a temporary fix until I found the cause.

define('CONCATENATE_SCRIPTS',false);

Jeff Starr 2012/12/17 11:53 am • Post Author

I will be posting about this soon, thanks for the feedback.
Matt McDowell 2012/12/17 11:58 am

I also had an issue with the 5G Blacklist disabling portions of the WordPress 3.5 admin area, specifically dropdowns and the post editor.

This issue is mentioned here.

For me, pasting in the modified code from that page returned a 500 error; to get it to work, I had to manually delete these two lines:

RewriteCond %{QUERY_STRING} [ [NC,OR]
RewriteCond %{QUERY_STRING} ] [NC]

and change

RewriteCond %{QUERY_STRING} : [NC,OR]

to

RewriteCond %{QUERY_STRING} : [NC]

Comments are closed for this post. Something to add? Let me know.

Xiaoyu 2012/11/16 12:46 pm

5G on my site block Google+1 button and throw me a google +1 button error. I delete the 5G on htaccess and then the Google+1 work. When I enable the 5G firewall, the Google+1 keep working on the page I already +1. But when I create a new +1 button on an different page when 5G still enable, same error again. Do anyone know a known problem in 5G that cause the Google +1 button error?
- Jeff Starr 2012/11/19 3:00 am • Post Author
  
  Hi Xiaoyu, can you post some example URLs that are getting blocked or not working? If so, please wrap each URL with <code> tags. Thanks!
Xiaoyu 2012/11/16 1:02 pm

Further investigation, it is this line in 5G block Google+1 (spent good 2 hours to test can comment out rules):

#RedirectMatch 403 (,|//|)+|/,/|{0}|(/(|...|+++|||\"\")

For now, I comment it out. But I wonder if this will cost a big security hole for my site. Any idea what I missing if I comment this line out? Any work around for letting Google+1 to work and still keep this line for tight security? Thanks you!
- Jeff Starr 2012/11/19 2:59 am • Post Author
  
  Hi Xiaoyu, thanks for the feedback and for narrowing it down to that specific rule, which itself blocks each of the strings that are separated by a vertical bar |. More than likely, it’s only one of these patterns that’s blocking the URL. So, if you can post the URL(s) here (plz wrap each with code tags), we can narrow it down even further.
  - Xiaoyu 2012/11/19 1:24 pm
    
    It is a Google+1 button. I have no idea how I can get the URL. I embed the standard Google+1 code. And then click the button, it has a JavaScript type of window popup with sharing info from Google. Then the window went away with the Google +1 button become a warning button. I click it again, it say Google +1 button error. I assume this is a common problem right? Since I used the standard Google+1 embed code and everything.
Ken 2012/11/20 4:48 pm

Hi Jeff,
I came across something I’ve not seen before re using Here’s the url
http://cd34.com/blog/web-security/hackers-bypass-htaccess-security-by-using-gets-rather-than-get/

Is this someting to address in 5G/6G?

Thanks for all your hard work!
- Jeff Starr 2012/11/21 1:33 am • Post Author
  
  Hi Ken, that’s very interesting.. I was unable to replicate similar GETS results on a similar setup, so honestly no idea if it’s still an issue or not. The post is two years old, doesn’t contain any references, and doesn’t mention the version of Apache, Zend, or anything else, so difficult to investigate. If you discover any related info plz post (I searched, but the apparently “get” is the same as “gets” to google).
Kristofer 2012/11/28 12:20 pm

Hi

Will this g5 list block wget and curl bad bot?

Best regards
Kristofer
- Jeff Starr 2012/12/01 3:53 pm • Post Author
  
  Nope. But they’re easy to block if needed, just add this line to the 5G User Agent rules:
  
  SetEnvIfNoCase User-Agent (curl|wget) keep_out
frances 2012/12/06 6:22 am

Hi Jeff,

these two lines seem to be a problem with WordPress 3.5 RC-4:

RewriteCond %{QUERY_STRING} \[ [NC,OR]
RewriteCond %{QUERY_STRING} \] [NC]

They cause load-scripts.php (in wp-admin) to throw a 404.

I’ve only briefly tested this, and it could be caused by a combination of other things, but commenting these two lines out fixes the problem. Possibly this will carry over to wp3.5.

Frances
- Jeff Starr 2012/12/06 12:32 pm • Post Author
  
  Hi frances, thanks for reporting this.. do you happen to know the URLs that are blocked because of these rules? Thanks.
  - frances 2012/12/08 9:28 am
    
    Hi Jeff, I’ve tested this a bit more. My version of the 5G was slightly different from the current one above, which doesn’t cause the problem, so call this solved.
    
    Frances
  - Jeff Starr 2012/12/08 2:06 pm • Post Author
    
    Thanks for the update, frances!
RJ 2012/12/11 10:18 am

Thank you for this. If anyone is interested, I found two lines that break applications that I use.

This line breaks Gallery Scraper:
RewriteCond %{QUERY_STRING} (menu|mod|path|tag)\=\.?/? [NC,OR]

This line breaks OpenX:
RewriteCond %{QUERY_STRING} \: [NC,OR]

Right now they are simply commented out since I don’t know what they’re supposed to do.
ale 2012/12/12 11:14 am

Hi!

Besides the “menu” removal from the query strings to be able to edit multiple wp menus, after the 3.5 update is necessary to remove these lines as well:

RewriteCond %{QUERY_STRING} [ [NC,OR]
RewriteCond %{QUERY_STRING} ] [NC]

I see this was reported before but I want to add that my 5g version is not modified. The url causing the conflict is this one:

http://www.example.com/wp-admin/load-scripts.php?c=1&load[]=swfobject,jquery,utils&ver=3.5

This is anywhere on the wp-admin section.

Hope it hels!

Thanks for sharing this great work!
- Jeff Starr 2012/12/12 1:33 pm • Post Author
  
  Thanks for the heads up on this ale. Removing the square brackets from 5G/6G is only a workaround. The issue itself is with WP’s use of unencoded/unsafe characters (ie, square brackets) in URLs. For more info see: http://www.blooberry.com/indexdot/html/topics/urlencoding.htm
- MickeyRoush 2012/12/13 3:01 am
  
  Instead of removing the query strings, you should consider whitelisting (using !) whatever is requiring their usage. At least that way you’ll have some protection against malicious strings. Either whitelist wp-admin (broad), or the file load-scripts.php (narrower), or by IP (most narrow) of whoever will need access to that url.
  - Jeff Starr 2012/12/13 12:24 pm • Post Author
    
    Excellent suggestion. Probably will apply this strategy for the 6G for dealing with unsafe characters in WP.
Micha 2012/12/12 3:19 pm

Removing those two lines broke even more – in fact I lost access to the WP admin after removing them (403 Forbidden)

To “fix” it, I had to remove RewriteRule .* - [F] as well.
- Jeff Starr 2012/12/12 3:25 pm • Post Author
  
  Actually removing the RewriteRule disables the entire block of code. Instead of doing that, after removing (or commenting out) the two bracket directives, replace this line:
  
  RewriteCond %{QUERY_STRING} \: [NC, OR]
  
  ..with this:
  
  RewriteCond %{QUERY_STRING} \: [NC]
  
  ..and you should be good to go.
Micha 2012/12/12 3:34 pm

Ah – I see (well, I don’t really “see”, as you can tell, but it works)

Thanks
Mike 2012/12/13 1:06 pm

Hi Jeff,
Are you going to update 5G for us easily confused users :) with this new information and recommended changes?
Happy trails, Mike
- Jeff Starr 2012/12/31 6:33 pm • Post Author
  
  Yes, here it is: Stop Using Unsafe Characters in URLs , which includes the fix for 5G near the end of the post. Cheers :)
hucste 2012/12/14 3:58 am

Hello,

In case, you use 5G blacklist with Munin, those block when you consult an url as :
http://server.kimsufi.com/munin/localdomain/localhost.localdomain/index.html#disk

I’ve modified this line on [Request String]:

# RedirectMatch 403 (base64|crossdomain|localhost|wwwroot|e107\_) # modified for munin
RedirectMatch 403 (base64|crossdomain|wwwroot|e107\_)
AndyL 2012/12/16 5:53 am

On my site, the two rules for square brackets caused the Widgets page to stop working and the buttons to disappear in New Posts page. Removing them and the preceeding OR fixed the problem.

I suggest you add a comment on this as a top post as a lot of people are hitting this issue, and at least some of them will have followed advice here.

Some further info which might help track down what is happening:
I have five plugins: Askimet, BackWPUp, Better WP Security, Fast Secure Contact Form and Subscribe2 HTML. Adding the following to wp-config.php provided a temporary fix until I found the cause.

define('CONCATENATE_SCRIPTS',false);
- Jeff Starr 2012/12/17 11:53 am • Post Author
  
  I will be posting about this soon, thanks for the feedback.
- Matt McDowell 2012/12/17 11:58 am
  
  I also had an issue with the 5G Blacklist disabling portions of the WordPress 3.5 admin area, specifically dropdowns and the post editor.
  
  This issue is mentioned here.
  
  For me, pasting in the modified code from that page returned a 500 error; to get it to work, I had to manually delete these two lines:
  
  RewriteCond %{QUERY_STRING} [ [NC,OR]
  RewriteCond %{QUERY_STRING} ] [NC]
  
  and change
  
  RewriteCond %{QUERY_STRING} : [NC,OR]
  
  to
  
  RewriteCond %{QUERY_STRING} : [NC]

« Previous Comments • 1…78910 • Newer Comments »