5G Blacklist 2012

♦ Posted by Jeff Starr in .htaccess, Security

Updated June 6, 2021 • 223 comments

[ 5G (2012) ] The 5G Blacklist helps reduce the number of malicious URL requests that hit your website. It’s one of many ways to improve the security of your site and protect against evil exploits, bad requests, and other nefarious garbage. If you’re tired of all the slow, bloated security plugins and expensive 3rd-party services, the 5G Blacklist is a solid solution to help protect your Apache-powered site.

Update: Check out the new and improved 6G Firewall »

Evolution

After extensive beta testing, the 5G Blacklist/Firewall is solid and ready to help secure sites hosted on Apache servers. In addition to beta testing for the 5G, this is the 5th major update of my “G”-series blacklists. Here is a quick overview of its evolution.

Along the way, I’ve explored a wide variety of different blacklist techniques. The 5G is the culmination of all these efforts, and will eventually be replaced by the imminent 6G Blacklist/Firewall.

What it does

The 5G Blacklist is a simple, flexible blacklist that checks all URI requests against a series of carefully constructed HTAccess directives. This happens quietly behind the scenes at the server level, saving resources for stuff like PHP and MySQL for all blocked requests.

How it works

Blacklists can block just about any part of a request: IP, user agent, request string, query string, referrer, and everything in between. But IP addresses change constantly, and user agents and referrers are easily spoofed. As discussed, request strings yield the best results: greater protection with fewer false positives.

The 5G works beautifully with WordPress, and should help any site conserve bandwidth and server resources while protecting against malicious activity.

How to use

To install the 5G Firewall, append the following code to your site’s root .htaccess:

# 5G BLACKLIST/FIREWALL
# @ https://perishablepress.com/5g-blacklist-2012/

# 5G:[QUERY STRINGS]
<ifModule mod_rewrite.c>
 RewriteEngine On
 RewriteBase /
 RewriteCond %{QUERY_STRING} (environ|localhost|mosconfig|scanner) [NC,OR]
 RewriteCond %{QUERY_STRING} (menu|mod|path|tag)\=\.?/? [NC,OR]
 RewriteCond %{QUERY_STRING} boot\.ini  [NC,OR]
 RewriteCond %{QUERY_STRING} echo.*kae  [NC,OR]
 RewriteCond %{QUERY_STRING} etc/passwd [NC,OR]
 RewriteCond %{QUERY_STRING} \=\\%27$   [NC,OR]
 RewriteCond %{QUERY_STRING} \=\\\'$    [NC,OR]
 RewriteCond %{QUERY_STRING} \.\./      [NC,OR]
 RewriteCond %{QUERY_STRING} \?         [NC,OR]
 RewriteCond %{QUERY_STRING} \:         [NC,OR]
 RewriteCond %{QUERY_STRING} \[         [NC,OR]
 RewriteCond %{QUERY_STRING} \]         [NC]
 RewriteRule .* - [F]
</ifModule>

# 5G:[USER AGENTS]
<ifModule mod_setenvif.c>
 SetEnvIfNoCase User-Agent ^$ keep_out
 SetEnvIfNoCase User-Agent (casper|cmsworldmap|diavol|dotbot)   keep_out
 SetEnvIfNoCase User-Agent (flicky|ia_archiver|jakarta|kmccrew) keep_out
 SetEnvIfNoCase User-Agent (libwww|planetwork|pycurl|skygrid)   keep_out
 SetEnvIfNoCase User-Agent (purebot|comodo|feedfinder|turnit)   keep_out
 SetEnvIfNoCase User-Agent (zmeu|nutch|vikspider|binlar|sucker) keep_out
 <limit GET POST PUT>
  Order Allow,Deny
  Allow from all
  Deny from env=keep_out
 </limit>
</ifModule>

# 5G:[REQUEST STRINGS]
<ifModule mod_alias.c>
 RedirectMatch 403 (https?|ftp|php)\://
 RedirectMatch 403 /(cgi|https?|ima|ucp)/
 RedirectMatch 403 /(Permanent|Better)$
 RedirectMatch 403 (\=\\\'|\=\\%27|/\\\'/?|\)\.css\()$
 RedirectMatch 403 (\,|//|\)\+|/\,/|\{0\}|\(/\(|\.\.\.|\+\+\+|\||\\\"\\\")
 RedirectMatch 403 \.(cgi|asp|aspx|cfg|dll|exe|jsp|mdb|sql|ini|rar)$
 RedirectMatch 403 /(contac|fpw|install|pingserver|register)\.php$
 RedirectMatch 403 (base64|crossdomain|localhost|wwwroot|e107\_)
 RedirectMatch 403 (eval\(|\_vti\_|\(null\)|echo.*kae|config\.xml)
 RedirectMatch 403 \.well\-known/host\-meta
 RedirectMatch 403 /function\.array\-rand
 RedirectMatch 403 \)\;\$\(this\)\.html\(
 RedirectMatch 403 proc/self/environ
 RedirectMatch 403 msnbot\.htm\)\.\_
 RedirectMatch 403 /ref\.outcontrol
 RedirectMatch 403 com\_cropimage
 RedirectMatch 403 indonesia\.htm
 RedirectMatch 403 \{\$itemURL\}
 RedirectMatch 403 function\(\)
 RedirectMatch 403 labels\.rdf
 RedirectMatch 403 /playing.php
 RedirectMatch 403 muieblackcat
</ifModule>

# 5G:[BAD IPS]
<limit GET POST PUT>
 Order Allow,Deny
 Allow from all
 # uncomment/edit/repeat next line to block IPs
 # Deny from 123.456.789
</limit>

That’s the golden ticket right there. The 5G Firewall is serious protection for your website: extensively tested, plug-n-play, and completely free. “Grab, gulp, n go” as they say. For more information, see the beta article (and comments).

Troubleshooting

Remember, test thoroughly. If something stops working when the 5G is installed, try removing the 5G. If things start working normally again, you can either pass on the 5G or investigate further. Investigating further is straightforward using something like the halving method, where you remove chunks of the 5G until isolating and identifying the issue. Here is a quick example:

I’ve installed the 5G, thanks Jeff.
Uh-oh, the page at http://example.com/indonesia.html stopped loading
Hmm, the URL contains the phrase “indonesia”, so let’s check the 5G for it
Yep, there’s a rule that blocks indonesia\.htm
Removing that line resolves the issue, thanks me.

Is it okay to remove rules that are blocking your own pages? Yes, the only downside is that malicious requests that would have otherwise been blocked will now get through. The 5G will continue to block a massive volume of malicious requests — it’ll just be a bit less effective. The protective effect is cumulative, not dependent on any one rule. So customization is encouraged. Once you dial it in, you’re all set.

Disclaimer

The 5G Firewall is provided “as-is”, with the intention of helping site administrators protect their sites against bad requests and other malicious activity. The code is open and free to use and modify as long as the first two credit lines remain intact. By using this code you assume all risk & responsibility for anything that happens, whether good or bad. In short, use wisely, test thoroughly, don’t sue me.

Learn more..

To learn more about the theory and development of the 5G Firewall, check out my articles on building the 3G, 4G and 5G Blacklist. A search for “blacklist” in the sidebar should also yield many results.

Happy securing!

About the Author

Jeff Starr = Designer. Developer. Producer. Writer. Editor. Etc.

223 responses to “5G Blacklist 2012”

Sameer Manas 2012/08/25 9:13 am

Is there a way to use this in Nginx ?
I am a fan of Nginx and i would like to utilise these rules for Nginx.

Jeremy Clarke 2012/09/13 8:25 am

Hi Jeff, I’ve been using the 5G blacklist and loving it. I think my biggest problems weren’t necessarily solved by your lines, but just having the list as a template for adding my own UAs, query strings and IPs has given me a toolkit for solving my problems.

That said I think you need to reconsider this line:

RewriteCond %{QUERY_STRING} ? [NC,OR]

As brasofilo and Christian Castellari pointed out in earlier comments, it causes issues with WP’s preview feature. Note that it ONLY affects the third and subsequent previews, not the first two, so it can be easy to miss.

The issue is that after a few previews WP starts generating URLs like this when you click the button:

http://test.site/wp-admin/post.php?t=1347548645469?t=1347548651124?t=1347548656685?t=1347548662469?t=1347548672300?t=1347548681615?

Usually the author would never see that URL because it gets redirected to the post permalink with ?redirect attached, but when that line of 5G is enabled it kills the redirect and the user sees a 403 instead. To make things worse, the issue also seems to affect pushing the “Update” button once you’ve previewed a few times causes a 403 as well, which can result in data loss as well as a LOT of user stress.

Obviously this is mostly WPs fault, they shouldn’t be creating those invalid URLs with so many ?t= values. Unfortunately while WP is doing that the 5G blacklist is probably better off maintaining compatibility and not causing this rare but aggravating problem.

Thanks for considering it. For now I’ll just remove the line from my own setup.

Alain Lesage 2012/09/13 10:10 am

Hi. May I asked what happened to the WordPress plugin, Block Bad Queries, that I understand you used to maintain?

Jeff Starr 2012/09/13 11:44 am • Post Author

Currently working on an update that should be available within the next couple of weeks.
- Danielle Signor 2012/09/13 3:17 pm
  
  Will this update work with the 5G blacklist, or replace it, in regards to WordPress?
- Jeff Starr 2012/09/14 11:11 am • Post Author
  
  It will work with the 5G/6G, but will block some of the same stuff.. it’s meant for those without access to .htaccess..
- Danielle Signor 2012/09/14 11:36 am
  
  Thanks Jeff, that answers the larger question I had. I use the Block Bad Queries plugin for all of my WordPress clients, but if the 5G/6G blacklist (plus WP addon) does the same thing and more, I’ll use that in future.

Randy 2012/09/17 7:01 am

I implemented 5G on a WP install with a totally custom theme: shamanscircle.com and users have not bee able to log in. My /wp-admin & /wp-login &/login redirect in a loop and had to remove the code from .htaccess.

I did see the removal of “menu” so it is like this:

RewriteCond %{QUERY_STRING} (mod|path|tag)=.?/? [NC,OR]

But it still didn’t work. Any ideas on what to do to fix this so i can continue to use 5G?

Cheers.

Amethyst Igor 2012/10/01 11:19 am

Hi Jeff, I use the 5G Blacklist, for which we are all thankful, I assure you. I have used your site for some time now. However I did detect one problem in this line:

RewriteCond %{QUERY_STRING} (menu|mod|path|tag)=.?/? [NC,OR]

This requires modification for WordPress compatibility. In my server log, I observed the following:

66.249.74.83 - - [30/Sep/2012:16:55:25 -0600] "GET /?tag=mooncakes HTTP/1.1" 403 687 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The IP address above is owned by Google, which we do not wish to serve 403’s. One solution is to remove “tag” from the query string, above, in .htaccess. I may make other mods as needed, but this is one thing I noticed right away in digging through my server log for 403’s.

Have a good day and please continue your work.

Jeff Starr 2012/10/01 12:02 pm • Post Author

Good catch — will make sure that tag is removed from the in-progress 6G Blacklist. Thanks for the infos!
Tim 2012/10/01 6:08 pm

Amethyst, Thanks for posting. If you or another moderator could share a little info on how to dig through server logs for 403’s. Is it just “grep ‘ 403 ‘ access.log? or are there other methods.

Bill 2012/10/03 8:21 am

Hi Jeff, just found a reference to your site from one of the joomla / virtuemart forums.

Just testing it out and it looks like I have a problem with this line:

RedirectMatch 403 (,|//|)+|/,/|{0}|(/(|...|+++|||\"\")

When trying to log in. Joomla 1.5.26 vm1.9

Gizmoscoop 2012/10/10 7:49 pm

Is the Ultimate htaccess Blacklist obsolete with this 2012 5G Blacklist?

Jeff Starr 2012/10/11 11:34 am • Post Author

Yes :)
- Gizmoscoop 2012/10/11 9:22 pm
  
  Hehe, thanks. Just deleted it, and my .htacess file looks shorter now.

Mike 2012/10/27 1:27 pm

Hi,
I have tried to implement this on my new wordpress install. When I do I get internal server error 500.

Where would I start to troubleshoot this?
Happy trails, Mike

Jeff Starr 2012/10/27 2:56 pm • Post Author

Check the access and error logs for your server, or ask your host where to find them. Server logs will give more information as to what caused the error.
- Mike 2012/10/27 3:12 pm
  
  Thanks Jeff. The error logs reported a missing closing tag ‘>’. My fault. Hey this htaccess doesnt block google bot does it?
  
  Thank you sir.
  Mike
- Jeff Starr 2012/10/27 3:33 pm • Post Author
  
  No, the 5G is designed to only block bad requests and allow everyone else, especially the search engines. Even so, I’m retesting the 5G with Google Webmaster Tools just to be safe. It’s been about a week and googlebot has had zero issues accessing any legit URLs.

Ron 2012/10/27 4:09 pm

Hi Jeff just letting you know that the code was giving me some problems with WordPress 3.4.2 when creating a custom menu I kept getting a 404 page not found error from this URL – /wp-admin/nav-menus.php?action=edit&menu=0

Really love the 5G would like a fix for it if possible

Dragan 2012/11/01 4:09 pm

Hi,

I’m using 5G firewall on a LiteSpeed server and I can’t seem to stop one rogue bot.

This is what he did last month, data taken from awstats
Unknown robot (identified by ‘bot*’)
Hits: 252,300+280
Bandwidth: 3.07 GB

But it is getting worse, because just today he ate 166.02 MB and had 16,120+6 hits, and this was only until 12:00pm.

My limit is 30 GB/month, so this is really too much. Can you please help me identify this bot and set up a htaccess rule to stop it?

Thank you in advance

Sumon farazi 2012/11/11 4:00 am

Hello sir. thank you for see me. I am a junior web developer. When i don’t understand anything, first visit this site and try to solved my problem by your post. However, i understand all everything about 5g except the last line ((( # Deny from 123.456.789 ))). Actually, i wanted to know can i replace my own IP when i use this 5g for my site or put this ((( 123.456.789 ))). please help me for choose the right IP when i using 5g for my site security. Thanks again.

Jeff Starr 2012/11/12 2:55 pm • Post Author

Hi Sumon, unless you have specific reason to do so, I would just omit the last (IP) section entirely. It’s mostly for blocking specific threats and is not needed for general security. If you do have an IP that you want to block, then add it to that section of the blacklist.

Sumon farazi 2012/11/14 2:58 am

Hello sir. thank you for reply me with a good solution. Now solved my problem. Actually, you r my teacher (WordPress security). I see your WordPress security tutorial from Lynda.com and learn how to secure a WordPress website. I want to be a good WordPress developer. Thank you very much.

Comments are closed for this post. Something to add? Let me know.

Sameer Manas 2012/08/25 9:13 am

Is there a way to use this in Nginx ?
I am a fan of Nginx and i would like to utilise these rules for Nginx.
Jeremy Clarke 2012/09/13 8:25 am

Hi Jeff, I’ve been using the 5G blacklist and loving it. I think my biggest problems weren’t necessarily solved by your lines, but just having the list as a template for adding my own UAs, query strings and IPs has given me a toolkit for solving my problems.

That said I think you need to reconsider this line:

RewriteCond %{QUERY_STRING} ? [NC,OR]

As brasofilo and Christian Castellari pointed out in earlier comments, it causes issues with WP’s preview feature. Note that it ONLY affects the third and subsequent previews, not the first two, so it can be easy to miss.

The issue is that after a few previews WP starts generating URLs like this when you click the button:

http://test.site/wp-admin/post.php?t=1347548645469?t=1347548651124?t=1347548656685?t=1347548662469?t=1347548672300?t=1347548681615?

Usually the author would never see that URL because it gets redirected to the post permalink with ?redirect attached, but when that line of 5G is enabled it kills the redirect and the user sees a 403 instead. To make things worse, the issue also seems to affect pushing the “Update” button once you’ve previewed a few times causes a 403 as well, which can result in data loss as well as a LOT of user stress.

Obviously this is mostly WPs fault, they shouldn’t be creating those invalid URLs with so many ?t= values. Unfortunately while WP is doing that the 5G blacklist is probably better off maintaining compatibility and not causing this rare but aggravating problem.

Thanks for considering it. For now I’ll just remove the line from my own setup.
Alain Lesage 2012/09/13 10:10 am

Hi. May I asked what happened to the WordPress plugin, Block Bad Queries, that I understand you used to maintain?
- Jeff Starr 2012/09/13 11:44 am • Post Author
  
  Currently working on an update that should be available within the next couple of weeks.
  - Danielle Signor 2012/09/13 3:17 pm
    
    Will this update work with the 5G blacklist, or replace it, in regards to WordPress?
  - Jeff Starr 2012/09/14 11:11 am • Post Author
    
    It will work with the 5G/6G, but will block some of the same stuff.. it’s meant for those without access to .htaccess..
  - Danielle Signor 2012/09/14 11:36 am
    
    Thanks Jeff, that answers the larger question I had. I use the Block Bad Queries plugin for all of my WordPress clients, but if the 5G/6G blacklist (plus WP addon) does the same thing and more, I’ll use that in future.
Randy 2012/09/17 7:01 am

I implemented 5G on a WP install with a totally custom theme: shamanscircle.com and users have not bee able to log in. My /wp-admin & /wp-login &/login redirect in a loop and had to remove the code from .htaccess.

I did see the removal of “menu” so it is like this:

RewriteCond %{QUERY_STRING} (mod|path|tag)=.?/? [NC,OR]

But it still didn’t work. Any ideas on what to do to fix this so i can continue to use 5G?

Cheers.
Amethyst Igor 2012/10/01 11:19 am

Hi Jeff, I use the 5G Blacklist, for which we are all thankful, I assure you. I have used your site for some time now. However I did detect one problem in this line:

RewriteCond %{QUERY_STRING} (menu|mod|path|tag)=.?/? [NC,OR]

This requires modification for WordPress compatibility. In my server log, I observed the following:

66.249.74.83 - - [30/Sep/2012:16:55:25 -0600] "GET /?tag=mooncakes HTTP/1.1" 403 687 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The IP address above is owned by Google, which we do not wish to serve 403’s. One solution is to remove “tag” from the query string, above, in .htaccess. I may make other mods as needed, but this is one thing I noticed right away in digging through my server log for 403’s.

Have a good day and please continue your work.
- Jeff Starr 2012/10/01 12:02 pm • Post Author
  
  Good catch — will make sure that tag is removed from the in-progress 6G Blacklist. Thanks for the infos!
- Tim 2012/10/01 6:08 pm
  
  Amethyst, Thanks for posting. If you or another moderator could share a little info on how to dig through server logs for 403’s. Is it just “grep ‘ 403 ‘ access.log? or are there other methods.
Bill 2012/10/03 8:21 am

Hi Jeff, just found a reference to your site from one of the joomla / virtuemart forums.

Just testing it out and it looks like I have a problem with this line:

RedirectMatch 403 (,|//|)+|/,/|{0}|(/(|...|+++|||\"\")

When trying to log in. Joomla 1.5.26 vm1.9
Gizmoscoop 2012/10/10 7:49 pm

Is the Ultimate htaccess Blacklist obsolete with this 2012 5G Blacklist?
- Jeff Starr 2012/10/11 11:34 am • Post Author
  
  Yes :)
  - Gizmoscoop 2012/10/11 9:22 pm
    
    Hehe, thanks. Just deleted it, and my .htacess file looks shorter now.
Mike 2012/10/27 1:27 pm

Hi,
I have tried to implement this on my new wordpress install. When I do I get internal server error 500.

Where would I start to troubleshoot this?
Happy trails, Mike
- Jeff Starr 2012/10/27 2:56 pm • Post Author
  
  Check the access and error logs for your server, or ask your host where to find them. Server logs will give more information as to what caused the error.
  - Mike 2012/10/27 3:12 pm
    
    Thanks Jeff. The error logs reported a missing closing tag ‘>’. My fault. Hey this htaccess doesnt block google bot does it?
    
    Thank you sir.
    Mike
  - Jeff Starr 2012/10/27 3:33 pm • Post Author
    
    No, the 5G is designed to only block bad requests and allow everyone else, especially the search engines. Even so, I’m retesting the 5G with Google Webmaster Tools just to be safe. It’s been about a week and googlebot has had zero issues accessing any legit URLs.
Ron 2012/10/27 4:09 pm

Hi Jeff just letting you know that the code was giving me some problems with WordPress 3.4.2 when creating a custom menu I kept getting a 404 page not found error from this URL – /wp-admin/nav-menus.php?action=edit&menu=0

Really love the 5G would like a fix for it if possible
Dragan 2012/11/01 4:09 pm

Hi,

I’m using 5G firewall on a LiteSpeed server and I can’t seem to stop one rogue bot.

This is what he did last month, data taken from awstats
Unknown robot (identified by ‘bot*’)
Hits: 252,300+280
Bandwidth: 3.07 GB

But it is getting worse, because just today he ate 166.02 MB and had 16,120+6 hits, and this was only until 12:00pm.

My limit is 30 GB/month, so this is really too much. Can you please help me identify this bot and set up a htaccess rule to stop it?

Thank you in advance
Sumon farazi 2012/11/11 4:00 am

Hello sir. thank you for see me. I am a junior web developer. When i don’t understand anything, first visit this site and try to solved my problem by your post. However, i understand all everything about 5g except the last line ((( # Deny from 123.456.789 ))). Actually, i wanted to know can i replace my own IP when i use this 5g for my site or put this ((( 123.456.789 ))). please help me for choose the right IP when i using 5g for my site security. Thanks again.
- Jeff Starr 2012/11/12 2:55 pm • Post Author
  
  Hi Sumon, unless you have specific reason to do so, I would just omit the last (IP) section entirely. It’s mostly for blocking specific threats and is not needed for general security. If you do have an IP that you want to block, then add it to that section of the blacklist.
Sumon farazi 2012/11/14 2:58 am

Hello sir. thank you for reply me with a good solution. Now solved my problem. Actually, you r my teacher (WordPress security). I see your WordPress security tutorial from Lynda.com and learn how to secure a WordPress website. I want to be a good WordPress developer. Thank you very much.

« Previous Comments • 1…678910 • Newer Comments »