Spring Sale! Save 30% on all books w/ code: PLANET24
Web Dev + WordPress + Security

5G Blacklist 2012

[ 5G (2012) ] The 5G Blacklist helps reduce the number of malicious URL requests that hit your website. It’s one of many ways to improve the security of your site and protect against evil exploits, bad requests, and other nefarious garbage. If you’re tired of all the slow, bloated security plugins and expensive 3rd-party services, the 5G Blacklist is a solid solution to help protect your Apache-powered site.

Update: Check out the new and improved 6G Firewall »

Evolution

After extensive beta testing, the 5G Blacklist/Firewall is solid and ready to help secure sites hosted on Apache servers. In addition to beta testing for the 5G, this is the 5th major update of my “G”-series blacklists. Here is a quick overview of its evolution.

  1. Ultimate htaccess Blacklist (Compressed Version)
  2. 2G Blacklist: Closing the Door on Malicious Attacks
  3. Perishable Press 3G Blacklist
  4. The Perishable Press 4G Blacklist
  5. 5G Firewall (Beta)

Along the way, I’ve explored a wide variety of different blacklist techniques. The 5G is the culmination of all these efforts, and will eventually be replaced by the imminent 6G Blacklist/Firewall.

What it does

The 5G Blacklist is a simple, flexible blacklist that checks all URI requests against a series of carefully constructed HTAccess directives. This happens quietly behind the scenes at the server level, saving resources for stuff like PHP and MySQL for all blocked requests.

How it works

Blacklists can block just about any part of a request: IP, user agent, request string, query string, referrer, and everything in between. But IP addresses change constantly, and user agents and referrers are easily spoofed. As discussed, request strings yield the best results: greater protection with fewer false positives.

The 5G works beautifully with WordPress, and should help any site conserve bandwidth and server resources while protecting against malicious activity.

How to use

To install the 5G Firewall, append the following code to your site’s root .htaccess:

# 5G BLACKLIST/FIREWALL
# @ https://perishablepress.com/5g-blacklist-2012/

# 5G:[QUERY STRINGS]
<ifModule mod_rewrite.c>
 RewriteEngine On
 RewriteBase /
 RewriteCond %{QUERY_STRING} (environ|localhost|mosconfig|scanner) [NC,OR]
 RewriteCond %{QUERY_STRING} (menu|mod|path|tag)\=\.?/? [NC,OR]
 RewriteCond %{QUERY_STRING} boot\.ini  [NC,OR]
 RewriteCond %{QUERY_STRING} echo.*kae  [NC,OR]
 RewriteCond %{QUERY_STRING} etc/passwd [NC,OR]
 RewriteCond %{QUERY_STRING} \=\\%27$   [NC,OR]
 RewriteCond %{QUERY_STRING} \=\\\'$    [NC,OR]
 RewriteCond %{QUERY_STRING} \.\./      [NC,OR]
 RewriteCond %{QUERY_STRING} \?         [NC,OR]
 RewriteCond %{QUERY_STRING} \:         [NC,OR]
 RewriteCond %{QUERY_STRING} \[         [NC,OR]
 RewriteCond %{QUERY_STRING} \]         [NC]
 RewriteRule .* - [F]
</ifModule>

# 5G:[USER AGENTS]
<ifModule mod_setenvif.c>
 SetEnvIfNoCase User-Agent ^$ keep_out
 SetEnvIfNoCase User-Agent (casper|cmsworldmap|diavol|dotbot)   keep_out
 SetEnvIfNoCase User-Agent (flicky|ia_archiver|jakarta|kmccrew) keep_out
 SetEnvIfNoCase User-Agent (libwww|planetwork|pycurl|skygrid)   keep_out
 SetEnvIfNoCase User-Agent (purebot|comodo|feedfinder|turnit)   keep_out
 SetEnvIfNoCase User-Agent (zmeu|nutch|vikspider|binlar|sucker) keep_out
 <limit GET POST PUT>
  Order Allow,Deny
  Allow from all
  Deny from env=keep_out
 </limit>
</ifModule>

# 5G:[REQUEST STRINGS]
<ifModule mod_alias.c>
 RedirectMatch 403 (https?|ftp|php)\://
 RedirectMatch 403 /(cgi|https?|ima|ucp)/
 RedirectMatch 403 /(Permanent|Better)$
 RedirectMatch 403 (\=\\\'|\=\\%27|/\\\'/?|\)\.css\()$
 RedirectMatch 403 (\,|//|\)\+|/\,/|\{0\}|\(/\(|\.\.\.|\+\+\+|\||\\\"\\\")
 RedirectMatch 403 \.(cgi|asp|aspx|cfg|dll|exe|jsp|mdb|sql|ini|rar)$
 RedirectMatch 403 /(contac|fpw|install|pingserver|register)\.php$
 RedirectMatch 403 (base64|crossdomain|localhost|wwwroot|e107\_)
 RedirectMatch 403 (eval\(|\_vti\_|\(null\)|echo.*kae|config\.xml)
 RedirectMatch 403 \.well\-known/host\-meta
 RedirectMatch 403 /function\.array\-rand
 RedirectMatch 403 \)\;\$\(this\)\.html\(
 RedirectMatch 403 proc/self/environ
 RedirectMatch 403 msnbot\.htm\)\.\_
 RedirectMatch 403 /ref\.outcontrol
 RedirectMatch 403 com\_cropimage
 RedirectMatch 403 indonesia\.htm
 RedirectMatch 403 \{\$itemURL\}
 RedirectMatch 403 function\(\)
 RedirectMatch 403 labels\.rdf
 RedirectMatch 403 /playing.php
 RedirectMatch 403 muieblackcat
</ifModule>

# 5G:[BAD IPS]
<limit GET POST PUT>
 Order Allow,Deny
 Allow from all
 # uncomment/edit/repeat next line to block IPs
 # Deny from 123.456.789
</limit>

That’s the golden ticket right there. The 5G Firewall is serious protection for your website: extensively tested, plug-n-play, and completely free. “Grab, gulp, n go” as they say. For more information, see the beta article (and comments).

Troubleshooting

Remember, test thoroughly. If something stops working when the 5G is installed, try removing the 5G. If things start working normally again, you can either pass on the 5G or investigate further. Investigating further is straightforward using something like the halving method, where you remove chunks of the 5G until isolating and identifying the issue. Here is a quick example:

  • I’ve installed the 5G, thanks Jeff.
  • Uh-oh, the page at http://example.com/indonesia.html stopped loading
  • Hmm, the URL contains the phrase “indonesia”, so let’s check the 5G for it
  • Yep, there’s a rule that blocks indonesia\.htm
  • Removing that line resolves the issue, thanks me.

Is it okay to remove rules that are blocking your own pages? Yes, the only downside is that malicious requests that would have otherwise been blocked will now get through. The 5G will continue to block a massive volume of malicious requests — it’ll just be a bit less effective. The protective effect is cumulative, not dependent on any one rule. So customization is encouraged. Once you dial it in, you’re all set.

Disclaimer

The 5G Firewall is provided “as-is”, with the intention of helping site administrators protect their sites against bad requests and other malicious activity. The code is open and free to use and modify as long as the first two credit lines remain intact. By using this code you assume all risk & responsibility for anything that happens, whether good or bad. In short, use wisely, test thoroughly, don’t sue me.

Learn more..

To learn more about the theory and development of the 5G Firewall, check out my articles on building the 3G, 4G and 5G Blacklist. A search for “blacklist” in the sidebar should also yield many results.

Happy securing!

About the Author
Jeff Starr = Web Developer. Security Specialist. WordPress Buff.
Blackhole Pro: Trap bad bots in a virtual black hole.

223 responses to “5G Blacklist 2012”

  1. Love it, Jeff. Thanks so much for keeping up on this AND for making it available. So valuable, man. Thanks again.

  2. JerryHopper 2012/01/11 12:53 pm

    Great stuff!
    i’ve been using last year’s edition, without problems. This is probably a must for all webmasters!

    thanks!

  3. rpoole@wh2p.com 2012/01/11 1:16 pm

    I’m thinking I’m using your last years version or maybe the version before? However I still get emails with “5 failed login attempts to account servername” every other day or so. Am I correct thinking this goes in the public_html directory? Or up a level?

    • Jeff Starr 2012/01/11 1:20 pm

      The public_html directory should do the trick, but if you’re getting errors, they can be tracked down in your server’s error log. Could be the 5G interfering, or it could be something else. Always good to investigate further.

  4. I’ve been using the 4G version (and added to it), so will implement this asap. Thanx for being so altruistic and sharing it.

    Cheers
    I

  5. Jon Bukiewicz 2012/01/11 5:16 pm

    Hi Jeff – the only issue I’m seeing is with timthumb files. Is there something I can comment out, or am I doing something wrong?

    http://dev.mostlyjunkfood.com/wp-content/themes/mostlyjunkfood-v.2/th.php?src=http://dev.mostlyjunkfood.com/treats/2012/01/beyonce_hov.jpeg&w=640&h=365

    • Jeff Starr 2012/01/11 9:39 pm

      Looks like a possible match with this line in the query-string section:

      RewriteCond %{QUERY_STRING} \: [NC,OR]

      try commenting that line out with a pound sign (#) and see what happens..

  6. Thanks Jeff, very cool. Is there any way of applying this at a server level or does it have to be on a site by site basis? Thanks.

    • Jeff Starr 2012/01/11 9:41 pm

      It would involve some reformatting of the code, but yes you could add this to Apache’s configuration file and apply to any/all sites on the server.

      • Scott Cariss 2012/01/12 1:42 am

        I’d realy love to see how you would do this. I’m about to move my clients onto a VPS and would be nice to get this running on a server level rather than a site by site basis. Any chance on a how to article bout this Jeff? I’d be forever in your debt.

  7. Is this something you might consider doing?

  8. Tam Nguyen Photography 2012/01/11 11:38 pm

    I’ve run into a few problems, and was able to fix them since last version. Here they are:

    # RewriteRule .* - [F] # having issue with timthumb if this line is turned on
    
    # RedirectMatch 403 (,|//|)+|/,/|{0}|(/(|...|+++|||\"\") # "//": Montastic, 100pulse
    
    # RedirectMatch 403 (base64|crossdomain|localhost|wwwroot) # crossdomain: cooliris

    Cheers,

  9. Hi Jeff,

    thanks for releasing this! I’ve been using the 5G since the beta and found it very handy, along with your blackhole.

    There is one thing that causes some problems in WordPress with nab menus: the second RewriteCond line contains ‘menu’, which for me on all of my installs (local and remote) causes a 403 t be thrown. Removing ‘menu’ fixes this.

    thanks again,

    Frances

  10. Happy New Year to you and yours, Jeff. Greets from the UK, innit?

    You know, I’ve been following your nG Block Lists for a while and I’ve gotten infected with the total OCD-ness of them. It’s like RAGE on steroids; I share your pain.

    Your 5G is cool, but consider this: the current Apache regex engine is pretty sophisticated (it’s Perl 5). You can optimize it with payback. Couple of tips:

    Concatenate patterns as much as you can. Up to a pattern length of 300-500 characters, there’s a significant performance boost compared to iterating multiple, shorter patterns. Above that, the overhead of long-pattern syntax-checking starts to kick in. Law of diminishing returns, I guess.

    Wherever possible, your regex patterns should look to start on some boundary, even if it’s just a word boundary (b). Try to avoid creating unnecessary back-references, which slow the engine.

    If you want to use mod_setenvif on the user agent, then the BrowserMatchNoCase subset is a better alternative to SetEnvIfNoCase User-Agent. Using a <Limit GET POST PUT> container is not actually required, and may not always get you the result you want. mod_access is part of core; the Deny directive can just be dropped in.

    If you’re denying or redirecting and you have a custom 403 or 410 page, you MUST prefix a mod-rewrite section with an ‘escape clause’ to let them through to that ‘error page’.

    FYI, inspired by your previous work, I too have been working on a firewall generator, based on analysis of my own access logs since Spring 2011.

    You can view the current state of progress. Mind-boggling I know, but I’ll be interested to hear what others think.

    Posting the generator script at ZB soon.

    Maybe we could knock heads on this? I have many other juicy bits to share.

    • I post here to get as much attention as possible:

      Deny from 78.160.15.253
      Deny from 78.160.52.13

      You are safe to block these 2 IP addresses, as they were used by some Turkish hackers on my site. My site was hacked 3 times in the last 1 month, it is very frustrating.

      This is also a good post regarding WordPress security.

      • That’s Turk Telekom. Its network is riddled with compromised domestic workstations on DSL. Botnet boxes, in other words. It’s hard to find an IP in the 78.60.0.0 range that isn’t known to be connected to hacking and/or spamming. Search on ‘AS report AS9121’, get an idea of the scale of their network.

        You could Deny from 78.160.0.0/11, but they’ll still get at you from another range. It’s not a trivial task.

      • True, banning IP’s is not the most reliable way of protection. I do not know if it is the same group, but the last time (today) I could see some actions from an US IP and one from Italy, too.

        Somehow they got access to wp-admin and could edit from there theme and plugin php files.

  11. Bobby Jones 2012/01/12 11:21 am

    Awesome! good work on the 5G Blacklist, I can confirm the Menu issue reported by frances removing the word menu from line two of the QUERY STRINGS section does indeed fix the problem. Also you may want to let Joomla users know that the word “path” in the same line will cause a 403 error if you attempt to access the Options menu from within the Article Manager. Removing the word path fixes this issue. Thanks again keep up the good work!

  12. Wow! Kiwis pick stuff up real quick!

Comments are closed for this post. Something to add? Let me know.
Welcome
Perishable Press is operated by Jeff Starr, a professional web developer and book author with two decades of experience. Here you will find posts about web development, WordPress, security, and more »
Digging Into WordPress: Take your WordPress skills to the next level.
Thoughts
I live right next door to the absolute loudest car in town. And the owner loves to drive it.
8G Firewall now out of beta testing, ready for use on production sites.
It's all about that ad revenue baby.
Note to self: encrypting 500 GB of data on my iMac takes around 8 hours.
Getting back into things after a bit of a break. Currently 7° F outside. Chillz.
2024 is going to make 2020 look like a vacation. Prepare accordingly.
First snow of the year :)
Newsletter
Get news, updates, deals & tips via email.
Email kept private. Easy unsubscribe anytime.