Spring Sale! Save 30% on all books w/ code: PLANET24
Web Dev + WordPress + Security

5G Blacklist 2013

[ 5G (2013) ] Following up on much feedback (and this post), here is an update for the 5G Blacklist for 2013. As explained in the 2012 article (and elsewhere), the 5G Blacklist helps reduce the number of malicious URL requests that hit your website. It’s one of many ways to improve the security of your site and protect against evil exploits, bad requests, and other nefarious garbage. If your site runs on Apache and you’re familiar with .htaccess, the 5G is an effective way to secure your site against malicious HTTP requests and other suspect activity.

Update: Check out the new and improved 6G Firewall »

About the 5G Blacklist

The 5G Blacklist is a simple, flexible blacklist that checks all URI requests against a series of carefully constructed HTAccess directives. This happens quietly behind the scenes at the server level, saving resources for stuff like PHP and MySQL for all blocked requests.

How it works

Blacklists can block just about any part of a request: IP, user agent, request string, query string, referrer, and everything in between. But IP addresses change constantly, and user agents and referrers are easily spoofed. As discussed, request strings yield the best results: greater protection with fewer false positives.

The 5G works beautifully with WordPress, and should help any site conserve bandwidth and server resources while protecting against malicious activity.

5G Blacklist 2013

Here is the third version of the 5th generation blacklist:

# 5G BLACKLIST/FIREWALL (2013)
# @ https://perishablepress.com/5g-blacklist-2013/

# 5G:[QUERY STRINGS]
<IfModule mod_rewrite.c>
	RewriteEngine On
	RewriteBase /
	RewriteCond %{QUERY_STRING} (\"|%22).*(<|>|%3) [NC,OR]
	RewriteCond %{QUERY_STRING} (javascript:).*(\;) [NC,OR]
	RewriteCond %{QUERY_STRING} (<|%3C).*script.*(>|%3) [NC,OR]
	RewriteCond %{QUERY_STRING} (\\|\.\./|`|=\'$|=%27$) [NC,OR]
	RewriteCond %{QUERY_STRING} (\;|\'|\"|%22).*(union|select|insert|drop|update|md5|benchmark|or|and|if) [NC,OR]
	RewriteCond %{QUERY_STRING} (base64_encode|localhost|mosconfig) [NC,OR]
	RewriteCond %{QUERY_STRING} (boot\.ini|echo.*kae|etc/passwd) [NC,OR]
	RewriteCond %{QUERY_STRING} (GLOBALS|REQUEST)(=|\[|%) [NC]
	RewriteRule .* - [F]
</IfModule>

# 5G:[USER AGENTS]
<IfModule mod_setenvif.c>
	# SetEnvIfNoCase User-Agent ^$ keep_out
	SetEnvIfNoCase User-Agent (binlar|casper|cmsworldmap|comodo|diavol|dotbot|feedfinder|flicky|ia_archiver|kmccrew|nutch|planetwork|purebot|pycurl|skygrid|sucker|turnit|vikspider|zmeu) keep_out
	<limit GET POST PUT>
		Order Allow,Deny
		Allow from all
		Deny from env=keep_out
	</limit>
</IfModule>

# 5G:[REQUEST STRINGS]
<IfModule mod_alias.c>
	RedirectMatch 403 (https?|ftp|php)\://
	RedirectMatch 403 /(https?|ima|ucp)/
	RedirectMatch 403 /(Permanent|Better)$
	RedirectMatch 403 (\=\\\'|\=\\%27|/\\\'/?|\)\.css\()$
	RedirectMatch 403 (\,|\)\+|/\,/|\{0\}|\(/\(|\.\.\.|\+\+\+|\||\\\"\\\")
	RedirectMatch 403 \.(cgi|asp|aspx|cfg|dll|exe|jsp|mdb|sql|ini|rar)$
	RedirectMatch 403 /(contac|fpw|install|pingserver|register)\.php$
	RedirectMatch 403 (base64|crossdomain|localhost|wwwroot|e107\_)
	RedirectMatch 403 (eval\(|\_vti\_|\(null\)|echo.*kae|config\.xml)
	RedirectMatch 403 \.well\-known/host\-meta
	RedirectMatch 403 /function\.array\-rand
	RedirectMatch 403 \)\;\$\(this\)\.html\(
	RedirectMatch 403 proc/self/environ
	RedirectMatch 403 msnbot\.htm\)\.\_
	RedirectMatch 403 /ref\.outcontrol
	RedirectMatch 403 com\_cropimage
	RedirectMatch 403 indonesia\.htm
	RedirectMatch 403 \{\$itemURL\}
	RedirectMatch 403 function\(\)
	RedirectMatch 403 labels\.rdf
	RedirectMatch 403 /playing.php
	RedirectMatch 403 muieblackcat
</IfModule>

# 5G:[REQUEST METHOD]
<ifModule mod_rewrite.c>
	RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
	RewriteRule .* - [F]
</IfModule>

# 5G:[BAD IPS]
<limit GET POST PUT>
	Order Allow,Deny
	Allow from all
	# uncomment/edit/repeat next line to block IPs
	# Deny from 123.456.789
</limit>

To use: include the entire 5G Blacklist in the root .htaccess file of your site. Remember to backup your original .htaccess file before making any changes. Test thoroughly while enjoying your favorite beverage. If you encounter any issues, please read the troubleshooting tips and/or leave a comment to report a bug.

Note: in some cases it may be necessary to place the QUERY STRING rules before WP-permalink rules.

Update (2015/04/03): removed jakarta from the user-agent portion of the list. Reason? LinkedIn actually includes the term “jakarta” in their user-agent string:

LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)

</update>

Pre-changelog notes

The changes made for 5G 2013 are aimed at maximizing compatibility. Unfortunately, a number of required changes are due to improper coding and ignoring HTTP specifications. As mentioned previously, using unsafe characters in URLs obsoletes security measures that are based on pattern-matching, which is integral to the process of blocking malicious activity.

To illustrate, it is possible to protect against a wide range of malicious requests by blocking unsafe characters such as unencoded question marks “?” included within the query string. Firewalls, blacklists, security plugins and scripts are able to safely block such bad requests UNTIL some widely used service such as Google Adwords decides to start including multiple unencoded question marks in their query strings. Suddenly blocking potentially dangerous “?” requests is useless because nobody wants to block legitimate (Google) traffic.

Moral of the story: if you develop for the Web, contribute to its security by encoding your URLs according to spec. If you use security plugins, firewalls/blackists, and scripts that rely on pattern-matching to protect your site, please encourage and educate others about the importance of adhering to HTTP specifications.</rant>

Changelog

Removed from QUERY STRINGS

  • Square brackets “[” and “]” (details)
  • Colon “:” (details)
  • Unencoded question mark “\?” (WP previews, Piwik, Adwords, et al)
  • Removed “(menu|mod|path|tag)\=\.?/?” (WP menus, WP Super Cache, Joomla, Googlebot, et al)
  • Removed “environ” (common string)
  • Removed “scanner” (various WP plugins)
  • Removed “%3E” (common string)
  • Escaped backslash, from “\” to “\\

Removed from USER AGENTS

  • Commented out match for blank/empty user-agent “^$” (PayPal, WP-Piwik, et al)
  • Removed match for “libwww” (used by Lynx browser)

Removed from REQUEST STRINGS

  • Double forward slash “//” (Pingdom, gtmetrix, et al)
  • Removed match for “/cgi/” (Fancy indexes, Authentication)

Added to QUERY STRINGS (5G 2013)

  • TRACE” and “TRACK
  • base64_encode.*\(
  • \|%3E
  • GLOBALS(=|\[|\%)
  • REQUEST(=|\[|\%)
  • `
  • (\"|%22).*(<|>|%3)
  • (<|%3C).*script.*(>|%3)
  • (javascript:).*(\;)
  • (\;|\'|\"|%22).*(union|select|insert|drop|update|md5|benchmark|or|and|if)

Other changes

Optimized syntax, improved formatting.

Troubleshooting

If there is an error, remove the code and make a backup of your original .htaccess file (if you haven’t already done so). Investigate the URL for whichever page is blocked or not working, making note of any non-alphanumeric characters or anything else that looks unusual. With a good idea of what to look for, examine the 5G directives to see if anything looks similar. If so, try removing (or commenting out) the offending line (or characters) and see if that resolves the issue.

If that doesn’t work, further investigation is required, and there are numerous ways of going about it. Here is a good walkthrough of my halving method of isolating problematic code, which I recommend unless you have your own favorite way of troubleshooting ;)

Show support

If you benefit from my work with the 5G and would like to show support, consider buying a copy of my book, .htaccess made easy. You’ll get a complete guide to .htaccess, exclusive forum access, and a ton of awesome techniques for configuring, optimizing, and securing your site. Your generous support allows me to continue developing 5G/6G and other awesome resources for the community. Thank you!

Disclaimer

The 5G Firewall is provided “as-is”, with the intention of helping site administrators protect their sites against bad requests and other malicious activity. The code is open and free to use and modify as long as the first two credit lines remain intact. By using this code you assume all risk & responsibility for anything that happens, whether good or bad. In short, use wisely, test thoroughly, don’t sue me.

Learn more..

To learn more about the theory and development of the 5G Firewall, check out my articles on building the 3G, 4G and 5G Blacklist. The 6G beta article also contains some good information. And if all that’s not enough, a quick search for “blacklist” in the sidebar should also yield many results.

About the Author
Jeff Starr = Creative thinker. Passionate about free and open Web.
Digging Into WordPress: Take your WordPress skills to the next level.

91 responses to “5G Blacklist 2013”

  1. Hello…just to clarify: does the current blacklist mean I can do without a plugin like WP-Ban? Or, should WP-Ban be used in conjunction with it? Thanks in advance.

  2. If accidentally there’s malicious code within my theme, can 5G Blacklist 2013 protect and safe my site by stopping that junk code activity?

  3. hello…where can i find a list of bad ip addresses to put in the code?

    take care

  4. Berry Sizemore 2013/03/12 9:13 pm

    Hi Jeff,

    I wanted to let you know that 5G works with vBulletin 3.8.5. I have successfully reduced my bandwidth consumption about 20%, and relieved my database of load. The main thing that spammers target is the register.php page. A tweak here and there, and what was 30% of my requests have become 70% of my 403s! :-)

    Best,
    Berry Sizemore

  5. Hi Jeff,
    I’m sure I’m missing something here that’s obvious.

    I’ve been using and loving 5G for some time.
    I’ve used the “# 5G:[BAD IPS]” to block IP addresses successfully. However, I’ve run into a wall when it comes to blocking IP ranges.
    I’ve pared it down so that it’s only “# 5G:[BAD IPS]” and the standard WordPress .htaccess
    I’ve tried doing things like you have in your “Stupid htaccess tricks as well as other things I’ve seen.
    This is consistent across 3 popular hosts.
    Here’s what I find..

    deny from xxx.xxx.xxx.xxx gives 403
    deny from xxx.xxx gives 403
    deny from xxx.xxx.* does not deny
    deny from xxx.aaa/zzz gives 500
    deny from xxx.aaa/zzz.xxx gives 500

    You can see the problem. Am I missing something blindingly obvious?

    Thanks!

    • Jeff Starr 2013/03/14 2:07 pm

      I’m not sure about the 500 errors without looking into it, but the 403 response looks correct for the first two, and the third rule should work fine without the asterix, so just:

      Deny from 123.456.

    • The 500 errors are due to invalid syntax. deny from must use an IP address, partial IP address, or IP address block, e.g., 192.168.0.0/23

  6. Thanks for the ref P.Don!

    I have checked using my own IP (184.166.88.159), so that I can see both the 403s and 500s when I experimented with different methods.
    184.166.88.0/255 That’s a valid range that potentially my IP address could be and I get a 500 error using the / .
    184.166.88 and 184.166.88. and both properly deny

    Perhaps I tried to state my case previously too generally.

    I’m trying to block a particular IP range. I have had clients (without 5g) hacked from the IP 125.136.130.016 and other similar IPs nearby.

    Going to http://www.whatmyip.co/info/whois/125.136.130.16 I can see that the source of that IP has a range of 125.128.0.0 – 125.159.255.255 and I would like to block that entire range.
    I had thought that I could use 125.128/159 but obviously not.

    So… How the heck do I block that range in 5G?

    • 184.166.88.0/255 isn’t a valid specification. You either would use 184.166.88.0/24 (same first 24 bits) or 184.166.88.0/255.255.255.0 (same first 3 octets, or 24 bits).

      For your target block range of 125.128.0.0 – 125.159.255.255, you can do either of these:

      125.128.0.0/11
      125.128.0.0/255.255.224.0

      There are web tools to manipulate these formats, just search some combo of {CIDR, net block, subnet, tool, calculator}.

      • oops, sorry, 2nd one should be:
        255.224.0.0

      • So to be clear, lines in .htaccess would either be:

        # full network/nnn CIDR spec:
        deny from 125.128.0.0/11

        or:

        # full network/netmask spec:
        125.128.0.0/255.255.224.0

      • oops again, clearly I’m having cut-and-paste issues. The second option should read:
        125.128.0.0/255.224.0.0
        perhaps Jeff can edit/delete, but I think you get the idea ;)

  7. Thank you!!
    Would you know if there’s any difference performance-wise on the server?

    I did find an online calculator that takes the effort out of figuring it out http://ip2cidr.com/

    Thanks again!

  8. thank bro for this awesome post..

  9. ok, this part of 5G in wordpress 3.5.1 creates 404 error, if you switch between menus in Options > Appearance > Menu

    # 5G:[QUERY STRINGS]

    Now, i’m not an expert for htaccess obviously, so if anyone can elaborate why is this happening, i woul appreciate it very much! Since the rest of the code works ok.

    Thanks for the 5G any way!

    Editor’s note: 5G code removed to save space.

    • Hi Alex, what is the URL(s) that aren’t working when that code is included?

      • well, i see now in the third version of 5G that my code above is not the same. However, the link should be yoursite.com/wp-admin/nav-menus.php?action=edit&menu=14, but it displays my theme’s 404 page.

        But, i am using the latest, i believe, third version of 5G now, and everything is working ok for now! ;)

  10. Hi Jeff
    I’m having trouble with the 5G:[USER AGENTS] section. Apache complains “Order not allowed here” .

    Thanks,
    Peter

  11. Gizmoscoop 2013/03/23 10:45 am

    Hi Jeff,

    Thanks for the new 5G Blacklist. Does this list supersede the “4G Series: The Ultimate User-Agent Blacklist, Featuring Over 1200 Bad Bots” list? I noticed that a lot of bots from the 4G are not included in the 5G.

    • Yes 5G replaces 4G and previous. Some bots now are much less of a threat, so the list has been updated.

Comments are closed for this post. Something to add? Let me know.
Welcome
Perishable Press is operated by Jeff Starr, a professional web developer and book author with two decades of experience. Here you will find posts about web development, WordPress, security, and more »
Digging Into WordPress: Take your WordPress skills to the next level.
Thoughts
I live right next door to the absolute loudest car in town. And the owner loves to drive it.
8G Firewall now out of beta testing, ready for use on production sites.
It's all about that ad revenue baby.
Note to self: encrypting 500 GB of data on my iMac takes around 8 hours.
Getting back into things after a bit of a break. Currently 7° F outside. Chillz.
2024 is going to make 2020 look like a vacation. Prepare accordingly.
First snow of the year :)
Newsletter
Get news, updates, deals & tips via email.
Email kept private. Easy unsubscribe anytime.