5G Blacklist 2013
Following up on much feedback (and this post), here is an update for the 5G Blacklist for 2013. As explained in the 2012 article (and elsewhere), the 5G Blacklist helps reduce the number of malicious URL requests that hit your website. It’s one of many ways to improve the security of your site and protect against evil exploits, bad requests, and other nefarious garbage. If your site runs on Apache and you’re familiar with .htaccess, the 5G is an effective way to secure your site against malicious HTTP requests and other suspect activity.
About the 5G Blacklist
The 5G Blacklist is a simple, flexible blacklist that checks all URI requests against a series of carefully constructed HTAccess directives. This happens quietly behind the scenes at the server level, saving resources for stuff like PHP and MySQL for all blocked requests.
How it works
Blacklists can block just about any part of a request: IP, user agent, request string, query string, referrer, and everything in between. But IP addresses change constantly, and user agents and referrers are easily spoofed. As discussed, request strings yield the best results: greater protection with fewer false positives.
The 5G works beautifully with WordPress, and should help any site conserve bandwidth and server resources while protecting against malicious activity.
5G Blacklist 2013
Here is the third version of the 5th generation blacklist:
To use: include the entire 5G Blacklist in the root .htaccess file of your site. Remember to backup your original .htaccess file before making any changes. Test thoroughly while enjoying your favorite beverage. If you encounter any issues, please read the troubleshooting tips and/or leave a comment to report a bug.
Note: in some cases it may be necessary to place the QUERY STRING rules before WP-permalink rules.
Update (2015/04/03): removed
jakarta from the user-agent portion of the list. Reason? LinkedIn actually includes the term “jakarta” in their user-agent string:
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
The changes made for 5G 2013 are aimed at maximizing compatibility. Unfortunately, a number of required changes are due to improper coding and ignoring HTTP specifications. As mentioned previously, using unsafe characters in URLs obsoletes security measures that are based on pattern-matching, which is integral to the process of blocking malicious activity.
To illustrate, it is possible to protect against a wide range of malicious requests by blocking unsafe characters such as unencoded question marks “
?” included within the query string. Firewalls, blacklists, security plugins and scripts are able to safely block such bad requests UNTIL some widely used service such as Google Adwords decides to start including multiple unencoded question marks in their query strings. Suddenly blocking potentially dangerous “
?” requests is useless because nobody wants to block legitimate (Google) traffic.
Moral of the story: if you develop for the Web, contribute to its security by encoding your URLs according to spec. If you use security plugins, firewalls/blackists, and scripts that rely on pattern-matching to protect your site, please encourage and educate others about the importance of adhering to HTTP specifications.
Removed from QUERY STRINGS
- Square brackets “
[” and “
- Colon “
- Unencoded question mark “
\?” (WP previews, Piwik, Adwords, et al)
- Removed “
(menu|mod|path|tag)\=\.?/?” (WP menus, WP Super Cache, Joomla, Googlebot, et al)
- Removed “
environ” (common string)
- Removed “
scanner” (various WP plugins)
- Removed “
%3E” (common string)
- Escaped backslash, from “
\” to “
Removed from USER AGENTS
- Commented out match for blank/empty user-agent “
^$” (PayPal, WP-Piwik, et al)
- Removed match for “
libwww” (used by Lynx browser)
Removed from REQUEST STRINGS
- Double forward slash “
//” (Pingdom, gtmetrix, et al)
- Removed match for “
/cgi/” (Fancy indexes, Authentication)
Added to QUERY STRINGS (5G 2013)
TRACE” and “
Optimized syntax, improved formatting.
If there is an error, remove the code and make a backup of your original .htaccess file (if you haven’t already done so). Investigate the URL for whichever page is blocked or not working, making note of any non-alphanumeric characters or anything else that looks unusual. With a good idea of what to look for, examine the 5G directives to see if anything looks similar. If so, try removing (or commenting out) the offending line (or characters) and see if that resolves the issue.
If that doesn’t work, further investigation is required, and there are numerous ways of going about it. Here is a good walkthrough of my halving method of isolating problematic code, which I recommend unless you have your own favorite way of troubleshooting ;)
If you benefit from my work with the 5G and would like to show support, consider buying a copy of my book, .htaccess made easy. You’ll get a complete guide to .htaccess, exclusive forum access, and a ton of awesome techniques for configuring, optimizing, and securing your site. Your generous support allows me to continue developing 5G/6G and other awesome resources for the community. Thank you!
The 5G Firewall is provided “as-is”, with the intention of helping site administrators protect their sites against bad requests and other malicious activity. The code is open and free to use and modify as long as the first two credit lines remain intact. By using this code you assume all risk & responsibility for anything that happens, whether good or bad. In short, use wisely, test thoroughly, don’t sue me.
To learn more about the theory and development of the 5G Firewall, check out my articles on building the 3G, 4G and 5G Blacklist. The 6G beta article also contains some good information. And if all that’s not enough, a quick search for “blacklist” in the sidebar should also yield many results.
Thanks for all your hard work, I have immediately replaced my old 5G with this one.
Up and running on my site. Thanks so much, Jeff!
Great to see an update – thanks for putting it together.
So is this to be used instead of the 6G beta? And no case-Insensitive RedirectMatch (?i) you used in the 6G beta? (not criticizing, just interested).
I just haven’t had time to work much more on the 6G, which remains on the horizon until later. There are a couple of tricks that will be used in 6G that aren’t used in 5G (as you mention), such as the case-insensitive RedirectMatch and positive/negative lookaheads (which are very cool).
I added two evil agent ZmEu and w00tw00t to the list
# SetEnvIfNoCase User-Agent ^$ keep_out
Again: Comodo is a right and safe company that make it’s personalized broswser from Firefox and Chromium.
More info: http://www.comodo.com/home/browsers-toolbars/internet-products.php
So, comodo match string is wrong.
Will the BBQ WordPress plugin adopt the updates, too?
Yes for the most part, some of the same rules are planned for the next BBQ update :)
For confirmation: If a wp site uses the BBQ plugin, is this code not required or it’s better if its implemented as well?
Thanks for sharing this awesome work!
There is a lot of overlap between BBQ and 5G, but they’re not identical. Personally I use 5G because the performance is a little better handling HTTP requests at the server level, but either one will help protect your site, and it’s fine to use both!
Works great with joomla sites.
At one stage, got internal 500 error, but that was because of a line break, that notepad must have put in …
the line under SetEnvIfNoCase User-Agent … as it was too long.
removed the line break when i pasted it into htaccess file, and all is well.
Thanks again Jeff
Do you still recommend using the 5G WordPress Add-on, or is that not needed any more?
The 5G WP addon continues to block items that aren’t included in regular 5G, so I do recommend both (until 6G!).
I’m new at this, but I really appreciate your hard work and the fact that you make this available. I just replaced the old 5G with this and I have a question for you. Does this include the Blacklist Candidate 2012-11-13: Evil Scanner Edition (mini firewall 2012-11-13) code or should I leave it in place in my .htaccess file? So far I’ve left it in place because I don’t see anything similar in the new 5G code, but, like I said, I am new at this. Thanks in advance!
Yes, the mini firewall protects against additional malicious requests, so it’s fine to use both :)
Definately the best version yet! I’ve found the
THE_REQUESTvariable especially effective for this more advanced anti-exploit code.
This is definately a one of a kind resource, you would be nuts not to use this. Can’t wait for the next versions!
Thank you Sir! Your kind feedback is very much appreciated and encouraging! :)
Thank you for your excellent work!!
Here’s some more for you to look at. Came across some new hacks being tried on our WordPress sites. Here’s some of the things I’m seeing in the url (seen 1/20/13)… (recorded by the “Better WP Security” WordPress plugin)
All my code is in the httpd.conf file (instead of .htaccess) to benefit the entire server (we have lots of domains). I hear putting the code in the .conf file supposedly helps lower the workload of the server a bit, too.
I know enough of this stuff to get around and break it kinda thing, but I’m no expert. Server logs had a few:
“Directory index forbidden by Options directive: /httpdocs/wp-includes/js/jquery/”
…for the time & ip, but not as many as recorded by the plugin.
Some of what I have helps blocks several common hacks to secure the wp-includes folder, like:
RewriteRule ^wp-admin/includes/ - [F,L]
RewriteRule !^wp-includes/ - [S=2]
RewriteRule ^wp-includes/js/tinymce/langs/.+\.php - [F,L]
RewriteRule ^wp-includes/theme-compat/ - [F,L]
…and then I just added:
RedirectMatch 403 \!\=\=
RedirectMatch 403 \=\=\=
RedirectMatch 403 jquery\/\&\&
Hopefully this offers some value you can use it to make your blacklist even better!
Thank you again!
Awesome stuff, Mike! Looking forward to checking out these rules in more depth soon :)