Perishable Press 3G Blacklist
![[ 3G Stormtroopers ]](https://perishablepress.com/wp/wp-content/images/2008/blacklist-3g/blacklist-release.jpg)
After much research and discussion, I have developed a concise, lightweight security strategy for Apache-powered websites. Prior to the development of this strategy, I relied on several extensive blacklists to protect my sites against malicious user agents and IP addresses.
Over time, these mega-lists became unmanageable and ineffective. As increasing numbers of attacks hit my server, I began developing new techniques for defending against external threats. This work soon culminated in the release of a “next-generation” blacklist that works by targeting common elements of decentralized server attacks.
Consisting of a mere 37 lines, this “2G” Blacklist provided enough protection to enable me to completely eliminate over 350 blacklisting directives from my site’s root htaccess file. This improvement increased site performance and decreased attack rates, however many bad hits were still getting through. More work was needed..
The 3G Blacklist
Work on the 3G Blacklist required several weeks of research, testing, and analysis. During the development process, five major improvements were discovered, documented, and implemented. Using pattern recognition, access immunization, and multiple layers of protection, the 3G Blacklist serves as an extremely effective security strategy for preventing a vast majority of common exploits. The list consists of four distinct parts, providing multiple layers of protection while synergizing into a comprehensive defense mechanism. Further, as discussed in previous articles, the 3G Blacklist is designed to be as lightweight and flexible as possible, thereby facilitating periodic cultivation and maintenance. Sound good? Here it is:
# PERISHABLE PRESS 3G BLACKLIST
# PART I: CHARACTER STRINGS
<IfModule mod_alias.c>
RedirectMatch 403 \:
RedirectMatch 403 \;
RedirectMatch 403 \<
RedirectMatch 403 \>
RedirectMatch 403 \/\,
RedirectMatch 403 \/\/
RedirectMatch 403 f\-\.
RedirectMatch 403 \.\.\.
RedirectMatch 403 \.inc
RedirectMatch 403 alt\=
RedirectMatch 403 ftp\:
RedirectMatch 403 ttp\:
RedirectMatch 403 \.\$url
RedirectMatch 403 \/\$url
RedirectMatch 403 \/\$link
RedirectMatch 403 news\.php
RedirectMatch 403 menu\.php
RedirectMatch 403 main\.php
RedirectMatch 403 home\.php
RedirectMatch 403 view\.php
RedirectMatch 403 about\.php
RedirectMatch 403 blank\.php
RedirectMatch 403 block\.php
RedirectMatch 403 order\.php
RedirectMatch 403 search\.php
RedirectMatch 403 errors\.php
RedirectMatch 403 button\.php
RedirectMatch 403 middle\.php
RedirectMatch 403 threads\.php
RedirectMatch 403 contact\.php
RedirectMatch 403 include\.php
RedirectMatch 403 display\.php
RedirectMatch 403 register\.php
RedirectMatch 403 authorize\.php
RedirectMatch 403 \/wp\-signup\.php
RedirectMatch 403 \/classes\/
RedirectMatch 403 \/includes\/
RedirectMatch 403 \/path\_to\_script\/
RedirectMatch 403 ImpEvData\.
RedirectMatch 403 head\_auth\.
RedirectMatch 403 db\_connect\.
RedirectMatch 403 check\_proxy\.
RedirectMatch 403 doeditconfig\.
RedirectMatch 403 submit\_links\.
RedirectMatch 403 change\_action\.
RedirectMatch 403 send\_reminders\.
RedirectMatch 403 comment\-template\.
RedirectMatch 403 syntax\_highlight\.
RedirectMatch 403 admin\_db\_utilities\.
RedirectMatch 403 admin\.webring\.docs\.
RedirectMatch 403 function\.main
RedirectMatch 403 function\.mkdir
RedirectMatch 403 function\.opendir
RedirectMatch 403 function\.require
RedirectMatch 403 function\.array\-rand
RedirectMatch 403 ref\.outcontrol
</IfModule>
# PART II: QUERY STRINGS
<ifmodule mod_rewrite.c>
RewriteCond %{QUERY_STRING} ftp\: [NC,OR]
RewriteCond %{QUERY_STRING} http\: [NC,OR]
RewriteCond %{QUERY_STRING} https\: [NC,OR]
RewriteCond %{QUERY_STRING} \[ [NC,OR]
RewriteCond %{QUERY_STRING} \] [NC]
RewriteRule .* - [F,L]
</ifmodule>
# PART III: USER AGENTS
SetEnvIfNoCase User-Agent "Jakarta Commons" keep_out
SetEnvIfNoCase User-Agent "Y!OASIS/TEST" keep_out
SetEnvIfNoCase User-Agent "libwww-perl" keep_out
SetEnvIfNoCase User-Agent "MOT-MPx220" keep_out
SetEnvIfNoCase User-Agent "MJ12bot" keep_out
SetEnvIfNoCase User-Agent "Nutch" keep_out
SetEnvIfNoCase User-Agent "cr4nk" keep_out
<Limit GET POST PUT>
order allow,deny
allow from all
deny from env=keep_out
</Limit>
# PART IV: IP ADDRESSES
<Limit GET POST PUT>
order allow,deny
allow from all
deny from 75.126.85.215 "# blacklist candidate 2008-01-02 = admin-ajax.php attack "
deny from 128.111.48.138 "# blacklist candidate 2008-02-10 = cryptic character strings "
deny from 87.248.163.54 "# blacklist candidate 2008-03-09 = block administrative attacks "
deny from 84.122.143.99 "# blacklist candidate 2008-04-27 = block clam store loser "
</Limit>Installation and Usage
Before using the 3G Blacklist, check the following system requirements:
- Linux server running Apache
- Enabled Apache module:
mod_alias - Enabled Apache module:
mod_rewrite - Ability to edit your site’s root htaccess file (or)
- Ability to modify Apache’s server configuration file
With these requirements met, copy and paste the entire 3G Blacklist into either the root htaccess file or the server configuration file. After uploading, visit your site and check proper loading of as many different types of pages as possible. For example, if you are running a blogging platform (such as WordPress), test different page views (single, archive, category, home, etc.), log into and surf the admin pages (plugins, themes, options, posts, etc.), and also check peripheral elements such as individual images, available downloads, and alternate protocols (FTP, HTTPS, etc.).
While the 3G Blacklist is designed to target only the bad guys, the regular expressions used in the list may interfere with legitimate URL access. If this happens, the browsing device will display a 403 Forbidden error. Don’t panic! Simply check the blocked URL, locate the matching blacklist string, and disable the directive by placing a pound sign ( # ) at the beginning of the associated line. Once the correct line is commented out, the blocked URL should load normally. Also, if you do happen to experience any conflicts involving the 3G Blacklist, please leave a comment or contact me directly. Thank you :)
Wrap Up..
As my readers know, I am serious about site security. Nothing gets my adrenaline pumping more than the thought of a giant meat grinder squirting out endless chunks of mangled cracker meat. Spam and other exploitative activity on the web has grown exponentially. Targeting and blocking individual agents and IP is no longer a viable strategy. By recognizing and immunizing against the broadest array of common attack elements, the 3G Blacklist maximizes resources while providing solid defense against malicious attacks.
Updates
Updates to the 3G Blacklist/firewall:
2008/05/14
Removed “RedirectMatch 403 \/scripts\/” from the first part of the blacklist due to conflict with Mint Statistics.
2008/05/18
Removed the following three directives to facilitate Joomla functionality:
RedirectMatch 403 \/modules\/
RedirectMatch 403 \/components\/
RedirectMatch 403 \/administrator\/2008/05/31
Removed “RedirectMatch 403 config\.php” from the first part of the list to ensure proper functionality with the “visual-editing” feature of the WordPress Admin Area.
84 responses to “Perishable Press 3G Blacklist”
This is solid gold :)
I’ve had no issue so far with the whole list past into my root .htaccess. I’ll see if it affects the administration of my blog in the futur.
Really great stuff as usual from you. The list is clean and comprehensive.
The formatting syntax with the little spaces before Redirect directives is a little strange; is it your new favorite syntax ? :d
Excellent, thanks for trying it out :) I did as much testing as possible (primarily via WordPress), however it is important to hear how the method works (or doesn’t) in other environments. One thing I failed to emphasize in the article series involves global noninterference. There are many possible strings to blacklist — many of which would be far more effective and efficient — but doing so would result in conflicts and errors on various platforms. Anyway, I am getting carried away here, but I am stoked that you are using 3G! (and yes, the indentation of nested directives is important — even in htaccess files ;)
I might be biased by the excellent post I recently read on Daringfireball – on “Why Apple won’t buy Adobe“, but it might be true in our case that making the rules generic enough so it won’t break any forum/CMS/whatever plateform, also makes the rules less relevant and powerful.
I think that it may be better to keep focused on one environnement only (i.e. WordPress), and do a heck of a job there. We could also have different variants of the list, for different plateforms.
Anyway, that is just my opinion, as a WordPress lover and PerishablePress reader.
About my question on indentation, it was more the fact that you are using one space to indent, instead of the classic tabulation. I wondered what did motivate this choice.
This 3G series is great. I’ve been following all the articles and updating my .htaccess in small chunks.
Everything is working magically. I’m glad that you got the update in for the Mint fix. I was going to say something, but you beat me to it.
As always great work man! I had a quick test run with it on WP; no holes!
On joomla (1.013) though a few issues that can be resolved my removing these lines:
RedirectMatch 403 /modules/RedirectMatch 403 /components/RedirectMatch 403 /administrator/I haven’t fully tested in Joomla though; will do when time permits. I also noticed you reduced the list of user agents. may i ask the benefits of the downsize?
@Louis: good point — the 3G Blacklist may very well evolve along such a trajectory in the future. For now, however, the strategy has proven general enough to protect across platforms without unintentional functional interference. Subsequent editing of 3G directives is critical for the fine-tuning of targeting specificity. This pruning process is greatly facilitated by 3G users such as Don, who continues to help test the list on the Joomla platform. OT: I think tabbed spacing in htaccess code is just too much, almost to the point of looking ridiculous ;)
@Alissa: Thanks for the positive feedback. I am glad that you are taking advantage of the information :) I should mention that the complete version of the 3G Blacklist (as presented in this article) contains slightly different content than presented in the series. You may want to compare the individual series chunks with the complete version, as only it will be kept current with updates and other findings. Also, good call on Mint — if anything else turns up, please let us know!
@Don: Thank you — list updated and credit given for the Joomla-conflicting directives. As always, thanks for your help with the Joomla testing. I look forward to hearing of any other potential issues. Actually, I have been considering removing the other three directory path patterns as well. If memory serves, their use in exploitative attacks is rarely seen outside of query strings containing full URL path information. Thus, explicitly blocking the path segments is essentially unnecessary, potentially doing more harm than good, especially given their prevalence in modern file architecture. So, I’ll hedge for now, but wouldn’t be surprised to see them disappear before the end of the month..
As for the downsizing of the user-agent blacklist, there are several benefits, including increased performance, scalability, flexibility, and relevance. Of course, 3G users are welcome to drop in a full copy of the Ultimate htaccess Blacklist or any other mega-list for that matter. For the 3G, I wanted to wipe the slate clean, give it a fresh start and add new agents on an “as-needed” basis. I could go on about it, but rather I will point you to this article for the full story.
Well, all of this is now implemented on a WordPress site for our company. Having tested all the features of the site, everything is still working just as great – but now with added security. Thanks Jeff!
On the admin side of things however, it appears that “RedirectMatch 403 config\.php” breaks the “visual” mode when creating/editing posts in WordPress for me. This is no big deal really, but I just wanted to confirm, if this is the case for you too, or if I missed something.
Cheers,
Sat.
Thanks for the feedback, Sat. Checking into this, I did notice that the visual-editing mode seemed not to work in WordPress 2.3. I have yet to test on WordPress 2.5, but even if visual editing works in that version, breaking any WordPress-related functionality is a bad idea (especially in 2.3!). Thus, I have removed the offending rule and updated the article accordingly. Thank you for your help in improving the quality of the 3G Blacklist!
Cheers,
Jeff
Hi Jeff,
I couldn’t wipe out of my head what John Gruber once said about WordPress and caching. It’s such a shame caching isn’t a part of WordPress core.
Therefore, after giving it some time to grow, I decided to give WP-SuperCache a chance. After some tweaking, it seems to be working on my blog.
In the process, I discovered that the following line of your wonderful 3G Blacklist was causing the plugin not to serve the .html or .html.gz files.
RedirectMatch 403 //
I commented it out and the rest of the blacklist seems to be okay.
I hope it helps :)
Louis! Good to see you again (I was beginning to wonder if everything was okay..;)
Anyway, check out Alissa Miller’s excellent article covering the Super-Cache/3G-Blacklist conflict. Apparently, Super Cache rewrites permalinks with an extra trailing slash, thus triggering the double-forward slash block from the blacklist. Alissa documents the issue extremely well and provides an easy fix that ensures continued success with the double-forward slash filter.
Regards,
Jeff