Latest TweetsAll o' me plugins freshly updated and ready for WP 5.0 :) profiles.wordpress.org/special…
Perishable Press

Perishable Press 3G Blacklist

[ 3G Stormtroopers ]

After much research and discussion, I have developed a concise, lightweight security strategy for Apache-powered websites. Prior to the development of this strategy, I relied on several extensive blacklists to protect my sites against malicious user agents and IP addresses.

Over time, these mega-lists became unmanageable and ineffective. As increasing numbers of attacks hit my server, I began developing new techniques for defending against external threats. This work soon culminated in the release of a “next-generation” blacklist that works by targeting common elements of decentralized server attacks.

Consisting of a mere 37 lines, this “2G” Blacklist provided enough protection to enable me to completely eliminate over 350 blacklisting directives from my site’s root htaccess file. This improvement increased site performance and decreased attack rates, however many bad hits were still getting through. More work was needed..

The 3G Blacklist

Work on the 3G Blacklist required several weeks of research, testing, and analysis. During the development process, five major improvements were discovered, documented, and implemented. Using pattern recognition, access immunization, and multiple layers of protection, the 3G Blacklist serves as an extremely effective security strategy for preventing a vast majority of common exploits. The list consists of four distinct parts, providing multiple layers of protection while synergizing into a comprehensive defense mechanism. Further, as discussed in previous articles, the 3G Blacklist is designed to be as lightweight and flexible as possible, thereby facilitating periodic cultivation and maintenance. Sound good? Here it is:

# PERISHABLE PRESS 3G BLACKLIST

# PART I: CHARACTER STRINGS
<IfModule mod_alias.c>
 RedirectMatch 403 \:
 RedirectMatch 403 \;
 RedirectMatch 403 \<
 RedirectMatch 403 \>
 RedirectMatch 403 \/\,
 RedirectMatch 403 \/\/
 RedirectMatch 403 f\-\.
 RedirectMatch 403 \.\.\.
 RedirectMatch 403 \.inc
 RedirectMatch 403 alt\=
 RedirectMatch 403 ftp\:
 RedirectMatch 403 ttp\:
 RedirectMatch 403 \.\$url
 RedirectMatch 403 \/\$url
 RedirectMatch 403 \/\$link
 RedirectMatch 403 news\.php
 RedirectMatch 403 menu\.php
 RedirectMatch 403 main\.php
 RedirectMatch 403 home\.php
 RedirectMatch 403 view\.php
 RedirectMatch 403 about\.php
 RedirectMatch 403 blank\.php
 RedirectMatch 403 block\.php
 RedirectMatch 403 order\.php
 RedirectMatch 403 search\.php
 RedirectMatch 403 errors\.php
 RedirectMatch 403 button\.php
 RedirectMatch 403 middle\.php
 RedirectMatch 403 threads\.php
 RedirectMatch 403 contact\.php
 RedirectMatch 403 include\.php
 RedirectMatch 403 display\.php
 RedirectMatch 403 register\.php
 RedirectMatch 403 authorize\.php
 RedirectMatch 403 \/wp\-signup\.php
 RedirectMatch 403 \/classes\/
 RedirectMatch 403 \/includes\/
 RedirectMatch 403 \/path\_to\_script\/
 RedirectMatch 403 ImpEvData\.
 RedirectMatch 403 head\_auth\.
 RedirectMatch 403 db\_connect\.
 RedirectMatch 403 check\_proxy\.
 RedirectMatch 403 doeditconfig\.
 RedirectMatch 403 submit\_links\.
 RedirectMatch 403 change\_action\.
 RedirectMatch 403 send\_reminders\.
 RedirectMatch 403 comment\-template\.
 RedirectMatch 403 syntax\_highlight\.
 RedirectMatch 403 admin\_db\_utilities\.
 RedirectMatch 403 admin\.webring\.docs\.
 RedirectMatch 403 function\.main
 RedirectMatch 403 function\.mkdir
 RedirectMatch 403 function\.opendir
 RedirectMatch 403 function\.require
 RedirectMatch 403 function\.array\-rand
 RedirectMatch 403 ref\.outcontrol
</IfModule>

# PART II: QUERY STRINGS 
<ifmodule mod_rewrite.c>
 RewriteCond %{QUERY_STRING} ftp\:   [NC,OR]
 RewriteCond %{QUERY_STRING} http\:  [NC,OR]
 RewriteCond %{QUERY_STRING} https\: [NC,OR]
 RewriteCond %{QUERY_STRING} \[      [NC,OR]
 RewriteCond %{QUERY_STRING} \]      [NC]
 RewriteRule .* -                    [F,L]
</ifmodule>

# PART III: USER AGENTS
SetEnvIfNoCase User-Agent "Jakarta Commons" keep_out
SetEnvIfNoCase User-Agent "Y!OASIS/TEST"    keep_out
SetEnvIfNoCase User-Agent "libwww-perl"     keep_out
SetEnvIfNoCase User-Agent "MOT-MPx220"      keep_out
SetEnvIfNoCase User-Agent "MJ12bot"         keep_out
SetEnvIfNoCase User-Agent "Nutch"           keep_out
SetEnvIfNoCase User-Agent "cr4nk"           keep_out
<Limit GET POST PUT>
 order allow,deny
 allow from all
 deny from env=keep_out
</Limit>

# PART IV: IP ADDRESSES
<Limit GET POST PUT>
 order allow,deny
 allow from all
 deny from 75.126.85.215  "# blacklist candidate 2008-01-02 = admin-ajax.php attack "
 deny from 128.111.48.138 "# blacklist candidate 2008-02-10 = cryptic character strings "
 deny from 87.248.163.54  "# blacklist candidate 2008-03-09 = block administrative attacks "
 deny from 84.122.143.99  "# blacklist candidate 2008-04-27 = block clam store loser "
</Limit>

Installation and Usage

Before using the 3G Blacklist, check the following system requirements:

  • Linux server running Apache
  • Enabled Apache module: mod_alias
  • Enabled Apache module: mod_rewrite
  • Ability to edit your site’s root htaccess file (or)
  • Ability to modify Apache’s server configuration file

With these requirements met, copy and paste the entire 3G Blacklist into either the root htaccess file or the server configuration file. After uploading, visit your site and check proper loading of as many different types of pages as possible. For example, if you are running a blogging platform (such as WordPress), test different page views (single, archive, category, home, etc.), log into and surf the admin pages (plugins, themes, options, posts, etc.), and also check peripheral elements such as individual images, available downloads, and alternate protocols (FTP, HTTPS, etc.).

While the 3G Blacklist is designed to target only the bad guys, the regular expressions used in the list may interfere with legitimate URL access. If this happens, the browsing device will display a 403 Forbidden error. Don’t panic! Simply check the blocked URL, locate the matching blacklist string, and disable the directive by placing a pound sign ( # ) at the beginning of the associated line. Once the correct line is commented out, the blocked URL should load normally. Also, if you do happen to experience any conflicts involving the 3G Blacklist, please leave a comment or contact me directly. Thank you :)

Wrap Up..

As my readers know, I am serious about site security. Nothing gets my adrenaline pumping more than the thought of a giant meat grinder squirting out endless chunks of mangled cracker meat. Spam and other exploitative activity on the web has grown exponentially. Targeting and blocking individual agents and IP is no longer a viable strategy. By recognizing and immunizing against the broadest array of common attack elements, the 3G Blacklist maximizes resources while providing solid defense against malicious attacks.

Updates

Updates to the 3G Blacklist/firewall:

2008/05/14

Removed “RedirectMatch 403 \/scripts\/” from the first part of the blacklist due to conflict with Mint Statistics.

2008/05/18

Removed the following three directives to facilitate Joomla functionality:

RedirectMatch 403 \/modules\/
RedirectMatch 403 \/components\/
RedirectMatch 403 \/administrator\/

2008/05/31

Removed “RedirectMatch 403 config\.php” from the first part of the list to ensure proper functionality with the “visual-editing” feature of the WordPress Admin Area.

Jeff Starr
About the Author Jeff Starr = Web Developer. Security Specialist. WordPress Buff.
Archives
84 responses
  1. Jeff Starr

    @Tony: Not sure then. The issue you describe is unique — this is the first I am hearing of anything like this in nearly a year since the 3G Blacklist was released. Looking at the general format of your URLs, there doesn’t seem to be anything that inadvertently would be blocked. But even so, the possibility does exist that the list was somehow involved, perhaps by conflicting with another process or script used by the site or server. Definitely strange though, whatever happened.

  2. Jeff Starr

    @Alex: Hey I checked out that plugin and it looks like there is a file that is being blocked by the 3G Blacklist, namely htmlparser.inc. This should be resolved easily by commenting out the following line:

    RedirectMatch 403 \.inc

    That should do the trick. Let me know how it goes!

  3. Surely not every backslash here and in building-the-3g-blacklist-part-1 etc. are needed.

    I sense you have gone hog wild on the backslashes, for fear of the unknown.

  4. Jeff Starr

    I’m going to have to call your bluff on this one, jidanni — exactly which characters do not need to be escaped with a backslash?

  5. I spent ten whole minutes in emacs,
    (rgrep “backslash” “*” “/usr/share/doc/apache2-doc/manual/en/”)
    (rgrep “escape” “*” “/usr/share/doc/apache2-doc/manual/en/”)
    (w3m-goto-url “http://www.google.com/search?q=apache+backslash+escape” nil nil)
    but couldn’t dig up the exact list. So… you win\.

  6. Jeff Starr

    Nor could I, thus the reason for the overly cautious application of escape characters. “Better safe than sorry” is sound advice, especially when dealing with anything involving HTAccess. Nice try though. ;)

  7. OK, added Bug 46782 – list of what characters need to be escaped is hard to find
    https://issues.apache.org/bugzilla/show_bug.cgi?id=46782

  8. Jeff Starr

    Nice. Keep me posted — would be great to clean things up a bit..

  9. Well, as you can see on the bug, they chucked it in the garbage.

  10. Jeff Starr

    Yes, but I did notice their advice to escape regular expressions as such, which has been my modus operandi when dealing with HTAccess. At least they didn’t keep you waiting for a week before responding.

  11. OK, we find
    Apache uses Perl Compatible Regular Expressions provided by the PCRE library.
    which
    $ man perlre
    perlre – Perl regular expressions
    should tell the deal.

  12. Trenton Scott June 20, 2011 @ 12:59 pm

    We use this .htaccess script for all of our clients at Rainleader LLC. Fully compatible with Ubuntu 10.04 and the latest version of Apache2. Thanks for making this easy.

[ Comments are closed for this post ]