Latest TweetsVerify any search engine or visitor via CLI Forward-Reverse Lookup perishablepress.com/cli-forwar…
Perishable Press

Building the 3G Blacklist, Part 4: Improving RedirectMatch in the Original 2G Blacklist

[ 3G Stormtroopers (Team Aqua) ]

In this continuing five-article series, I share insights and discoveries concerning website security and protecting against malicious attacks. In this fourth article, I build upon previous ideas and techniques by improving the directives contained in the original 2G Blacklist. Subsequent articles will focus on key blacklist strategies designed to protect your site transparently, effectively, and efficiently. At the conclusion of the series, the five articles will culminate in the release of the next generation 3G Blacklist.

Improving RedirectMatch in the Original 2G Blacklist

In the first version (2G) of the next-generation blacklist, a collection of malicious attack strings were compiled from access logs, blocked via Apache’s RedirectMatch directive, tested for effectiveness and transparency (i.e., non-interference), and finally converted into the 2G Blacklist. This 2G Blacklist continues to serve as my primary line of defense here at Perishable Press, replacing several other lists, including a million individual IP addresses and even the Ultimate htaccess Blacklist itself. So far, the results have extraordinary.

Of course, as discussed in Building the 3G Blacklist, Part 3, static, “fix-it-and-forget-it”-type strategies ultimately fail when dealing with the perpetually evolving nature of the enemy (i.e., malicious attackers, crackers, and other scumbags). Thus, as stated in the original 2G article, the blacklist is a work in progress, designed for periodic updates as new attack patterns and trends are identified. In the remainder of this article, we consider various aspects of the previous blacklist and consider various improvements.

Note that it is not necessary to assemble and transcribe any of the new blacklist directives presented herein. At the conclusion of this series, each of the different strategies will be combined into a single, comprehensive security strategy: The 3G Blacklist!

Miscellaneous Character Strings

Besides query strings, miscellaneous, random, and unusual character strings are perhaps the most frequently observed characteristics of common server attacks. For example, each of these sequences is a common sight to those of us concerned with site traffic and security:

  • /,
  • //
  • f-.
  • ...

Yet these strings are missing from the previous version of the blacklist. In the new version, we block these character strings, as well as several others:

  • :
  • ;
  • <
  • >
  • .inc
  • alt=
  • ftp:
  • ttp:
  • .$url
  • /$url
  • /$link

This is powerful stuff. Any URL request containing one (or more) of these character strings will be denied via an immediate 403 Forbidden error. Attackers frequently target server-side includes (.inc), which are protected here in the fifth line. Inclusion of various hypertext protocols are also common, and are blocked here in the third and fourth lines. Another common attack vector involves the ubiquitous $url variable (often recorded as \\\'.$url.\\\'), which is also protected in this set in the last two lines. Compared with the equivalent section of the previous blacklist,

  • .inc
  • alt=
  • http://

..the new version is much improved, covering a far more expansive field of vulnerability using only a few additional lines.

PHP File Types

In the previous blacklist, common PHP file targets were added to the list in ”shotgun”-like fashion. While sorting out the madness, readers pointed out several conflicts related to the blacklist:

  • file.php — interferes with WordPress administration
  • port.php — interferes with WordPress administration
  • index2.php — interferes with Joomla backend [ reported by Don ]

In addition to correcting these discrepancies, PHP-related expressions that are highly specific to a particular platform or context (e.g., admin.webring.docs.php) have been generalized to match any file type. Conversely, PHP-related expressions that are more general in application (e.g., about.php) have been left unaltered. Further, several new PHP file names have been added, some have been replaced, and some have been removed.

Here is the PHP-related section as it appears in the previous, 2G Blacklist:

redirectmatch 403 menu\.php
redirectmatch 403 main\.php
redirectmatch 403 file\.php
redirectmatch 403 home\.php
redirectmatch 403 view\.php
redirectmatch 403 about\.php
redirectmatch 403 order\.php
redirectmatch 403 index2\.php
redirectmatch 403 errors\.php
redirectmatch 403 config\.php
redirectmatch 403 button\.php
redirectmatch 403 middle\.php
redirectmatch 403 threads\.php
redirectmatch 403 contact\.php
redirectmatch 403 display\.cgi
redirectmatch 403 display\.php
redirectmatch 403 include\.php
redirectmatch 403 register\.php
redirectmatch 403 db_connect\.php
redirectmatch 403 doeditconfig\.php
redirectmatch 403 send\_reminders\.php 
redirectmatch 403 admin_db_utilities\.php
redirectmatch 403 admin\.webring\.docs\.php

And likewise, here is the refined PHP-related section as it appears in the new, 3G Blacklist:

RedirectMatch 403 news\.php
RedirectMatch 403 menu\.php
RedirectMatch 403 main\.php 
RedirectMatch 403 home\.php
RedirectMatch 403 view\.php
RedirectMatch 403 about\.php
RedirectMatch 403 blank\.php
RedirectMatch 403 block\.php
RedirectMatch 403 order\.php
RedirectMatch 403 search\.php
RedirectMatch 403 errors\.php
RedirectMatch 403 config\.php
RedirectMatch 403 button\.php
RedirectMatch 403 middle\.php
RedirectMatch 403 threads\.php
RedirectMatch 403 contact\.php
RedirectMatch 403 include\.php
RedirectMatch 403 display\.php
RedirectMatch 403 register\.php
RedirectMatch 403 authorize\.php
RedirectMatch 403 \/wp\-signup\.php

Directory Path Segments

Also new in the 3G Blacklist are several common directory path segments. These directory names appear frequently in malicious attacks, even though they have no business being requested directly via http protocol, especially by someone trying to hack your site. ;)

Here are the newly added directory path segments:

RedirectMatch 403 \/scripts\/
RedirectMatch 403 \/modules\/
RedirectMatch 403 \/classes\/
RedirectMatch 403 \/includes\/
RedirectMatch 403 \/components\/
RedirectMatch 403 \/administrator\/
RedirectMatch 403 \/path\_to\_script\/

Certainly straightforward. The \/ on either side of these names ensure that only directories are matched. Including these terms in post titles is completely safe. As an aside, I may end up removing \/path\_to\_script\/ in the 3G Blacklist. If you think it should remain, drop a comment and explain. Likewise, if you see potential conflicts with any of the other expressions, please let me know.

Miscellaneous Functions

Another commonly observed component of malicious attacks includes a variety of miscellaneous functions appended onto the end of URLs. The previous, 2G Blacklist does not include such functions, but I finally got sick of seeing the little bastards squirming their way into my access logs. Thus, the new blacklist blocks any of the following:

RedirectMatch 403 function\.main
RedirectMatch 403 function\.mkdir
RedirectMatch 403 function\.opendir
RedirectMatch 403 function\.require
RedirectMatch 403 function\.array\-rand
RedirectMatch 403 ref\.outcontrol

These directives are my favorite part of the new list. If you know of other, similar functions commonly used for exploits, please share them so that they may added to the complete blacklist.

Omissions and Other Changes

Other potential improvements to the previous, 2G Blacklist provide protection for users running a vBulletin Board by blocking the following strings:

  • f-\.html
  • ImpEvData\.php
  • check_proxy\.html

These rules are recommended by Peter and have been assimilated into 3G.

Lastly, I will mention the removal of eleven esoteric, randomly generated character strings that are no longer necessary. These were most likely one-time vulnerability probes no longer in service. Here are the dropped entries:

redirectmatch 403 keds\.lpti
redirectmatch 403 r\.verees
redirectmatch 403 pictureofmyself
redirectmatch 403 remoteFile
redirectmatch 403 mybabyboy
redirectmatch 403 mariostar
redirectmatch 403 zaperyan
redirectmatch 403 babyboy
redirectmatch 403 aboutme
redirectmatch 403 xAou6
redirectmatch 403 qymux

And finally, the redirectmatch directives as written in the 2G Blacklist have been rewritten to adhere to accepted standards:

  • OLD: redirectmacth
  • NEW: RedirectMacth

A subtle improvement, perhaps, but an improvement nonetheless!

Wrap Up..

With this “sneak peak” into the rules comprising the upcoming 3G Blacklist, we enjoy the opportunity to suggest changes, add directives, and improve the overall strategy before the final release. The changes discussed here represent a majority of the improvements made involving RedirectMatch directives, however there remain several other changes that will be included in the upcoming release of the 3G Blacklist. Also, if you see any potential issues or conflicts (or worse) involving any of the code presented here, please speak up and leave a comment. Thank you!

Next..

Stay tuned for the conclusion of the Building the 3G Blacklist series, Part 5: Improving Site Security by Selectively Blocking Individual IPs. If you have yet to do so, I highly encourage you to subscribe to Perishable Press. As always, thank you for your generous attention.

Jeff Starr
About the Author Jeff Starr = Creative thinker. Passionate about free and open Web.
Archives
4 responses
  1. As always you’ve managed to deliver exciting and refreshing information. Thank you.

    I’ve only just recently began reading your articles on the 2G and 3G Blacklists and am not really qualified to talk about Apache/.htaccess rules etc.

    However, I wondered whether you’ve had to re-consider your approach to the Blacklists in relation to Apache performance, with the long .htaccess files.

    I’m definitely not implying that performance is superior to security, and I’d just be interested to hear about your findings on the performance changes you’ve noticed, if any.

    Kind Regards,

    Sat.

  2. Jeff Starr

    Hi Sat :) Thank you for the great feedback. As discussed in part 3 of the series, I have found that excessively large blacklists will in fact reduce the performance of your site. Of course, the degree to which this occurs depends on many environmental factors, such as available server memory and the amount of traffic you happen to be receiving. In other words, a site with excellent performance without insanely large htaccess files will probably still do well with them. Conversely, sites with mediocre or poor performance to begin with should probably try to keep the amount of htaccess processing to a minimum.

    Personally, I am running this site on a shared server that usually performs well, however I have been doing everything possible to maximize performance. In the process of optimizing, I did notice a improvement in page loading times after I had removed several large blacklists that had been piling up. I enjoyed the performance boost so much that I began to develop alternate methods of blacklisting scumbags that don’t require hundreds upon hundreds of htaccess operations.

  3. Discovered a Bug??
    Im way over my head…
    Far from an expert….
    with a bit of trial and error got
    RedirectMatch 403 // good
    BUT using phpnuke, the avatar on the front home page login disappeared yet appeared everywhere else
    I found the url to the avatar for some reason had a // in it
    Since the hack stuff is =http:// changing the code to
    RedirectMatch 403 ://
    fixed it
    ALSO have come across another hack form
    /modules.php?set_albumName=http%3A%2F%2Fwww.winbd.net%2Fadmin etc etc
    =http% being the common denomiator

  4. Jeff Starr

    Hi Steptoe,
    Nice work! I think changing the \/\/ to \:\/\/ as you have done is an excellent solution. Granted, a few invalid requests containing the double forward slashes may get through, but the majority of them will still be stopped cold. Even so, I would be interested in seeing an example of the phpNuke URL string that contains the double slashes. Such character sequences are commonly employed in query strings, but to place them in the root portion of the URL seems odd..

    As for the encoded transfer protocol, that is indeed a good common denominator. Currently, the query-string section of the 3G Blacklist blocks any URL containing http:, which accounts for a large majority of attempted exploits. However, I have also seen a few URL-encoded versions slip through (i.e., http%3A, etc.). I think blocking these URL-encoded query-string protocol requests is an excellent idea. I will be adding the following directive to the second section (query strings) of the next version of the 3G Blacklist:

    RewriteCond %{QUERY_STRING} http\%3A [NC,OR]

    Thanks for the comment — your help is greatly appreciated!

[ Comments are closed for this post ]