Latest Tweets404 Fix: Block Nuisance Requests for Non-Existent Files: perishablepress.com/block-nuis…
Perishable Press

The Perishable Press 4G Blacklist

[ 4G Stormtrooper ] At last! After many months of collecting data, crafting directives, and testing results, I am thrilled to announce the release of the 4G Blacklist! The 4G Blacklist is a next-generation protective firewall that secures your site against a wide range of automated attacks and other malicious activity.

Update: Check out the new and improved 6G Blacklist/Firewall »

Like its 3G predecessor, the 4G Blacklist is designed for use on Apache servers and is easily implemented via HTAccess or the httpd.conf configuration file. In order to function properly, the 4G Blacklist requires two specific Apache modules, mod_rewrite and mod_alias. As with the third generation of the blacklist, the 4G Blacklist consists of multiple parts:

  • HTAccess Essentials
  • Request-Method Filtering
  • IP Address Blacklist
  • Query-String Blacklist
  • URL Blacklist

Each of these methods is designed to protect different aspects of your site. They may be used independently, mixed and matched, or combined to create the complete 4G Blacklist. This modularity provides flexibility for different implementations while facilitating the testing and updating process. The core of the 4G Blacklist consists of the last two methods, the Query-String and URL Blacklists. These two sections provide an enormous amount of protection against many potentially devastating attacks. Everything else is just icing on the cake. Speaking of which, there are also two more completely optional sections of the 4G Blacklist, namely:

These two sections have been removed from the 4G Blacklist and relegated to “optional” status because they are no longer necessary. Put simply, the 4G Blacklist provides better protection with fewer lines of code. Even so, each of these blacklists have been updated with hundreds of new directives and will be made available here at Perishable Press in the near future. But for now, let’s return to the business at hand..

Presenting the Perishable Press 4G Blacklist

As is custom here at Perishable Press, I present the complete code first, and then walk through the usage instructions and code explanations. So, without furhter ado, here is the much-anticipated 4G Blacklist [for personal use only – may not be posted elsewhere without proper link attribution]:

### PERISHABLE PRESS 4G BLACKLIST ###

# ESSENTIALS
RewriteEngine on
ServerSignature Off
Options All -Indexes
Options +FollowSymLinks

# FILTER REQUEST METHODS
<IfModule mod_rewrite.c>
 RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK) [NC]
 RewriteRule ^(.*)$ - [F,L]
</IfModule>

# BLACKLIST CANDIDATES
<Limit GET POST PUT>
 Order Allow,Deny
 Allow from all
 Deny from 75.126.85.215   "# blacklist candidate 2008-01-02 = admin-ajax.php attack "
 Deny from 128.111.48.138  "# blacklist candidate 2008-02-10 = cryptic character strings "
 Deny from 87.248.163.54   "# blacklist candidate 2008-03-09 = block administrative attacks "
 Deny from 84.122.143.99   "# blacklist candidate 2008-04-27 = block clam store loser "
 Deny from 210.210.119.145 "# blacklist candidate 2008-05-31 = block _vpi.xml attacks "
 Deny from 66.74.199.125   "# blacklist candidate 2008-10-19 = block mindless spider running "
 Deny from 203.55.231.100  "# 1048 attacks in 60 minutes"
 Deny from 24.19.202.10    "# 1629 attacks in 90 minutes"
</Limit>

# QUERY STRING EXPLOITS
<IfModule mod_rewrite.c>
 RewriteCond %{QUERY_STRING} \.\.\/    [NC,OR]
 RewriteCond %{QUERY_STRING} boot\.ini [NC,OR]
 RewriteCond %{QUERY_STRING} tag\=     [NC,OR]
 RewriteCond %{QUERY_STRING} ftp\:     [NC,OR]
 RewriteCond %{QUERY_STRING} http\:    [NC,OR]
 RewriteCond %{QUERY_STRING} https\:   [NC,OR]
 RewriteCond %{QUERY_STRING} mosConfig [NC,OR]
 RewriteCond %{QUERY_STRING} ^.*(\[|\]|\(|\)|<|>|'|"|;|\?|\*).* [NC,OR]
 RewriteCond %{QUERY_STRING} ^.*(%22|%27|%3C|%3E|%5C|%7B|%7C).* [NC,OR]
 RewriteCond %{QUERY_STRING} ^.*(%0|%A|%B|%C|%D|%E|%F|127\.0).* [NC,OR]
 RewriteCond %{QUERY_STRING} ^.*(globals|encode|config|localhost|loopback).* [NC,OR]
 RewriteCond %{QUERY_STRING} ^.*(request|select|insert|union|declare|drop).* [NC]
 RewriteRule ^(.*)$ - [F,L]
</IfModule>

# CHARACTER STRINGS
<IfModule mod_alias.c>
 # BASIC CHARACTERS
 RedirectMatch 403 \,
 RedirectMatch 403 \:
 RedirectMatch 403 \;
 RedirectMatch 403 \=
 RedirectMatch 403 \@
 RedirectMatch 403 \[
 RedirectMatch 403 \]
 RedirectMatch 403 \^
 RedirectMatch 403 \`
 RedirectMatch 403 \{
 RedirectMatch 403 \}
 RedirectMatch 403 \~
 RedirectMatch 403 \"
 RedirectMatch 403 \$
 RedirectMatch 403 \<
 RedirectMatch 403 \>
 RedirectMatch 403 \|
 RedirectMatch 403 \.\.
 RedirectMatch 403 \/\/
 RedirectMatch 403 \%0
 RedirectMatch 403 \%A
 RedirectMatch 403 \%B
 RedirectMatch 403 \%C
 RedirectMatch 403 \%D
 RedirectMatch 403 \%E
 RedirectMatch 403 \%F
 RedirectMatch 403 \%22
 RedirectMatch 403 \%27
 RedirectMatch 403 \%28
 RedirectMatch 403 \%29
 RedirectMatch 403 \%3C
 RedirectMatch 403 \%3E
 RedirectMatch 403 \%3F
 RedirectMatch 403 \%5B
 RedirectMatch 403 \%5C
 RedirectMatch 403 \%5D
 RedirectMatch 403 \%7B
 RedirectMatch 403 \%7C
 RedirectMatch 403 \%7D
 # COMMON PATTERNS
 Redirectmatch 403 \_vpi
 RedirectMatch 403 \.inc
 Redirectmatch 403 xAou6
 Redirectmatch 403 db\_name
 Redirectmatch 403 select\(
 Redirectmatch 403 convert\(
 Redirectmatch 403 \/query\/
 RedirectMatch 403 ImpEvData
 Redirectmatch 403 \.XMLHTTP
 Redirectmatch 403 proxydeny
 RedirectMatch 403 function\.
 Redirectmatch 403 remoteFile
 Redirectmatch 403 servername
 Redirectmatch 403 \&rptmode\=
 Redirectmatch 403 sys\_cpanel
 RedirectMatch 403 db\_connect
 RedirectMatch 403 doeditconfig
 RedirectMatch 403 check\_proxy
 Redirectmatch 403 system\_user
 Redirectmatch 403 \/\(null\)\/
 Redirectmatch 403 clientrequest
 Redirectmatch 403 option\_value
 RedirectMatch 403 ref\.outcontrol
 # SPECIFIC EXPLOITS
 RedirectMatch 403 errors\.
 RedirectMatch 403 config\.
 RedirectMatch 403 include\.
 RedirectMatch 403 display\.
 RedirectMatch 403 register\.
 Redirectmatch 403 password\.
 RedirectMatch 403 maincore\.
 RedirectMatch 403 authorize\.
 Redirectmatch 403 macromates\.
 RedirectMatch 403 head\_auth\.
 RedirectMatch 403 submit\_links\.
 RedirectMatch 403 change\_action\.
 Redirectmatch 403 com\_facileforms\/
 RedirectMatch 403 admin\_db\_utilities\.
 RedirectMatch 403 admin\.webring\.docs\.
 Redirectmatch 403 Table\/Latest\/index\.
</IfModule>

That’s the juice right there. This 4G Blacklist is some powerful stuff, blocking and filtering a wide range of potential attacks and eliminating tons of malicious nonsense. Much care has been taken to beta test this firewall on multiple configurations running various types of software, however, due to my limited financial resources, it is impossible to test the 4G as comprehensively as I would have preferred. Even so, for the average site running typical software, everything should continue to work perfectly. With that in mind, please read through the remainder of the article before implementing the 4G Blacklist.

Installation and Usage

Before implementing the 4G Blacklist, ensure that you are equipped with the following system requirements:

  • Linux server running Apache
  • Enabled Apache module: mod_alias
  • Enabled Apache module: mod_rewrite
  • Ability to edit your site”s root htaccess file (or)
  • Ability to modify Apache’s server configuration file

With these requirements met, copy and paste the entire 4G Blacklist into either the root HTAccess file or the server configuration file ( httpd.conf ). After uploading, visit your site and check proper loading of as many different types of pages as possible. For example, if you are running a blogging platform (such as WordPress), test different page views (single, archive, category, home, etc.), log into and surf the admin pages (plugins, themes, options, posts, etc.), and also check peripheral elements such as individual images, available downloads, and alternate protocols (FTP, HTTPS, etc.).

While the 4G Blacklist is designed to target only the bad guys, the regular expressions used in the list may interfere with legitimate URL or file access. If the directives in the blacklist are blocking a specific URL, the browsing device will display a 403 Forbidden error; similarily, if the blacklist happens to block a file or resource required for some script to function properly, the script (JavaScript, PHP, etc.) may simply stop working. If you experience either of these scenarios after installing the blacklist, don’t panic! Simply check the blocked URL or file, locate the matching blacklist string, and disable the directive by placing a pound sign ( # ) at the beginning of the associated line. Once the correct line is commented out, the blocked URL should load normally. Also, if you do happen to experience any conflicts involving the 4G Blacklist, please leave a comment or contact me directly.

Set for Stun

As my readers know, I am serious about site security. Nothing gets my juices flowing like the thought of chopping up mindless cracker whores into small, square chunks and feeding their still-twitching flesh to a pack of starving mongrels. That’s good times, but unfortunately there are probably laws against stuff like that. So in the meantime, we take steps to secure our sites using the most effective tools at our disposal. There is no one single magic bullet that will keep the unscrupulous bastards from exploiting and damaging your site, but there are many cumulative steps that may be taken to form a solid security strategy. Within this cumulative context, the 4G Blacklist recognizes and immunizes against a broad array of common attack elements, thereby maximizing resources while providing solid defense against malicious attacks.

Many Thanks

A huge “Thank You” to the dedicated people who helped beta test the 4G Blacklist. Your excellent feedback played an instrumental role in the development of this version. Thank you!

Further Reading

For more insight into the mysterious realms of blacklisting, the creation of the Perishable Press Blacklist, and DIY site security in general, check out some of my other articles:

Next Up

Next up in the March 2009 Blacklist Series: The Ultimate User-Agent Blacklist. Don’t miss it!

Updates

Since the release of the 4G Blacklist, several users have discovered issues with the following 4G directives.

Joomla

In the query-string section, Joomla users should delete the following patterns:

request
config
[
]

In the character-string section, Joomla users should comment-out or delete the following lines:

RedirectMatch 403 \,
RedirectMatch 403 \;
RedirectMatch 403 config\.
RedirectMatch 403 register\.

WordPress

In the query-string section of the 4G Blacklist, the following changes have been made:

"%3D" character-string has been changed to "%5C"

Likewise, in the character-string section, the following change has been made:

"wp\_" character-string has been removed

And in the request-method filtering section, the following change has been made:

"HEAD" method has been removed

Also, the following changes may be necessary according to which plugins you have installed:

Ozh' Admin Drop Down Menu - remove "drop" from the query-string rules
WordPress' Akismet - remove "config" from the query-string rules

OpenID

OpenID users should read the information in this comment.

SMF

SMF users should read the information in this comment.

vBulletin

vBulletin users should read the information in these comments.

Jeff Starr
About the Author Jeff Starr = Fullstack Developer. Book Author. Teacher. Human Being.
Archives
233 responses
  1. Hi, i used the list, broke it into two as directed, made the joomla adjustments, took out a few yahoo and google exceptions, and it seemed to work fine. That’s in IE8, FF, Opera etc.

    However, two weeks down the track I happened to test the site in IE7, and it came up with a 403 forbidden access error. No authority to access this server … error while attempting 500.

    This only happened in IE7.

    As mine is a tourist portal, ranking no.1 in google, it was important the site was operational, so i’ve removed the entire list for now and it works in IE7.

    I’m wondering what the issue could be, any thoughts anyone?

  2. Garrett W. April 19, 2010 @ 8:45 am

    Tell you what, Igor — how about you try it out and let us know if there are any problems — that way Jeff will have that knowledge for his 5G list ;)

  3. Does anyone know whether the 4G list plays nice with Phorum, an open-source message board system?

    I would like to thank Jeff for his useful, well-written and informative site. I refer to this site often and consider it one of the best for design tips for intermediate level or above.

  4. Hi Garret, will do! I am getting ready to test it out myself, because I think the potential benefits may outweigh the potential problems. I was also interested to learn about Project Honeypot, mentioned above. Thanks and I will be back to learn more and share whatever I discover in my experiment, but it may be a week or so…

  5. This seems is a GREAT blacklist for security. However, as someone who has no understanding of code or website stuff aside from simply setting them up…how can I implement this?

    It seems like you really have to customize your backlist to your site, which requires extensive understanding of which lines of codes need to be taken out, edited, etc.?

    I wouldn’t be able to do that because I don’t understand any of it…so how can I install this blacklist easily?

    Thanks!

  6. @Peter, yes, you do need to do some customisation for your site.

    Go thru all the posts above, and make notes on what’s relevant for you as you go.

    Most important is to take the advice, in an early post, about splitting the list into two lines, instead of one.

    Other than that it’s easy, just literally copy and paste the code into your .htaccess file and see what happens.

    I still havent solved my issue of IE7 displaying error page, and yet it works in all other browsers and versions.

    good luck
    Shane

  7. After installation, I detected sluggishness in the server response time, possibly due to the fat .htaccess which slows things down a bit.

    In Phorum, I received 403 errors trying to log off and trying to register, due in one case to rejection of ?0 and in another case due to rejection of /register something or other. For this reason, I do not recommend 4G for other admins that host forum software of any kind. I deleted entire paragraphs in 4G rather than single lines, because there is the chance that there may be other “gotchas,” and I am working with a live site. So I wound up gutting the 4G blacklist. What I have remaining is this, which is little different from what I had before:

    # ESSENTIALS
    RewriteEngine on
    ServerSignature Off
    Options -Indexes

    ### Note: good stuff, all, though I already had this in place before.

    # FILTER REQUEST METHODS
    RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK) [NC]
    RewriteRule ^(.*)$ - [F,L]

    ### I did not detect any problems from this, but on the other hand I have not found these methods being used in my system log either, and I have yet to read any explanation as to why these methods need to be filtered. I wonder too if filtering them will interfere with the site statistics utility. I suspect that is “highly likely” based upon my experience, so these too will be removed.

    # BLACKLIST CANDIDATES
    Order Allow,Deny
    Allow from all
    Deny from 75.126.85.215
    Deny from 128.111.48.138
    Deny from 87.248.163.54
    Deny from 84.122.143.99
    Deny from 210.210.119.145
    Deny from 66.74.199.125
    Deny from 203.55.231.100
    Deny from 24.19.202.10

    ### No problem here either. But IP candidates seem dubious as they are two years old anyway. There are many black hats and they use multiple IP addresses. Black hats do not generally like to be pinned down to a few IPs, though it would certainly be nice if they would agree to that.

    On my host, causes an internal server error (500) and had to be removed. It is not necessary anyway as my host supports these modules.

    btw: a bit of documentation as to the reasoning for each of the various filters would be most helpful.

  8. Jeff Starr

    @Igor: Thanks for the input. Here is a full explanation of the research and logic behind the 4G Blacklist:

    Building the Perishable Press 4G Blacklist

  9. Hey Jeff, What are your thoughts on this technique for speeding up htaccess?
    http://www.chicagostyleseo.com/2010/02/speed-up-the-rewrite-engine-for-wordpress/

    “Notably, the unnecessary and potentially-problematic container is completely removed… “

  10. Thanks Jeff. I wanted to emphasize again, your site has proven very useful to me in terms of configuring my .htaccess file and learning about CSS. You are doing a great service to the community here.

    btw, someone has stolen your content:
    http://www.SPAMSITEkingf1SPAMSITE.com/2010/03/21/stupid-htaccess-tricks.

    Remove SPAMSITE to access. That appeared just below your site in Google results, and I clicked on it by accident because it seemed to have a more recent date.

    Here is the bot-banning portion of my .htaccess, derived in part from your suggestions.

    # FILTER REQUEST METHODS
    #
    #I kept these in anyway because they do not appear to cause any harm
    #
    RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK) [NC]
    RewriteRule ^(.*)$ - [F,L]

    # BLACKLIST CANDIDATES
    #
    # These I detected manually by analyzing my log file and
    # referencing other sources on the web
    #

    Order Allow,Deny
    Allow from all
    Deny from 91.205.96.19 "#2010-04-22: www.SPAMSITEpuritysearchSPAMSITE.net
    Deny from 81.52.143. "#voila
    Deny from 193.252.149. "#voila
    Deny from 78.26.187.62 "#ukraine bot?
    Deny from 89.149.254.73 "#Turkish bot?
    Deny from 66.33.235.24 "#port scanner
    Deny from 89.122.29. "#too many javabots

    # BLACKLISTED USER AGENTS
    #
    # The first two are my choices, the rest yours, but
    # some of your choices were detected in my log file
    # as well. In particular, Nutch, Jakarta are common.
    # I get thousands of hits from Voilabot, but no visitors!
    #
    SetEnvIfNoCase User-Agent "VoilaBot" keep_out
    SetEnvIfNoCase User-Agent ^Java keep_out
    SetEnvIfNoCase User-Agent ^$ keep_out
    SetEnvIfNoCase User-Agent "Y!OASIS/TEST" keep_out
    SetEnvIfNoCase User-Agent "libwww-perl" keep_out
    SetEnvIfNoCase User-Agent "Jakarta.Commons" keep_out
    SetEnvIfNoCase User-Agent "MJ12bot" keep_out
    SetEnvIfNoCase User-Agent "Nutch" keep_out
    SetEnvIfNoCase User-Agent "cr4nk" keep_out
    SetEnvIfNoCase User-Agent "MOT-MPx220" keep_out
    SetEnvIfNoCase User-Agent "SiteCrawler" keep_out
    SetEnvIfNoCase User-Agent "SiteSucker" keep_out
    SetEnvIfNoCase User-Agent "Doubanbot" keep_out
    SetEnvIfNoCase User-Agent "Sogou" keep_out

    Order Allow,Deny
    Allow from all
    Deny from env=keep_out

    I get some activity from Yodaobot, TurnItIn (a bot supposedly checking for student plagiarism), and ia_archiver, too, but haven’t banned them yet. I know ia_archiver is a non-profit, but I’m not too thrilled with their purpose, as I don’t necessarily want old pages being stored somewhere “for research purposes”.

  11. Jeff Starr

    @Igor: Thanks for the scraper tip! And for sharing your version of the blacklist. Looks awesome :)

  12. iambic5p May 4, 2010 @ 5:39 am

    We’re really finding this useful, stunning work, thanks for sharing…

    So far no problems except it seems to have killed stone dead our Mint Stats, so far I’ve tracked it down to this line…

    RewriteCond %{QUERY_STRING} ^.*(globals|encode|config|localhost|loopback).* [NC,OR]

    …and removed “encode” needed for the end of the query string mint produces. Actually Mint requires “encoded” is there a way to allow “encoded” through and prevent “encode”?

[ Comments are closed for this post ]