Perishable Press 3G Blacklist

♦ Posted by Jeff Starr in .htaccess, Security

Updated January 24, 2019 • 84 comments

After much research and discussion, I have developed a concise, lightweight security strategy for Apache-powered websites. Prior to the development of this strategy, I relied on several extensive blacklists to protect my sites against malicious user agents and IP addresses.

Over time, these mega-lists became unmanageable and ineffective. As increasing numbers of attacks hit my server, I began developing new techniques for defending against external threats. This work soon culminated in the release of a “next-generation” blacklist that works by targeting common elements of decentralized server attacks.

Consisting of a mere 37 lines, this “2G” Blacklist provided enough protection to enable me to completely eliminate over 350 blacklisting directives from my site’s root htaccess file. This improvement increased site performance and decreased attack rates, however many bad hits were still getting through. More work was needed..

The 3G Blacklist

Work on the 3G Blacklist required several weeks of research, testing, and analysis. During the development process, five major improvements were discovered, documented, and implemented. Using pattern recognition, access immunization, and multiple layers of protection, the 3G Blacklist serves as an extremely effective security strategy for preventing a vast majority of common exploits. The list consists of four distinct parts, providing multiple layers of protection while synergizing into a comprehensive defense mechanism. Further, as discussed in previous articles, the 3G Blacklist is designed to be as lightweight and flexible as possible, thereby facilitating periodic cultivation and maintenance. Sound good? Here it is:

# PERISHABLE PRESS 3G BLACKLIST

# PART I: CHARACTER STRINGS
<IfModule mod_alias.c>
 RedirectMatch 403 \:
 RedirectMatch 403 \;
 RedirectMatch 403 \<
 RedirectMatch 403 \>
 RedirectMatch 403 \/\,
 RedirectMatch 403 \/\/
 RedirectMatch 403 f\-\.
 RedirectMatch 403 \.\.\.
 RedirectMatch 403 \.inc
 RedirectMatch 403 alt\=
 RedirectMatch 403 ftp\:
 RedirectMatch 403 ttp\:
 RedirectMatch 403 \.\$url
 RedirectMatch 403 \/\$url
 RedirectMatch 403 \/\$link
 RedirectMatch 403 news\.php
 RedirectMatch 403 menu\.php
 RedirectMatch 403 main\.php
 RedirectMatch 403 home\.php
 RedirectMatch 403 view\.php
 RedirectMatch 403 about\.php
 RedirectMatch 403 blank\.php
 RedirectMatch 403 block\.php
 RedirectMatch 403 order\.php
 RedirectMatch 403 search\.php
 RedirectMatch 403 errors\.php
 RedirectMatch 403 button\.php
 RedirectMatch 403 middle\.php
 RedirectMatch 403 threads\.php
 RedirectMatch 403 contact\.php
 RedirectMatch 403 include\.php
 RedirectMatch 403 display\.php
 RedirectMatch 403 register\.php
 RedirectMatch 403 authorize\.php
 RedirectMatch 403 \/wp\-signup\.php
 RedirectMatch 403 \/classes\/
 RedirectMatch 403 \/includes\/
 RedirectMatch 403 \/path\_to\_script\/
 RedirectMatch 403 ImpEvData\.
 RedirectMatch 403 head\_auth\.
 RedirectMatch 403 db\_connect\.
 RedirectMatch 403 check\_proxy\.
 RedirectMatch 403 doeditconfig\.
 RedirectMatch 403 submit\_links\.
 RedirectMatch 403 change\_action\.
 RedirectMatch 403 send\_reminders\.
 RedirectMatch 403 comment\-template\.
 RedirectMatch 403 syntax\_highlight\.
 RedirectMatch 403 admin\_db\_utilities\.
 RedirectMatch 403 admin\.webring\.docs\.
 RedirectMatch 403 function\.main
 RedirectMatch 403 function\.mkdir
 RedirectMatch 403 function\.opendir
 RedirectMatch 403 function\.require
 RedirectMatch 403 function\.array\-rand
 RedirectMatch 403 ref\.outcontrol
</IfModule>

# PART II: QUERY STRINGS 
<ifmodule mod_rewrite.c>
 RewriteCond %{QUERY_STRING} ftp\:   [NC,OR]
 RewriteCond %{QUERY_STRING} http\:  [NC,OR]
 RewriteCond %{QUERY_STRING} https\: [NC,OR]
 RewriteCond %{QUERY_STRING} \[      [NC,OR]
 RewriteCond %{QUERY_STRING} \]      [NC]
 RewriteRule .* -                    [F,L]
</ifmodule>

# PART III: USER AGENTS
SetEnvIfNoCase User-Agent "Jakarta Commons" keep_out
SetEnvIfNoCase User-Agent "Y!OASIS/TEST"    keep_out
SetEnvIfNoCase User-Agent "libwww-perl"     keep_out
SetEnvIfNoCase User-Agent "MOT-MPx220"      keep_out
SetEnvIfNoCase User-Agent "MJ12bot"         keep_out
SetEnvIfNoCase User-Agent "Nutch"           keep_out
SetEnvIfNoCase User-Agent "cr4nk"           keep_out
<Limit GET POST PUT>
 order allow,deny
 allow from all
 deny from env=keep_out
</Limit>

# PART IV: IP ADDRESSES
<Limit GET POST PUT>
 order allow,deny
 allow from all
 deny from 75.126.85.215  "# blacklist candidate 2008-01-02 = admin-ajax.php attack "
 deny from 128.111.48.138 "# blacklist candidate 2008-02-10 = cryptic character strings "
 deny from 87.248.163.54  "# blacklist candidate 2008-03-09 = block administrative attacks "
 deny from 84.122.143.99  "# blacklist candidate 2008-04-27 = block clam store loser "
</Limit>

Installation and Usage

Before using the 3G Blacklist, check the following system requirements:

Linux server running Apache
Enabled Apache module: mod_alias
Enabled Apache module: mod_rewrite
Ability to edit your site’s root htaccess file (or)
Ability to modify Apache’s server configuration file

With these requirements met, copy and paste the entire 3G Blacklist into either the root htaccess file or the server configuration file. After uploading, visit your site and check proper loading of as many different types of pages as possible. For example, if you are running a blogging platform (such as WordPress), test different page views (single, archive, category, home, etc.), log into and surf the admin pages (plugins, themes, options, posts, etc.), and also check peripheral elements such as individual images, available downloads, and alternate protocols (FTP, HTTPS, etc.).

While the 3G Blacklist is designed to target only the bad guys, the regular expressions used in the list may interfere with legitimate URL access. If this happens, the browsing device will display a 403 Forbidden error. Don’t panic! Simply check the blocked URL, locate the matching blacklist string, and disable the directive by placing a pound sign ( # ) at the beginning of the associated line. Once the correct line is commented out, the blocked URL should load normally. Also, if you do happen to experience any conflicts involving the 3G Blacklist, please leave a comment or contact me directly. Thank you :)

Wrap Up..

As my readers know, I am serious about site security. Nothing gets my adrenaline pumping more than the thought of a giant meat grinder squirting out endless chunks of mangled cracker meat. Spam and other exploitative activity on the web has grown exponentially. Targeting and blocking individual agents and IP is no longer a viable strategy. By recognizing and immunizing against the broadest array of common attack elements, the 3G Blacklist maximizes resources while providing solid defense against malicious attacks.

Updates

Updates to the 3G Blacklist/firewall:

2008/05/14

Removed “RedirectMatch 403 \/scripts\/” from the first part of the blacklist due to conflict with Mint Statistics.

2008/05/18

Removed the following three directives to facilitate Joomla functionality:

RedirectMatch 403 \/modules\/
RedirectMatch 403 \/components\/
RedirectMatch 403 \/administrator\/

2008/05/31

Removed “RedirectMatch 403 config\.php” from the first part of the list to ensure proper functionality with the “visual-editing” feature of the WordPress Admin Area.

About the Author

Jeff Starr = Creative thinker. Passionate about free and open Web.

84 responses to “Perishable Press 3G Blacklist”

AXZM 2008/11/24 6:41 am

I can’t say enough good things about your work. I have had countless websites hacked over the years, some my fault, some the hosting provider, some the middleware I was using but in all cases if I had taken the right precautions in my .htaccess, I would have saved myself a lot of time and trouble. Thanks again…

Jeff Starr 2008/11/27 3:48 pm • Post Author

@Michel: My pleasure.. thanks for the heads up on the db.php vector; l will be implementing it into the next (4G) version of the blacklist. As you mention, the kombi.txt URL is blocked via query-string character match against http:, so no worries there. Ossec is another good call, of course. Thanks for the tips!

@AXZM: Thanks for the feedback! It is always good to hear positive reports about the blacklist. Also, as previously mentioned, I am working on the next version of the blacklist, which will protect against a broader range of attacks using optimized directives and improved overall performance. Stay tuned..

Tony 2008/11/27 5:55 pm

I just added the 3G Blacklist to my .htaccess, thanks for the wonderful tips you’re sharing. I’ve been implementing your various tips the last few days, and the coldform too, that I’m starting to feel a little guilty, hehe.

I guess the “RedirectMatch 403 \/\/” isn’t conflicting with WP Super Cache anymore. Everything is working great on my blog.

Thanks!

PS: I wanted to be sure there are no problems with the list before submitting the comment, and found out that validator.w3.org can’t access my page, and neither can whatsmyip.org/mod_gzip_test/ (was using it to check if Super Cache is working). I guess these services just append our URL to the end of theirs, and the result is a double forward slash // , which conflicts, again, with this line:

RedirectMatch 403 \/\/

I guess you knew about this… Anyway we can comment it out when we use these services.
I don’t know what the line means, but I guess it’s not that important since you’re not using it. The validator could access perishablepress.com, and found one error. I’ll report you to the code police now!

I’m joking, don’t worry. I won’t tell them.

Jeff Starr 2008/11/30 3:21 pm • Post Author

@Tony: Glad to hear you are getting the most out of my site!That’s what it’s here for, so I am certainly glad you find the content useful.

That is good news that WP Super Cache is no longer conflicting with the directives in the blacklist. I had heard that they recently updated the plugin, and so it looks as if they must have fixed the issue referred to here. Great news :)

As for the various validators, they should still be able to access your pages. The RedirectMatch 403 // directive is actually targeting the main part of the URL itself and does not block anything in the query string. Additionally, the blacklist is only blocking requests for URLs that are a part of your domain; their appending your page URL to their URL would only be affected if they were using the blacklist on their own domain (which I’m sure they’re not). Nonetheless, I have seen this issue before with the directives used in the Ultimate htaccess Blacklist, so you may want to check that possibility as well.

Fred 2008/12/02 11:35 am

Hi, I followed a link to this article and have been reading up on the 3G Blacklist. I copied the file and paste it first in my configuration but the server won’t load so I tried it in the .htaccess and get Forbidden: Access Permission Denied.

I then start removing lines.
RedirectMatch 403
Only when I remove these two line the site loads but still will not work from my configuration file.

I’m new to all of this. Any suggesstions and is it OK to leave these out?

Thanks

Fred

Jeff Starr 2008/12/03 3:42 pm • Post Author

@Fred: By all means, remove any lines that cause your server to crash. Unfortunately, when it comes to blacklisting, it is practically impossible to forge a “one-size-fits-all” strategy. For the most part, the blacklist works great out of the box, but various server/site configurations may require some fine-tuning. To do so, simply comment out or remove the offending lines and you should be good to go. Each of the different RedirectMatch directives operates independently, so that removing any of them simply disables the particular character-string match for which they are intended to block. In other words, the blacklist will still protect against all types of attacks represented by the remaining directives.

Fred 2008/12/04 9:25 pm

Jeff, thanks for the response and I’m pleased for the comments. I have couple of sites running off the same server and these bots really over load my server with the frequency at which they load pages. Because I have access to root, I’m able to httpd.reload and free resources up. For the next few days I will closely monitor to see the results.

Again, Many Thanks

Fred

Tony 2008/12/06 11:59 am

Sorry for the delay, it’s been a very busy week!

I found out that even though I could access my blog and everything is working fine, the validators couldn’t access it because of “RedirectMatch 403 \/\/” when WP Super Cache is enabled. If any one of them is disabled, then everything works fine.

Thanks again Jeff for this great list!

Sebastian 2008/12/08 5:16 am

Thanks for your work.

For Joomla 1.0.15 I needed to disable:
# RedirectMatch 403 \/\/
# RedirectMatch 403 \/includes\/

First rule is very site specific as tinyMCE inserted images with a double //
Second rule had prevented loading of some of the js scripts needed for administration. (eg. Save / Close links not working).

Jeff Starr 2008/12/08 9:46 pm • Post Author

@Sebastian: Thank you — this information will be integrated into the next version of the blacklist, which hopefully will be bubbling its way to the surface here very soon..

Otto 2008/12/13 4:32 am

Jeff, thanks for sharing the 3G list. One thing I noticed is that the character strings group blocked some of my pages. The 3G lines in question were:

# RedirectMatch 403 news.php
# RedirectMatch 403 contact.php

I am probably not the only one who named the pages on a more “traditional way”, such as contact.php. Any reason why these pages should be blocked?
Otto

Jeff Starr 2008/12/15 10:06 pm • Post Author

@Otto: Of course, comment out or remove any lines that prevent access to actual pages on your site. The two pages you mention are are very common and thus frequently targeted by attackers. If the pages actually exist, the number of misdirected site errors will decrease, however variations on the file names in question will persist.

Comments are closed for this post. Something to add? Let me know.

AXZM 2008/11/24 6:41 am

I can’t say enough good things about your work. I have had countless websites hacked over the years, some my fault, some the hosting provider, some the middleware I was using but in all cases if I had taken the right precautions in my .htaccess, I would have saved myself a lot of time and trouble. Thanks again…
Jeff Starr 2008/11/27 3:48 pm • Post Author

@Michel: My pleasure.. thanks for the heads up on the db.php vector; l will be implementing it into the next (4G) version of the blacklist. As you mention, the kombi.txt URL is blocked via query-string character match against http:, so no worries there. Ossec is another good call, of course. Thanks for the tips!

@AXZM: Thanks for the feedback! It is always good to hear positive reports about the blacklist. Also, as previously mentioned, I am working on the next version of the blacklist, which will protect against a broader range of attacks using optimized directives and improved overall performance. Stay tuned..
Tony 2008/11/27 5:55 pm

I just added the 3G Blacklist to my .htaccess, thanks for the wonderful tips you’re sharing. I’ve been implementing your various tips the last few days, and the coldform too, that I’m starting to feel a little guilty, hehe.

I guess the “RedirectMatch 403 \/\/” isn’t conflicting with WP Super Cache anymore. Everything is working great on my blog.

Thanks!

PS: I wanted to be sure there are no problems with the list before submitting the comment, and found out that validator.w3.org can’t access my page, and neither can whatsmyip.org/mod_gzip_test/ (was using it to check if Super Cache is working). I guess these services just append our URL to the end of theirs, and the result is a double forward slash // , which conflicts, again, with this line:

RedirectMatch 403 \/\/

I guess you knew about this… Anyway we can comment it out when we use these services.
I don’t know what the line means, but I guess it’s not that important since you’re not using it. The validator could access perishablepress.com, and found one error. I’ll report you to the code police now!

I’m joking, don’t worry. I won’t tell them.
Jeff Starr 2008/11/30 3:21 pm • Post Author

@Tony: Glad to hear you are getting the most out of my site!That’s what it’s here for, so I am certainly glad you find the content useful.

That is good news that WP Super Cache is no longer conflicting with the directives in the blacklist. I had heard that they recently updated the plugin, and so it looks as if they must have fixed the issue referred to here. Great news :)

As for the various validators, they should still be able to access your pages. The RedirectMatch 403 // directive is actually targeting the main part of the URL itself and does not block anything in the query string. Additionally, the blacklist is only blocking requests for URLs that are a part of your domain; their appending your page URL to their URL would only be affected if they were using the blacklist on their own domain (which I’m sure they’re not). Nonetheless, I have seen this issue before with the directives used in the Ultimate htaccess Blacklist, so you may want to check that possibility as well.
Fred 2008/12/02 11:35 am

Hi, I followed a link to this article and have been reading up on the 3G Blacklist. I copied the file and paste it first in my configuration but the server won’t load so I tried it in the .htaccess and get Forbidden: Access Permission Denied.

I then start removing lines.
RedirectMatch 403
Only when I remove these two line the site loads but still will not work from my configuration file.

I’m new to all of this. Any suggesstions and is it OK to leave these out?

Thanks

Fred
Jeff Starr 2008/12/03 3:42 pm • Post Author

@Fred: By all means, remove any lines that cause your server to crash. Unfortunately, when it comes to blacklisting, it is practically impossible to forge a “one-size-fits-all” strategy. For the most part, the blacklist works great out of the box, but various server/site configurations may require some fine-tuning. To do so, simply comment out or remove the offending lines and you should be good to go. Each of the different RedirectMatch directives operates independently, so that removing any of them simply disables the particular character-string match for which they are intended to block. In other words, the blacklist will still protect against all types of attacks represented by the remaining directives.
Fred 2008/12/04 9:25 pm

Jeff, thanks for the response and I’m pleased for the comments. I have couple of sites running off the same server and these bots really over load my server with the frequency at which they load pages. Because I have access to root, I’m able to httpd.reload and free resources up. For the next few days I will closely monitor to see the results.

Again, Many Thanks

Fred
Tony 2008/12/06 11:59 am

Sorry for the delay, it’s been a very busy week!

I found out that even though I could access my blog and everything is working fine, the validators couldn’t access it because of “RedirectMatch 403 \/\/” when WP Super Cache is enabled. If any one of them is disabled, then everything works fine.

Thanks again Jeff for this great list!
Sebastian 2008/12/08 5:16 am

Thanks for your work.

For Joomla 1.0.15 I needed to disable:
# RedirectMatch 403 \/\/
# RedirectMatch 403 \/includes\/

First rule is very site specific as tinyMCE inserted images with a double //
Second rule had prevented loading of some of the js scripts needed for administration. (eg. Save / Close links not working).
Jeff Starr 2008/12/08 9:46 pm • Post Author

@Sebastian: Thank you — this information will be integrated into the next version of the blacklist, which hopefully will be bubbling its way to the surface here very soon..
Otto 2008/12/13 4:32 am

Jeff, thanks for sharing the 3G list. One thing I noticed is that the character strings group blocked some of my pages. The 3G lines in question were:

# RedirectMatch 403 news.php
# RedirectMatch 403 contact.php

I am probably not the only one who named the pages on a more “traditional way”, such as contact.php. Any reason why these pages should be blocked?
Otto
Jeff Starr 2008/12/15 10:06 pm • Post Author

@Otto: Of course, comment out or remove any lines that prevent access to actual pages on your site. The two pages you mention are are very common and thus frequently targeted by attackers. If the pages actually exist, the number of misdirected site errors will decrease, however variations on the file names in question will persist.

« Previous Comments • 1…34567 • Newer Comments »