The Perishable Press 4G Blacklist

Karthik Viswanathan – #1

This is a comprehensive yet compact way to deal with malicious attacks. I’m truly impressed with the effort you’ve put into this. Unfortunately, this line poses a small problem to my website:

RedirectMatch 403 \/\/

How important would you consider blocking the // to be? A few WordPress plugins such as the WP-Clickmap require this string to function properly.

Awesome Blacklist – #3

This is an awesome blacklist.

Very effective :D

Jonathan Ellse – #4

Works a treat. Very comprehensive customisation methods.

Thanks very much

Arwin – #5

Great blacklist and clear story, fun to read!

I had one problem,
I have to remove ‘config’ from the ‘QUERY STRING EXPLOITS’ because otherwise my Mambo and Joomla site will not work properly.
Problem with the configuration of the configuration.php in the administrator backend of Mambo/Joomla (403 Redirect)
Still no other problems found :)

Thank you very much!

John – #6

You really put a lot of good effort into building this 4G blacklist to share with everyone else. I, as with many other silent supporters, thank you for your efforts. Previously, some of my other sites that I am the head manager of got hacked from time to time, and I’ll be eagerly waiting to see if there are future hacking attempts after I’ve implemented this.

Again, thank you so much for making the world a bit better with your efforts.

Greg – #7

Just tested, and for the moment have just to lines to be commented out for me.

One is for my downloads links which are unfortunately with some comma so:
#RedirectMatch 403 \,

And my registration page got the name “register” in the url so:
#RedirectMatch 403 register\.

Check few more things…

Greg – #9

Changed this one:
RewriteCond %{QUERY_STRING} ^.*(\[|\]|\(|\)||'|"|;|\?|\*).* [NC,OR]

to:
RewriteCond %{QUERY_STRING} ^.*(\[|\]|\(|\)||'|"|\?|\*).* [NC,OR]

Because of my pdf links on articles which are with semicolon

I’m under joomla 1.5.9

Greg – #11

Ok Jeff ;-)

Just posted your 4G on the french joomla forum, so we will have more feedback soon for joomla (hope it’s ok for you ?)

Greg – #12

Commented this one for joomla:
#RedirectMatch 403 config\.

Because of the name of an icon of the administration:
/icon-48-config.png

Sven – #13

Thanks a lot Jeff :-)

Are there any advantages when you use
“RedirectMatch 403″ and not “RewriteCond” ?

Arwin – #14

Jeff, I also need to remove the [ and ] from the QUERY STRING EXPLOITS section, otherwise you can not set the plugin-options in Joomla(1.5.9).

Example url:
http://....option=com_plugins&view=plugin&client=site&task=edit&cid[]=1

Arwin – #15

Oeps, that example url did not do what I want.. Without the http.. : index.php?option=com_plugins&view=plugin&client=site&task=edit&cid[]=1

Greg – #18

Same as Arwin said:

just done this:
RewriteCond %{QUERY_STRING} ^.*(\(|\)||'|"|\?|\*).* [NC,OR]

Sven – #19

Thank for the reply Jeff.

I asked because I would like to direct my visitors to a “friendly page” if they are blocked by httaccess (I also log all blocked requests in a txt-file). This is only possible if I use a Rewrite, or…?

Sven – #21

Thanks Jeff :-)

Greg – #22

Hi,

For Joomla 1.5.9

Found today that we need to remove the “request” from the query string exploits conditions.

Because some components use it with ajax script on the backend (com_xmap for example)

So this:
RewriteCond %{QUERY_STRING} ^.*(request|select|insert|union|declare|drop).* [NC]

Become:
RewriteCond %{QUERY_STRING} ^.*(select|insert|union|declare|drop).* [NC]

Andrew – #23

Great list, thanks for sharing it!

The only major issue I’ve had so far was with the Blacklist Candidates (IP Blocking) section. My apache (Apache 2.2.8) kept giving me “Order not allowed here”, “Allow not allowed here”, and “Deny not allowed here” no matter what I tried.

I ultimately found out that the 2.2 module’s (mod_authz_host, previously mod_access) directives are only allowed inside <Directory>, <Files>, and <Location> contexts and .htaccess files: http://httpd.apache.org/docs/2.2/mod/mod_authz_host.html

After wrapping the entire <Limit> block in a Directory context matching all possible directories (<Directory /*>), apache gave me no more trouble.

Strange that I seem to be the only one having this problem.

Also, I initially had wrapped that section in an IfModule block, which I think is a good idea for any block that depends on a module.

Greg – #24

@Andrew

Hi,

Got Apache 2.2.10 and don’t have your issue….

Where’s located your htaccess ? in your www or elsewhere on up directory ?

Deb Phillips – #25

Jeff, I implemented your 4G Blacklist code about two days ago, and I’m amazed at the “calmness” in my visitor logs now! I no longer feel besieged by one unruly bot after another. What a relief! I just had to say “Thank you!” Your blacklist is wonderful, and it is so good of you to share it.

All the best,
Deb

Tony – #26

Jeff,
I’ve been waiting forever for the 4G! I installed it on my first Wordpress blog for testing purposes. Here is what I noticed:

1. If you have the WP plugin “Ozh’ Admin Drop Down Menu” (latest version is 3.1.3.3.7 now), you’ll have to delete “drop” from the last line of the Query Strings Exploits:

RewriteCond %{QUERY_STRING} ^.*(request|select|insert|union|declare|drop).* [NC]

Becomes:

RewriteCond %{QUERY_STRING} ^.*(request|select|insert|union|declare).* [NC]

2. If you use the TinyMCE Advanced plugin (latest version is 3.2 now), the buttons won’t be visible because the css won’t load, because it’s located in a directory wp_theme. Comment out the line that says: RedirectMatch 404 wp\_

And a question: Wordpress gives a 404 instead of Apache giving a 403, unless I create a custom 403 ErrorDocument. Any idea why?

Thanks Jeff, and I’ll be testing more in the next few days.

Deb Phillips – #27

I can confirm that the default TinyMCE toolbar buttons also do not show up correctly. Only by removing the 4G code from the .htaccess file will the TinyMCE toolbar be displayed correctly.

I use the 404 Notifier plugin to alert me to any 404s occurring, and I’ve been getting a number 404s generated only when the 4G code is loaded, Unfortunately, I’m not savvy enough to pinpoint the exact cause of these errors, but almost all of the 404s occur at the point of WordPress’ cron check. I’m getting hundreds of 404s — daily — that are related to the cron check.

Another issue that I wonder if it’s related to the 4G blacklist is with one bot repeatedly coming in looking for the 403.shtml file, and a 404 is generated. I don’t *think* this was occurring before I began using the 4G code. Perhaps this is what Tony is referring to.

Even with these few issues, though, I really do appreciate all the work that went into the 4G blacklist. Hopefully, we’ll get the few kinks worked out.

Thanks very much.

Tony – #28

Deb, you’re right. I could have sworn that the standard tinyMCE loaded fine when I tried, and I had already cleared my browser’s cache. Now I try again and the standard tinyMCE also doesn’t load okay.

The problem is this address that tinymce calls:
/wp-includes/js/tinymce/themes/advanced/skins/wp_theme/ui.css?ver=20081129

BTW, I used Firebug to pinpoint that. For those interested, Firebug is a Firefox Extension.

Tony – #30

Jeff,
It’s the least I can do!

wp-cron is making HEAD requests, so it’s being blocked. What do you think we should do, besides removing HEAD from the 4G’s Request Methods Filter?

From the raw access log:
"HEAD /wp-cron.php?check=561caf12167cb54c25589d71581df596 HTTP/1.0" 403 - "-" "WordPress/2.7.1"

I apologize for repeating the question I asked above, but maybe nobody noticed it:
Wordpress gives a 404 instead of Apache giving a 403, unless I create a custom 403 ErrorDocument. Any idea why?

Thanks!

Deb Phillips – #31

Yes, I strongly second Tony’s inquiry regarding wp-cron. I’m getting deluged with 404 notifications related to it. These are email alerts I receive from the 404 Notifier plugin. They’re occurring about every one or two minutes.

Also, Tony, what issue were you running into with the Ozh’ Admin Drop Down Menu plugin? I only ask because I’m running the Ozh’ Better Feed plugin. I’m guessing there’s probably not any basically similar code between the two, but I wanted to ask. Thanks.

Tony – #32

Deb,

Sorry, I should have been more clear about Ozh’ Admin Drop Down Menu. This plugin calls this address:

/wp-content/plugins/ozh-admin-drop-down-menu/inc/adminmenu.css.php?p=%2Fwp-content%2Fplugins%2Fozh-admin-drop-down-menu%2Finc&i=1&w=1&m=0&c=0&h=0&f=1&g=%23676768&n=0

Since there’s the “drop” word in the query string, it gets a 403 and the whole admin page gets messed up.

And some good news: googlebot accessed my site today and got a 200. That’s the one thing I was worried about the most!

Deb Phillips – #33

Tony, thanks for the info on the Ozh’ plugin.

I’m glad Google is showing your site as 200.

As clarification, I’ve not encountered any 404 pages when actually visiting my website. The 404 notifications I’ve referred to are all happening behind the scenes. I can also see them showing up in the cPanel “Latest Visitors” module.

I’ve also received some 404s related to accessing files in the wp-content/uploads directory. But every time I’ve gone to the pages containing those files on the website, the images are all there, and there’s no problem on those pages. So I suspect the 404s on those image files are somehow related to the 404 triggered by the wp-cron issue that occurred on those pages.

These are my observations so far. I don’t know all the technically correct terms to use, so I’m trying to relay the info as clearly as I can!

Tony – #34

Deb, you’re welcome! Your site is really nice, and the design is great, not cluttered at all. I opened several pages and every single item (css, images, js, etc.) on every page loads flawlessly. I really recommend using Firebug when trying something as serious as the 4G blacklist.

Deb Phillips – #35

Thanks so much, Tony. I’m honored that you stopped by my website.

As far as Firebug, I do use it, but I usually use it related to experimenting with graphics or layout changes prior to actually modifying the CSS.

Can you describe the steps (maybe just the buttons or the menu sequence) you typically take — without having to spend a lot of time to answer this — to use Firebug to troubleshoot issues such as the ones we’ve encountered here? (I had mainly been looking at the Latest Visitor module in cPanel and noting the error codes that were generated on specific URLs.)

Please don’t take a lot of time to answer this. (In that case, I’ll just have to do a little research on it.) I don’t want to create work for other people!

Thanks!

Deb Phillips – #36

By the way, Tony. Please tell me it was you who happened to do some “testing” while visiting my site. That’ll make me feel a little better. Thanks!

Tony – #37

Deb,
Jeff shared so much of his work with us; how do a few minutes of my time compare to that? :-D

For troubleshooting this kind of stuff, I use the “Net” tab in Firebug. Some screenshots of the Net tab from the official site:
http://getfirebug.com/net.html

Whenever an item of the page you’re loading gets a 403/404 response or other errors, you’ll see it in red. You can right click and copy its location, and do other stuff.

If you want to see how the page items are loading, but are only interested in, let’s say, the css files only, click on the button that says CSS above the Net tab. Of course the Net tab should be active for you to be able to even see those buttons.

I’m not that good at explaining things, especially when it’s not concrete. So please don’t be shy to ask questions. I am new at this myself, so we’ll both learn something! ;-)

Tony – #38

Yes it was me who tested. I even wrote hideb.inc in the URL. I hope it’s okay…

Deb Phillips – #39

Thanks for the Firebug info, Tony. I’ll do some experimenting with it.

Yes, I saw the “hideb.inc.” I’m relieved that was you. I was beginning to think the Internet was more treacherous than I’d imagined!

P.S. Go UNC!

Tony – #41

Jeff,

I’m happy to help, at least a little!

As for wp-cron, this is not the plugin (which is really, really outdated). It’s the standard wp-cron.php in the Wordpress package.

Is it possible to keep HEAD in the request-method filtering rules, but make some kind of exception for wp-cron.php?

If I find out why Wordpress is returning the 404s, I’ll post here.

Andrew – #42

@Greg and @Jeff:

Oh, sorry, I forgot to mention: I was putting this in my httpd.conf file, not a .htaccess file. Sorry for the confusion!

As I mentioned, those directives are allowed in htaccess files, so if you just drop it into an htaccess file, it should be fine. It’s only in the httpd.conf file that it needs the extra wrapper.

Deb Phillips – #43

Just to get some clarification, Jeff and Tony:

It sounds like the WP-Cron fix you’ve just implemented, Jeff, does not address the 404s received in relation to wp-cron.php? That’s what I’m continuing to receive droves of 404s on. (I’ve not tried your revised code yet.)

Thank you.

Tony – #44

Deb,
I don’t think Jeff revised any of the code. He just advised to remove HEAD.

If you removed it yourself, can you paste the line from your raw access log where it says wp-cron.php?

Also, when you remove HEAD, be sure to remove the “|” after it.

Tony – #47

Jeff,
That’s very good to know! I’m already waiting for the 5G! :)

Deb Phillips – #48

Thanks so much, Jeff. I must say, you’re one hard-working, conscientious dude! We’re fortunate you’re on the “good side”!

Greg – #49

Ok with that.

No more feedback for joomla at this time.

Thanks for your work Jeff

eezz – #50

Hi, Great work… one thing I have just picked up with Joomla when using OpenID… the redirect url back to Joomla from the OpenID server is denied. The url is very long and contains http and ? and plenty of %XX. I’ve tried a few OpenID’s and they work with the following mod:

RewriteCond %{QUERY_STRING} \.\.\/ [NC,OR]
RewriteCond %{QUERY_STRING} boot\.ini [NC,OR]
RewriteCond %{QUERY_STRING} tag\= [NC,OR]
RewriteCond %{QUERY_STRING} ftp\: [NC,OR]
RewriteCond %{QUERY_STRING} https\: [NC,OR]
RewriteCond %{QUERY_STRING} mosConfig [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(\(|\)||'|"|;|\*).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(%0|%A|%B|%C|%D|%E|%F|127\.0).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(globals|encode|localhost|loopback).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(select|insert|union|declare|drop).* [NC]
RewriteRule ^(.*)$ - [F,L]

Garrett W. – #51

Ran into a few problems so I thought I’d let you know.

Under Query String Exploits:
- Blocking “config” interferes with the Akismet settings page in Wordpress
- Blocking “request” interferes with phpMyID ( http://siege.org/projects/phpMyID/ ) if I remember correctly
- Blocking “;” interferes with all but a few pages of SMF ( http://www.simplemachines.org/ )

I had too many problems with phpMyID anyway and decided to stop using it, but I had to remove the other two offending strings.

Also, under Character Strings:
- Blocking “,” interferes with SMF forum/subforum/topic pages IF search-engine friendly URLs are enabled

Also, under Query String Exploits:
- Blocking “select” interferes with the “Announce topic” option when creating a new topic in SMF. URL: “.../forums/?action=announce;sa=selectgroup;topic=2;goback”

Greg – #52

Hi Garrett W.

Ok I give it a try on my own SMF forum too (wthout sef rewrite)
Tell you some feedbacks…

Garrett W. – #54

I don’t mind at all - I’m glad you did :)

eezz – #55

Glad to help… bandwidth usage has almost halved on my site with using both the 4G and UA ‘firewalls’ you have made. This is a fantastic contribution the fight against the bot hoards.

Garrett W. – #57

Jeff: have you heard of Project Honeypot? ( http://projecthoneypot.org/ )
I recently blocked 4 IPs from my site due to comment spam (and now I’m getting none at all!) and sure enough, all 4 IPs were listed as comment spammers in the Honeypot database.
Wish I had noticed it sooner, but they offer a service called http:BL (BL standing for Blacklist) that queries their database about every visitor to your site (in the form of a quick DNS request), and blocks the ones that are malicious. It even comes in the form of a Wordpress plugin! So I think I’m gonna try it out.
http://projecthoneypot.org/httpbl_implementations.php

Donace | The Nexus – #59

@Garrett; bad behaviour plugin has honeypot blacklist implemented into it; I advise you check that out.

Would also recommend doing subtle changes to the comment field names as that is how harvesters harvest the urls to spam ;)

Deb Phillips – #60

Hey, Jeff —

I don’t want to put you on the spot, but I’m a real novice at the bot wars and how to deal with them. So when the mention was made in previous comments about using plugins to combat bad bots, it made me wonder why you chose to take the time to come up with code to add to the .htaccess file versus using a plugin or two.

Are there advantages to taking your route versus plugins? Perhaps WordPress performance advantages? Or something else?

I’m just curious! :) Thanks.

Deb

Garrett W. – #61

Using .htaccess stuff heads ‘em off at the front, instead of allowing them to see your site first. Plus, it’s probably faster than letting the plugins deal with them in PHP.

Deb Phillips – #62

Then I’ll stay on this path, Garrett W.! Thanks for answering.

Tony – #63

What Garrett said, plus the wonderful thing about this blacklist is that you can use it even if you don’t use WordPress.

I feel great when I know that some malicious bots don’t consume any of my bandwidth and don’t slow my websites. Stop them at the door! ;-)

Yieu – #65

I just wanted to say that I implemented the 4G Blacklist on my website, and it is a regular website — not a blogging website such as Wordpress. It is very handy, as I know of no other blacklist such as this, and it appears to be very comprehensive. I do hope the fact that it is geared towards Wordpress does not leave out some security for regular websites, though. Perhaps a version geared towards regular websites might benefit other webmasters?

I also make use of the proxy blocking blacklist found here which helps keep spammers away, and the universal URL canonization directive. Thank you for making all of these available, it has definitely helped greatly.

Yieu – #67

When I said a version geared towards regular websites, I did not mean to only include directives relevant to regular websites and to leave out the Wordpress directives. I noticed that you were removing directives because they were in conflcit with Wordpress, so one geared towards regular websites would re-include those directives for a tighter level of security — so it would leave in the Wordpress directives, add in the ones that were removed for Wordpress compatibility, and add directives geared specifically for regular websites on top of that.

I am not sure if this is asking for a too much (the current 4G Blacklist is very nice as it is), but it would also be a more universal security solution for websites and I am sure that would be useful. Such a list may require some explanation for some of the directives that are more liable to cause issues, though (or to simply continue leaving some out if they are too restrictive).

Greg – #68

Hi,

today I found that the %3D block some of my google ads, so:

RewriteCond %{QUERY_STRING} ^.*(%22|%27|%3C|%3D|%3E|%7B|%7C).* [NC,OR]

become (for me):

RewriteCond %{QUERY_STRING} ^.*(%22|%27|%3C|%3E|%7B|%7C).* [NC,OR]

Yieu – #71

I have just installed vBulletin version 3.8.2 on my website which runs the 4G Blacklist, and I had to comment out the following directives:

# RedirectMatch 403 display\.
# RedirectMatch 403 register\.

The 403 display directive was blocking the individual forums from displaying, because their link includes “forumdisplay.php”, and the 403 register directive was preventing the registering of new accounts because the link to register includes “register.php”.

Dave Stuttard – #72

Hi, this innovative package looks great. Just implemented the 4G Blacklist in .htaccess on 3 sites (all standard, ie not WP or Joomla!), two being static, one being dynamic with a CMS. All features work fine, except:

1) I had to comment out http\ in the Query String Exploits before my Flash elements displayed - thought I’d mention that because nobody else has noted it.
2) I had to comment out RedirectMatch 403 config\ in the SPECIFIC EXPLOITS before the wysywyg Editor box was displayed for User data entry on the dynamic site - the code with ‘config’ in it is (the id):

Now I will study the stats to make sure Google, Yahoo! and MSN get 200s. I’ll let you know….. Many thanks

Dave Stuttard – #73

Sorry, the code entered in my last post is:

<input type="hidden" id="LongDesc" name="LongDesc" value="" style="display:none" /><input type="hidden" id="LongDesc___Config" value="" style="display:none" /><iframe id="LongDesc___Frame" src="/editor/editor/fckeditor.html?InstanceName=LongDesc&amp;Toolbar=Default" width="100%" height="400" frameborder="no" scrolling="no"></iframe>

Yieu – #74

In order to allow private messages on vBulletin, I had to remove “insert” from the following directive:

RewriteCond %{QUERY_STRING} ^.*(request|select|insert|union|declare|drop).* [NC]

becomes:

RewriteCond %{QUERY_STRING} ^.*(request|select|union|declare|drop).* [NC]

Dave Stuttard – #75

Hi, I found another thing needing to be commented out (this time from 403 errors):
RedirectMatch 403 \,
This was because there are some %2s, ie commas, in dynamically-generated property page urls.
Not much happening in my stats (it’s Sunday); I’ll check Monday evening to see if the search engines get 200s for any pages.
If it’s OK I’ll buy you a pint (and follow your progress of course)!
The importance of checking every user operation/display on the websites is clearly emphasised.
Thanks, Dave
PS: Don’t know about the ban proxy server script - I imagine thousands of legit ISP proxys needing to be identified and allowed to get any visitors (I posted an email on this but don’t know if it was sent).

Greg – #76

Hi again ^^

Just commented out this one:
#RedirectMatch 403 \;

Cause block the vidéos player links of the jomtube component for Joomla 1.5.10

Dave Stuttard – #77

Hi, just wanted to confirm that G,Y and M appear to be crawling successfully on my 3 sites. Wow, that means this blacklist is an excellent addition to a website’s security setup (with such things as captchas and URL verifications in forms and email address obfuscating) - when thorough testing has been done and any problem entries have been commented out. It saves a lot of time trying to keep track of and banning bad bots individually. Hope my posts were useful. Looking forward to its further development, eg versions for different types of website, with possible problem entries highlighted to help diagnosis when things go wrong?
I know - I owe the author a pint (Jeff?). And thanks to others who have contributed useful suggestions.
Regards, Dave

Greg – #79

More precisions about my previous %3D submission here:http://perishablepress.com/press/2009/03/16/the-perishable-press-4g-blacklist/#comment-71184

This one is only useful to be applied only when you need to preview some ads on your site, when you are in your googlads administration.

Greg – #80

On my forum SMF 2.0RC1

RewriteCond %{QUERY_STRING} ^.*(select|insert|union|declare|drop).* [NC]

becomes:
RewriteCond %{QUERY_STRING} ^.*(select|insert|union|declare).* [NC]

Because all my dropdown menus were blocked

Greg – #81

Hi,

Joomla 1.5.10

change:
RewriteCond %{QUERY_STRING} ^.*(%0|%A|%B|%C|%D|%E|%F|127\.0).* [NC,OR]

to:
RewriteCond %{QUERY_STRING} ^.*(%0|%A|%B|%C|%D|%F|127\.0).* [NC,OR]

because the %E blocked the poll component

Tom – #82

What a great resource :) Works a treat, but I encountered a problem with Remository on Joomla 1.5.x. Here’s the offending URL string that triggers a 403:

/component/remository/func-download/248/chk,8e2144d3ae6dbfd7591edaa4c16526df/no_html,1/

Greg – #83

Hi, Tom

did you check the updates for joomla on the end of this article ?

Tom – #84

Did I check the updates? Not yet :(

I did narrow it down to the filtering of the comma in the URL string. If I comment that out, Remository downloads work. However, I detect from your writing that you really don’t like eliminating an entire class of filter, because it then leaves other potential exploits available. That’s why I was supplying the “bad” URL string…which has *two* instances of a comma in it.

Will check the Joomla updates later this morning. And Greg: thanks again. You provide some wonderful stuff :)

Greg – #85

ok, thanks Tom

…the stuff master is Jeff Starr ;-)

Tom – #86

I see Dave Stuttard in comment #75 already picked up this particular issue, albeit it with a different Joomla! component. The comma in a URL does affect Remository in Joomla! 1.5.x, and one way to fix it is to comment out the “RedirectMatch 403 \,” directive.

Dave Stuttard – #87

Tom
Just a thought - I don’t use Joomla! or Wordpress or SMF or vBulletin, etc. My site with the comma in some address strings is a conventional php/MySQL driven dynanic site - the point is that the 4G Blacklist may have been aimed at Joomla! and Wordpress scenarios originally but it can also apply to other scenarios like mine.

General Point
‘SpamBlockerUtility’, ScoutJet and maybe others are still getting in so maybe these should be dealt with individually with ‘RewriteCond’s (otherwise one usually has to deny ranges of IPs, not single IPs, possibly blocking some legitimate visitors):

RewriteCond %{HTTP_USER_AGENT} spamblockerutility [NC,OR]
RewriteCond %{HTTP_USER_AGENT} scoutjet [NC]
RewriteRule ^.* - [F,L]

(note the absense of ^ before the UA element - without it we can ban any UA string that includes that element (somebody correct me please if my code is wrong) and don’t forget the NC to make it case-insensitive.

Chaos Inc. – #88

Hiya Jeff!

Already mailed ya yesterday regarding this but I want to add a little more (a lot more lol) feedback with the ones I already mentioned in the mail. Also, I hope that this will help you and in turn, help us, to generate the ultimate GX (generation X) Blacklist.

After implementing only your 4g blacklist on my blog:

My Wordpress Auto Thumb Generation plugin (TimThumb) is not working, and it maybe because the auto generated images/thumbs carries some & or % characters/strings.

Users browsing from all and any kind of mobile platform is being shown the
403.

Users browsing from Opera mobile, opera mini, and other symbian/iphone
applications are being blocked. There are two kinds of connection for Opera
mini : Socket and Http based connections. Neither works. Some mobile phone
browsers also user shadow connects by default and those are also being
blocked.

Both symbian, python and flashlite based browsers are being shown 403.

I’m a mobile theme and ui developer and people similar to me who actually
produces content are being scraped severely, suffering stolen content
dilemmas and other general dos based attacks. Is there a way I can block
someone/everyone who regularly visits a site like scraping.com and visiting
mine as well. What i mean to say is that can i block someone who is probably
stealing my content and posting in his site? Since there is a usage pattern
and trend for that visitor like he visits his site first and then visits
mine to copy and he does this for every post i make. Cant they be blocked
based on their behaviorat pattern?

Almost all telecom operators usually provide 3g, gprs or 2.5g based mobile
internet to all their clients. Typically a mobile user from a country like
Bangladesh (my country) will have the ip of Norway on his phone because the
telecom or gsm operator’s mobile internet gateway for all the countries
they operate in are going through a single wireless internet platform (to be
cost effective and so on). This platform actually works as a hub for their
global 3g network in different countries. Is there a way to somehow let
users enter my site by skipping all the htaccess rules set by me when they
are from a mobile phone/pda/iphone etc.

Lastly, almost anyone who have implemented cname records for his WAP or
Online Store for a subdomain on his site cannot go back to the main
page/normal-pages when they have visited the store ( otherwise referred by
the store). This can be because of the weird caniocal url for any content on
the store/wapsite typically produce (not sure though). Is there a way to let
users browse my site normally when referred from a specific
domain/subdomain/platform etc?

Some mobile phone focused search engine crawlers are being blocked. Like
those from google mobile or mobile meta tag search. Is there a way to fix
this and let them in?

Greg – #89

Done this today for SMF 2.0 RC1

RewriteCond %{QUERY_STRING} ^.*(\(|\)||'|"|\?|\*).* [NC,OR]

Because with the \[|\] we can’t erase some messages in the private box of the forum (MP)

Lee – #90

Ah yes! Working great! Thanks, man. We all appreciate it.

John Hoff - WpBlogHost – #91

Jeff, this is an excellent list of directives. Thank you for sharing.

Question. One of the services we provide is hardening people’s WordPress blog. I’d like to include these in our upgrades. Would you be open to letting me paste these directives in to our customer’s .htaccess files? I’d be sure to leave the comment showing your site and anything else you might request.

Thanks. Stumbling this article and Tweeting it!

Nihar – #92

Awesome list.

I will add it in .htaccess file.

Have on one doubt, Where should i add the above list? after wordpress lines in htaccess file or before?

Garrett W. – #93

Before.

Douglas Machado – #95

Have you ever tested Apache mod_security?
Which is the best (most reliable) and faster security solution 4G Blacklist of mod_security?

ModSecurity website:
http://www.modsecurity.org/projects/modsecurity/apache/index.html

MileHighTechGuy – #97

This is great stuff.

Under WordPress I had to comment out the following ‘config’ line for the ‘Deans FCK Editor’ to work:

# SPECIFIC EXPLOITS
#this line conflicts with Deans FCK Editor WP plugin
#RedirectMatch 403 config\.

I’m posting a security related article later today that will reference your work.

Thanks for the great effort.

~Jeff (MileHighTechGuy)

MileHighTechGuy – #99

I’m glad you think my post might have some value. Thanks for the feedback.

Not only is it posted here on my blog:
http://milehightechguy.com/2009/06/how-to-guide-for-securing-wordpress-and-protecting-websites/

But it is also posted here at Examiner.com:
http://www.examiner.com/…/WordPress-installation-and-recove

Vladimir – #100

Jeff, if Apache stores its logs in CLF format and you have Linux, here’s a nice command to get all 404’s:

cat *.log | grep " 404 " | awk '{ print $7; }' | sort | uniq

I guess it can give you some more patterns to check against ;-)

Brandon – #101

This line:

RewriteRule ^(.*)$ - [F,L]

stops WordPress 2.8’s widget manipulation screen from working. It kills the javascript.

Brandon – #103

Sorry, that was query-string exploits.

MIke Selvage – #104

Mike Selvage: Great work! But I have a question. I get the idea the 4G Blacklist supersedes the User_Agent blacklist … but does it also supersede the Referrer blacklist?

MIke Selvage – #105

Mike Selvage: Disregard the last post, please. In the past 2 minutes I learned how to read …

Garrett W. – #106

Just found another problem: The following from Wordpress is caught and returns a 403.
/wp-admin/plugins.php?action=delete-selected&checked[0]=adsense-manager%2Fadsense-manager.php&plugin_status=all&paged=1&_wpnonce=159d52cca3

…but I can’t figure out which block is triggering it. I tried taking out everything that blocked [ and ], as well as %5B and %5D, and it still wouldn’t go through. Any idea?

Tom – #108

I have to echo Brandon in comment #101. I’ve upgraded a couple of sites to WordPress 2.8, and cannot modify or move widgets. When I replace the 4G list with a standard WordPress .htaccess file, the ajaxy-goodness of WP 2.8 works again. Restore 4G in the .htaccess, and it does not. I’m sorry I can’t be more specific than that. I’m not really seeing what wp-admin is trying to call when I’m attempting to twiddle with widgets. My workaround for the moment is to temporarily replace the 4G-based .htaccess with a vanilla .htaccess, but that is not particularly convenient. Thoughts?

Garrett W. – #109

Ah, so now that I’m experiencing the same thing, I realize that someone already commented about it. lol.
So the blacklist is responsible? That’s good - saved me from having to trudge through WordPress’s code.
However, I did look through it a little, and I found the offending piece of info.

These lines:
RewriteCond %{QUERY_STRING} (request|select|insert|union|declare|drop) [NC]
RewriteRule .* - [F,L]

are being triggered by:
wp-admin/load-scripts.php?[…]load=[…],jquery-ui-droppable,[…]

Tom – #111

Jeff, your change in #110 solved the problem for me in WordPress 2.8. And thanks Garrett for providing enough detail, where my information was sorely lacking!

Garrett W. – #112

fyi, my name has two Ts in it ;)

Garrett W. – #114

Thanks. I wouldn’t have mentioned it, but you were consistently misspelling it… and that’s a pet peeve of mine. ;)

Garrett W. – #116

Aw, you poor thing .. it’s ok, I forgive you :)

Scotia – #118

Joomla 1.5.12
Community Builder v1.21
SOBI2 v2.9.2.1
sh404SEF dev. 1.0.20 Build 237
Smartoptimizer v1.7b
CssJscompress v1.1

CBlogin module (to log into user accounts to work) I had to comment out:

# RewriteCond %{QUERY_STRING} ^.*(select|insert|union|declare|drop).* [NC] - I tried each of select|insert|union|declare|drop separately, still would not work. - What is the impact of losing this one?

SOBI2 Search functionality was messed until I commented out:

# RewriteRule ^(.*)$ - [F,L] - Can I change anything here for this to work? What is the impact of losing this one?

CBLogin mod Lost password functionality lost (Ajax) until one of these two was commented out:

# RedirectMatch 403 register\.
# Redirectmatch 403 password\.

I hope this is usefull for anyone else running Joomla, in combination with CB, sh404 etc.

Greg – #119

For select|insert|union|declare|drop , so this is probably not the real cause.

For # RewriteRule ^(.*)$ - [F,L], this is the rewrite rule, so if you comment it out, all the directives abvo becomes inneficient.

Add something like at the first place of the query strings block: RewriteCond %{HTTP_REFERER} !^(.*)(put_the_name_of_thepage_suspected_here).* [NC] could help…

But you have to find the real parameter that is blocking SOBI.

Greg – #120

Hi,

so many like that in my log file:

"HEAD /favicon.ico HTTP/1.0" 403 - "-" "-"

How should we deal with thoses ?

Should we had :

RewriteCond %{REQUEST_URI} !^/favicon.ico

Just after the # FILTER REQUEST METHODS ?

Greg – #122

Thanks Jeff

Ok with that, but I ask this, because those are requests with the HEAD method, so I was wondering how to deal with the favicon requested with HEAD.
Should we consider those as legitimate ?
Or it must be only allowed from a GET ?

Greg – #124

Found it !

Lol it was me….all my tests with firebug an YSlow were made all this HEAD requests.

ken – #126

If you block some ip addresses would you block out alot of people that do not have a static ip. I also saw some question about mobile phones not being able to access site. I did not see answer or if it is a problem.

duck – #127

Firstly, thanks for such a great resource, this post and the building of the 4G blacklist are great posts.

I discovered that after implementing the 4G blacklist on a Wordpress 2.8.4 website that I was unable to delete plugins from the Plugin menu (was mentioned in comment 106). The link of the delete button takes the form:

...plugins.php?action=delete-selected&checked[0]=hello.php&plugin_status=all&paged=1&_wpno...

To fix this you need to comment out the lines blocking [ and ] as well as removing select from the last condition in the Query string exploits section.

(It might be useful to comment out the lines blocking %5B and %5D, but it wasn’t necessary for me)

duck – #128

I need a bit of help as an implementation of the 4G blacklist doesn’t seem to be working. It is working perfectly on my local machine, giving me 403 errors if I put “password.” at the end of the url for example, however on the production server it is producing 404 errors instead. Looking in cPanel redirects I see that it is interpreting all the RedirectMatch 403 rules as pointing to a directory 403, being type permanent and the redirect url being something like \. or \,.

Do you think that this is an issue with cPanel, my webhost’s setup or anything else you have come across before?

jon (aka duck) – #130

I discovered that, at least for my situation, the server was still responding with a 403 status header but just not displaying an Access Denied page. All I had to do was declare an ErrorDocument in my htaccess file e.g.

ErrorDocument 403 "<h1>Access Forbidden!</h1><h2>Error 403</h2>"

and it worked. This is obviously something to do with the way the server had been configured as it was all working perfectly on my local test install of apache.

Cooltad – #132

I’m insane about speed and optimization.
Do you have a flattened version of your 4G list and it’s optional components?

Cooltad – #134

Most certainly. We all should know that, even if there are whitespaces and new lines, each of those 2 require one byte, or 6 bits. Now, with your massive 4g list compiled together, you’d save a very large amount of bytes, which would make processing server-side for the .htaccess faster.
You wouldn’t think it would, because it doesn’t ‘parse’ whitespaces etc, however, it still needs to sift through such spaces to get to the real stuff.

Greg – #135

I think all the Redirectmatch 403, could be in one line.
…for example.

Peekay – #137

Xoops users who allow user registration are going to need to comment out:

# RedirectMatch 403 register\.

from the ‘Specific Exploits’ section.

Omar Ramos – #138

I was wondering how we might be able to test that the blacklist is working correctly?

For example, it seems like it would be as easy as using “select” within the query string of the URL, but when I try doing that I’m not getting a 403 response, the page just ends up loading the same way it did before (I’m using the above code within an .htaccess file).

Any thoughts on how this can be tested simply, just to make sure the rules are in effect?

Thank you!

Omar Ramos – #139

I tried the visiting the following URL:
index.php?option=com_newsflash&id=8+and+1=1+union+select+1,username,password,4+from+mos_users&catid=0

On our remaining Joomla 1.0 site and it seems to be working.

I got the link from the milw0rm site here:
http://milw0rm.com/exploits/7718

( Hopefully you don’t mind I post the links on here, they seem to be a whitehat group:
http://en.wikipedia.org/wiki/Milw0rm )

tygern8r – #140

This is the most AWESOME collection of helpful info, along with Askapache.com I have found and want to thank Jeff and everyone who posted helpful comments. After much trial and error, I finally got the desired result on a shared linux host with GoDaddy, and wanted to share my results.

I use a much less restrictive list with only these:

#custom 403 that writes timestamp, IP, and request to a log file and then redirects to porn site - just for fun
ErrorDocument 403 /403.php

RedirectMatch 403 /*\=
RedirectMatch 403 \/\/

#block injection attempts with =http in query string
RewriteCond %{QUERY_STRING} ^(.*)=http: [NC]
RewriteRule ^(.*)$ - [F,L]

#WP permalinks by post number
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

Custom Apache logs don’t work for me on GoDaddy, but I can make the 403.php write a log file. Now none of this is filling up my BadBehavior logs (which was catching all of them, but it was annoying having to look at them everyday).

This setup requires a custom 403 or it doesn’t work. Thanks again!

Peekay – #142

PHPWCMS users may need to remove the pipe character from the redirect match section and also remove it’s matching %7C from redirect-match and also from the query-string exploits section. PHPWCMS uses a pipe in some of the file manager requests.

spirit – #143

That sounds great! Did someone rewrite the 4G Blacklist to be compatible witn nginx webserver ?

Ken Dawes – #144

Hi Jeff,
This 4G is way great!

Do you know anything of the bozos going by the name of the Palestinian_hackerz North Storm Team? Will 4G address their attacks?

Is 5G near?

“Digging into Wordpress” is fantastic! I have the original pdf, missd the 1st ed print… but will be getting the 2ed as soon as it’s available!

Ken

jermaine – #146

This looks pretty good! 4G may be the best….

shoulders – #147

I am not a complete noob, but still learning.

In the section # QUERY STRING EXPLOIT is it possible to put a line in for an exception for a particular URL so all the rules apply but not for that URL or URL with a particular pattern.

–

Does using %{QUERY_STRING} mean that you do not have to write seperate rules for %{HTTP_REFERER}, %{HTTP_COOKIE}, %{REQUEST_FILENAME} and so on.

Thanks

Marc – #149

Is mod_alias really needed in addition to mod_rewrite? My service provider claims that I don’t need it to use this blacklist, but I can’t help thinking that I’ve been told incorrect information by the support staff.

Could you include instructions for enabling it?

matt – #151

Just implemented this on a new site we’re developing, The core site is built on ModX, with sef urls enabled, and bolted on is a webshop that we custom built from the ground up. We’re running it on an (mt) DV 3.5 Apache / Linux setup and everything is running flawlessy! No editing required.

Really, this is an incredible contribution. We’ve just moved to a dedicated server and I’m running in tin-foil hat mode. I’m really pleased to have stumbled accross this. Big Thanks!

Matt.

Roger – #152

Very impressive! Thanks for the hard work. Silly question but (novice here) as soon as I implemented this, my wordpress installation loads fine but the rest of our HTML pages return a 403 error. Removing the “QUERY_STRING-EXPLOITS” section returns things to normal… What obvious thing am I missing?!

matt – #153

I’ve been playing with this a couple of days and took it a little further. Recently I implemented a little spider trap that catches bastard bots ignoring robots.txt directives. The script grabs the bot’s IP and adds a directive into our .htaccess instantly blocking that IP from the entire site. (yes, only a short term solution but at least it ends the session until they switch IP).

As an addition to the excellent 4G Blacklist I’m sending the user to a custom 403 which does a similar job, on the first “403 access denied” the IP gets logged to a DB, if another 403 gets triggered by the same IP within 30min, the IP gets added to our .htaccess which blocks that IP from the entire site, an email gets sent alerting me that the 403 has banned someone. This is great for catching script kiddies feeling around for an XSS venerability, sure the skiddies can keep switching IP but that’s a pain, and they’ll still have to get through the rest of our security. You can be more or less generous with the time period and the strike outs, for us, we don’t expect a regular user to trigger a 403 twice in one session but I guess you could set it as high as 10 strikes before a ban is put in place. We’ll clear the bans every X number of days to lower the chances that any innocent users on a dynamic IPs don’t get caught up.

The new site isn’t live just yet, but I’m looking forward to testing this in the wild!

forum software from xmb – #154

Works well with xmb forum but have to comment out the following directive, ie:

# RedirectMatch 403 display\.

as the 403 display directive was blocking forums from being loaded via forumdisplay.php

Thanks for the great work.

Vincent – #155

Hi,

Just wanted to add a fix that causes major problems to Wordpress such as block of timthumb images, highligt and paste certain words, published buttons and tabs can’t be pressed…etc. I had to manually went over each item on the list and found the culprit, “RewriteCond %{QUERY_STRING} http\: [NC,OR]”. Just add a pound sign in front of it, and your Wordpress should work normally. I’ve tried this on three of my Wordpress websites that had similar problems.

Vincent – #156

Forgot to say thanks. Thanks..:)

Shane – #157

Hi, i used the list, broke it into two as directed, made the joomla adjustments, took out a few yahoo and google exceptions, and it seemed to work fine. That’s in IE8, FF, Opera etc.

However, two weeks down the track I happened to test the site in IE7, and it came up with a 403 forbidden access error. No authority to access this server … error while attempting 500.

This only happened in IE7.

As mine is a tourist portal, ranking no.1 in google, it was important the site was operational, so i’ve removed the entire list for now and it works in IE7.

I’m wondering what the issue could be, any thoughts anyone?

Igor – #158

Does anyone know whether the 4G list plays nice with Phorum, an open-source message board system?

I would like to thank Jeff for his useful, well-written and informative site. I refer to this site often and consider it one of the best for design tips for intermediate level or above.

Garrett W. – #159

Tell you what, Igor — how about you try it out and let us know if there are any problems — that way Jeff will have that knowledge for his 5G list ;)

Igor – #160

Hi Garret, will do! I am getting ready to test it out myself, because I think the potential benefits may outweigh the potential problems. I was also interested to learn about Project Honeypot, mentioned above. Thanks and I will be back to learn more and share whatever I discover in my experiment, but it may be a week or so…

Peter – #161

This seems is a GREAT blacklist for security. However, as someone who has no understanding of code or website stuff aside from simply setting them up…how can I implement this?

It seems like you really have to customize your backlist to your site, which requires extensive understanding of which lines of codes need to be taken out, edited, etc.?

I wouldn’t be able to do that because I don’t understand any of it…so how can I install this blacklist easily?

Thanks!

Shane – #162

@Peter, yes, you do need to do some customisation for your site.

Go thru all the posts above, and make notes on what’s relevant for you as you go.

Most important is to take the advice, in an early post, about splitting the list into two lines, instead of one.

Other than that it’s easy, just literally copy and paste the code into your .htaccess file and see what happens.

I still havent solved my issue of IE7 displaying error page, and yet it works in all other browsers and versions.

good luck
Shane

Igor – #163

After installation, I detected sluggishness in the server response time, possibly due to the fat .htaccess which slows things down a bit.

In Phorum, I received 403 errors trying to log off and trying to register, due in one case to rejection of ?0 and in another case due to rejection of /register something or other. For this reason, I do not recommend 4G for other admins that host forum software of any kind. I deleted entire paragraphs in 4G rather than single lines, because there is the chance that there may be other “gotchas,” and I am working with a live site. So I wound up gutting the 4G blacklist. What I have remaining is this, which is little different from what I had before:

# ESSENTIALS
RewriteEngine on
ServerSignature Off
Options -Indexes

### Note: good stuff, all, though I already had this in place before.

# FILTER REQUEST METHODS
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK) [NC]
RewriteRule ^(.*)$ - [F,L]

### I did not detect any problems from this, but on the other hand I have not found these methods being used in my system log either, and I have yet to read any explanation as to why these methods need to be filtered. I wonder too if filtering them will interfere with the site statistics utility. I suspect that is “highly likely” based upon my experience, so these too will be removed.

# BLACKLIST CANDIDATES
Order Allow,Deny
Allow from all
Deny from 75.126.85.215
Deny from 128.111.48.138
Deny from 87.248.163.54
Deny from 84.122.143.99
Deny from 210.210.119.145
Deny from 66.74.199.125
Deny from 203.55.231.100
Deny from 24.19.202.10

### No problem here either. But IP candidates seem dubious as they are two years old anyway. There are many black hats and they use multiple IP addresses. Black hats do not generally like to be pinned down to a few IPs, though it would certainly be nice if they would agree to that.

—

On my host, causes an internal server error (500) and had to be removed. It is not necessary anyway as my host supports these modules.

btw: a bit of documentation as to the reasoning for each of the various filters would be most helpful.

Lane – #165

Hey Jeff, What are your thoughts on this technique for speeding up htaccess?
http://www.chicagostyleseo.com/2010/02/speed-up-the-rewrite-engine-for-wordpress/

“Notably, the unnecessary and potentially-problematic container is completely removed… “

Igor – #166

Thanks Jeff. I wanted to emphasize again, your site has proven very useful to me in terms of configuring my .htaccess file and learning about CSS. You are doing a great service to the community here.

btw, someone has stolen your content:
http://www.SPAMSITEkingf1SPAMSITE.com/2010/03/21/stupid-htaccess-tricks.

Remove SPAMSITE to access. That appeared just below your site in Google results, and I clicked on it by accident because it seemed to have a more recent date.

Here is the bot-banning portion of my .htaccess, derived in part from your suggestions.

# FILTER REQUEST METHODS
#
#I kept these in anyway because they do not appear to cause any harm
#
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK) [NC]
RewriteRule ^(.*)$ - [F,L]

# BLACKLIST CANDIDATES
#
# These I detected manually by analyzing my log file and
# referencing other sources on the web
#

Order Allow,Deny
Allow from all
Deny from 91.205.96.19 "#2010-04-22: www.SPAMSITEpuritysearchSPAMSITE.net
Deny from 81.52.143. "#voila
Deny from 193.252.149. "#voila
Deny from 78.26.187.62 "#ukraine bot?
Deny from 89.149.254.73 "#Turkish bot?
Deny from 66.33.235.24 "#port scanner
Deny from 89.122.29. "#too many javabots

# BLACKLISTED USER AGENTS
#
# The first two are my choices, the rest yours, but
# some of your choices were detected in my log file
# as well. In particular, Nutch, Jakarta are common.
# I get thousands of hits from Voilabot, but no visitors!
#
SetEnvIfNoCase User-Agent "VoilaBot" keep_out
SetEnvIfNoCase User-Agent ^Java keep_out
SetEnvIfNoCase User-Agent ^$ keep_out
SetEnvIfNoCase User-Agent "Y\!OASIS\/TEST" keep_out
SetEnvIfNoCase User-Agent "libwww\-perl" keep_out
SetEnvIfNoCase User-Agent "Jakarta.Commons" keep_out
SetEnvIfNoCase User-Agent "MJ12bot" keep_out
SetEnvIfNoCase User-Agent "Nutch" keep_out
SetEnvIfNoCase User-Agent "cr4nk" keep_out
SetEnvIfNoCase User-Agent "MOT\-MPx220" keep_out
SetEnvIfNoCase User-Agent "SiteCrawler" keep_out
SetEnvIfNoCase User-Agent "SiteSucker" keep_out
SetEnvIfNoCase User-Agent "Doubanbot" keep_out
SetEnvIfNoCase User-Agent "Sogou" keep_out

Order Allow,Deny
Allow from all
Deny from env=keep_out

I get some activity from Yodaobot, TurnItIn (a bot supposedly checking for student plagiarism), and ia_archiver, too, but haven’t banned them yet. I know ia_archiver is a non-profit, but I’m not too thrilled with their purpose, as I don’t necessarily want old pages being stored somewhere “for research purposes”.

iambic5p – #168

We’re really finding this useful, stunning work, thanks for sharing…

So far no problems except it seems to have killed stone dead our Mint Stats, so far I’ve tracked it down to this line…

RewriteCond %{QUERY_STRING} ^.*(globals|encode|config|localhost|loopback).* [NC,OR]

…and removed “encode” needed for the end of the query string mint produces. Actually Mint requires “encoded” is there a way to allow “encoded” through and prevent “encode”?

iambic – #170

Jeff, encode$ works perfectly.

Collect one beer.

Adam – #171

Hi , this looks like an amazing resource. Problem is i ran it through my htaccess syntax checker and it came up with 16 errors. any idea why?

the link is above

victoria – #173

I had someone from 89.248.168.41 try out the following today..

/index.php?cat=999+UNION+SELECT+null,CONCAT(666,CHAR(58),user_pass,CHAR(58),666,CHAR(58)),null,null,null+FROM+wp_users+where+id=1/*

/index.php?cat=%2527+UNION+SELECT+CONCAT(666,CHAR(58),user_pass,CHAR(58),666,CHAR(58))+FROM+wp_users+where+id=1/*

So instantly started to wonder at how I could block these annoying idiots from trying to access parts of the site they shouldn’t and came across your site! Fantastic!

Thank you

Adam – #174

Hi, Jeff thanks for reply to that htaccess syntax checker. Just curious i know this code come with no implied guarantees at all but do you personally use this code for any clients sites. How confident are you in recommending it to someone to use on a moneymaking site.

Steve – #175

Whoa! Powerful stuff. Blocked me from my own site. (403)

Then through the server for a backwards loop. LOL must have thought I was a bad guy. Took out the code and worked normally.

# Enabled Apache module: mod_alias
# Enabled Apache module: mod_rewrite

Not sure if the above are enabled by the Host or if I can go into phpMyAdmin or add something I haven’t?

Garrett W. – #176

Steve: Normally mod_alias and mod_rewrite are enabled by the host, since their usage is widespread. phpMyAdmin has absolutely nothing to do with them.

Igor – #177

Steve: I got a 500 Internal Server Error until I took out all of the IF statements checking for module.c. I also took out the sections checking for exploits due to conflicts with the Phorum message board system. For me, the best part of 4G are the IP bans and the User-Agent bans. Very useful framework to work with and excellent suggestions on User-Agents, as well. I added a link to this site on our links page.

Steve – #178

Thank you . .

I appreciate the tips. I know success is just on the other side of whatever this little hiccup is.

Have a good one

Steve – #179

cgi-bin
htodocs
_wpblog

Okay in htdocs there is an htacess file with “Options -Indexes” in it.

One folder below in my wpblog root there is the standard htaccess with
# BEGIN WordPress <IfModule mod_ etc, etc . . # END WordPress

Which am I supposed to be working with here and applying these mods to?

Igor – #180

Steve, “all of the them” is the answer to your question.

The server adheres to rules found in the uppermost .htaccess file, which would be your root ( / ). There is one notable exception to this rule. Whenever a child directory has its own .htaccess, then all of the parent’s rules are discarded and replaced by what is found there. Therefore, to implement the 4G sitewide, you must insert it in every .htaccess that allows public access. There is no point in inserting it in any .htaccess where you “Deny from All,” however, and you may not be quite so concerned about areas that require password validation–but the choice is up to you.

Steve – #181

Igor,

Okay I understand now. Do appreciate it .

Steve – #182

This is what I get.

Access Forbidden

Access denied. Please click on the back button to return to the former page.

Garrett W. – #183

Igor: so .htaccess files are only obeyed at the level closest to the resource being accessed, and not all the way up the chain?

My understanding was that .htaccess parsing/obedience started at the lowest directory level (i.e. closest to the filesystem root) containing such a file, and if that didn’t block or redirect the traffic, parsing continued at the next-higher .htaccess file in the hierarchy.
So I am incorrect?

Igor – #184

Garret, you are correct! I had erred on the side of caution and copied certain of the root’s .htaccess security elements to another .htaccess lurking within a subdirectory.

But I just did a little experiment and discovered that this was unnecessary! The root’s .htaccess parameters are indeed inherited when not specifically overridden by the local .htaccess.

Now one of my .htaccess is much shorter than before as a result of this new knowledge.

Thank you, Garrett.

Dave Stuttard – #185

Boys! Stop! 4G is a block of server (Apache) instructions to ADD to .htaccess, which can include other instructions such as redirects. 4G should just be in the .htaccess at the root where it will protect the whole website. Any .htaccess in a lower level directory only affects that directory and any directories below it (it does not cancel the main .htaccess - it adds to it for that directory only). Whenever any address on your website is requested (either from outside or via links on your pages), the instructions in .htaccess files are read by Apache before doing anything, so this introduces a delay and it is not necessary (or wise) to have multiple 4gs.

Igor – #186

Hey guys, how do I ban an unknown IP address that shows up in the logs as:

.

My host does not always provide an IP address. I have searched on Google for an answer to this literally for hours, and nobody seems to know anything. The only advice I found was to contact my host company. I’d rather just ban unknown IP addresses in .htaccess and be done with it. Has anyone else encountered this issue?

Dave – #187

Igor, if you see a hostname in your logs, you can go here and get the IP:

http://www.hcidata.info/host2ip.cgi

Or if its a User Agent you want to ban use that if it’s shown, eg:

RewriteCond %{HTTP_USER_AGENT} ia_archiver [NC,OR]
RewriteCond %{HTTP_USER_AGENT} nutch [NC]
RewriteRule ^.* - [F,L]

(in this example, any UA with ‘nutch’ anywhere in it will be banned, because there is no ‘^’ before ‘nutch’). Don’t forget the ‘NC’.

Igor – #188

Thanks Dave. I do indeed have many user agent bans. Jeff suggested plenty of good ones, and I used his list as the beginning of my own. Here’s what I was referring to in my comment, taken from my server log:

. - - [30/Apr/2010:08:28:36 -0600] "GET /messageboard/register.shtml HTTP/1.0" 200 6321 "http://www.google.com/" "Mozilla/4.7 (compatible; OffByOne; Windows 2000) Webster Pro V3.4"

As you can see, no IP address is specified. Where you see a “.” at the beginning of the line should be either an IP address or else a hostname. Here it is again:

. - - [30/Apr/2010:23:14:33 -0600] "GET /messageboard/login.shtml HTTP/1.0" 200 39966 "http://www.google.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; Alexa Toolbar; (R1 1.5))"

Another case is shown above. This has a different User-Agent; thus it would not appear possible to ban by User-Agent alone. My question is, how is it possible the IP address does not appear in the log (only a “.”)? It appears to me based upon the activity that this is a spambot attempting to post spam.

I have googled about this issue before, but have not found a method of banning bots that do not reveal their origin IP address.

Dave – #189

Igor, it’s not for me to rule, but I suggest that your requests for help concerning banning of specific IPs and UAs should be directed to something like http://www.webmasterworld.com/. Jeff’s 4G Blacklist does include a few of these, but only to supplement the main function of the blacklist, which is to catch morons in a more unique and general way - however, here’s a few more comments from me for what they’re worth that may help you:

I don’t know why an IP (or Hostname, which is just as good) is not shown sometimes…
I have cPanel and I look at ‘Latest Visitors’ each day (can’t be arsed to plough through server logs) to check for suspicious behaviour. Each entry shows me:

‘Host’, which may be either an IP or a Hostname or a mixture of the two or nothing.

The file requested with its code (200, 301, 403, 404, 410, etc).

‘Referrer’, which will be blank if it’s from a UA or the file on your website if it’s an internal link.

and ‘Agent’, which can be either User Agent details or a surfer’s browser details.

Then its all detective work and common sense deciding if any should be banned:

If an IP or Hostname is shown, check it out in http://www.hcidata.info/host2ip.cgi and do a reverse lookup there (that should give you a clue). And if an IP is shown, put it in your address bar to see if its a website (another clue). If neither are shown, move on…

If its a file that should normally only be requested by a link on your own website, any referrer should be a file on your website (duh!); legitimate direct requests (’Referrer’ blank) can be made by legitimate Search Engines (UAs) that you have not banned (Google, Slurp, MSN, etc). Of course an exception is where someone has used a ‘Favourites’ link in their browser.

Agent? This should tell you if its Google, Slurp, MSN, etc, or an unknown UA, otherwise it will be a string representing someone’s browser. If a URL is included, click on it to see who it is. If you can make out tne UA name, check it out in http://browsers.garykeith.com/tools/agent-checker.asp. This seems to be a good checker, based on a lot of research - it tells you if its banned or if its just a browser.

There may be other clues from behaviour on your website and if someone is fishing using rubbish URLs or just grabbing images, css files and js files. It’s a big subject, which Jeff has studied to make 4G and there will always be the clever twerps one step ahead of us (hiding their IP/Hostname!).

Dave – #190

Further to my last post, I’ve looked at Webster Pro - this UA needs to be banned as a spam harvester (I suggest ‘Webster’ is enough - they use Webster.Pro and Webster Pro). Alexa Toolbar; (R1 1.5) is from an IE6 browser and apparently not necessary to ban, but it’s interestingly the one looking at ‘login.shtml’! maybe they used Webster to harvest an email address, then tested it via a browser. I use captchas in contact forms, email validators in those and registration forms and I obfuscate email links, to minimise risk of being spambotted. Hope that helps.

Igor – #192

Thanks Jeff! I tried your third variation:

# FILTER REMOTES WITH NO IP ADDRESS
RewriteCond %{REMOTE_ADDR} ^.$ [NC]
RewriteCond %{REMOTE_ADDR} ^$ [NC]
RewriteRule .* - [F,L]

It did not cause any unexpected surprises. ;-) I’ll dig into my logs again next week and see whether Mr. Doesn’t-wanna-reveal-his-IP-address gets the 403 he deserves.

Dave: thanks for the info. I did check webmasterworld, and in fact they discussed this very issue a couple of times but didn’t offer any .htaccess solution. Thank goodness for Perishable Press! As for User-Agents, often they are forged, so a bot using a certain user-agent does not mean the user-agent is bad. I had googled Webster before and found a mixed verdict (once I got past all the references to the *Dictionary*), but will heed your advice.

I watch AWSTATS but like digging through the logs to get the real story. Never know what you may find. Much of the traffic AWSTATS claims is human is really bot. I was amazed by how many bots are hitting us all the time disguised as humans. Most of them go straight for the forums, trying to register and login with one second elapsing between attempts, so they are easy to spot. I have had some success dosing the dumber spambots with spam poison.

Igor – #193

Checked my logs today and observed that the .htaccess ban was ineffectual. About two dozen requests, some with completely different user-agents, had their IP resolve to “.” or “localhost”. I found an explanation as to how this can occur. See http://www.webmasterworld.com/forum10/8430.htm if interested.

Ken Dawes – #194

Hi,
Just an FYI…

I have been setting up a new WordPress installation for a client (hosted by Network Solutions if that has any relevance).

The “old” site has html pages in the root and an old Wordpress in a folder… I am installing a new WordPress into a new folder.

I have placed 4g into a .htaccess in the root with the following result… Root html files-unaffected, but both WP installations immediately go to a 403 error page.

I fiddled with the 4g and the problem resolves by commenting out the line
“ #RedirectMatch 403 \~ ”

I’m happy that this “fixes” the problem, but I’d love it if someone could explain the “why”.

Any clues?

Ken

Steve – #195

@Ken Dawes – #194

immediately go to a 403 error page.

I fiddled with the 4g and the problem resolves by commenting out the line
“ #RedirectMatch 403 \~ ”

I’m happy that this “fixes” the problem, but I’d love it if someone could explain the “why”.

Ken same problem I just noticed your fix and I did this a while ago and was finally allowed to log in.

Hopefully this #RedirectMatch 403 \~ isn’t a critical part of security. LOL .. oh man.

Ed – #197

Just to let you know I’ve found the following lines cause breakages in certain components - so if you use them, comment/remove the relevant parts…

- Image thumbail generation when given a complete image URL (ie including http://) in the ?src= query string of your thumbnail generator
…solution: comment out “RewriteCond %{QUERY_STRING} http\: [NC,OR]”
(in fact this would apply to any code that takes a complete URL as source…though obviously any code of that type would have to be very carefully designed to stop issues, so blocking http:// might not be a bad idea for most people)

- JQuery module jquery-ui-droppable - if referenced as part of the query string - eg: in Wordpress admin area, this reference call is made by the WooThemes framework custom navigation options page:
<script type="text/javascript" src="/wp-admin/load-scripts.php?c=0&load=hoverIntent,common,jquery-color,jquery-ui-sortable,jquery-ui-droppable&ver=">
…however I’m sure there are lots of other sites/cmses/plugins that use the JQuery drag and drop framework, not just WooThemes. Wp 3.0 might even use it for the new in-built menus, would have to check.
…solution: remove “|drop” from this line
“RewriteCond %{QUERY_STRING} ^.*(request|select|insert|union|declare|drop).* [NC]”
…or else change “|drop” to “|drop%20” to at least catch anything with a space after the word drop, but not words like “droppable”

Hope that helps someone! :-)

Peekay – #198

Coppermine users (depending on the version) may need to remove INSERT from the query-string section. It may block bulk image additions after FTP upload.

RewriteCond %{QUERY_STRING} (request|select|union|insert|declare|drop) [NC]

I have been using this for a few months now and I must say it has significantly reduced traffic from malicious bots. If you do get a ‘permission denied’ error, just check the query string to see if the application is using a word that is blocked in the blacklist.

Ed – #199

Found another issue with Wordpress 3.0.

The following line:

RewriteCond %{QUERY_STRING} ^.*(request|select|insert|union|declare|drop).* [NC]

breaks the image editing feature in the media section:

To fix, remove “select|” or else change it to “select%20|” to at least leave some of the protection.

Ed – #200

The comments filter removed part of the last post.

The line that gets broken is:

<script type="text/javascript" src="[removed]/wp-admin/load-scripts.php?c=0&load=hoverIntent,common,jquery-color,jquery-ui-core,jquery-ui-sortable,wp-ajax-response,imgareaselect,image-edit&ver=[removed]"</script>

Steve – #201

Seems to be something in the 4G that is preventing my Widgets panel from functioning properly. This is stuff such as can not drag and drop or open any drop down menus.

Any tips?

Garrett W. – #202

Steve: At least part of your problem might be resolved by removing or modifying the bit that blocks the word “drop” in the query string. See comment #110 or the end of comment #197.

Steve – #203

Garret thanks. I’ll check it out. Also though, I enabled accessibility mode (widgets) and that works fine with 4G.

Steve – #204

Garrett . .

Sorry, 2 T’s LOL

Steve – #205

Oh, How to apply 4G to SMF 2.0 RC3.

Advise or tips appreciated. Have a fresh install of this. “Will test for Tips”