#2 — Perishable

Good question.. adding it to the list was suggested by a fellow forum member awhile ago. I sniffed around Google a bit but found no evidence of maleficence, so I have removed it from the list. Thanks for pointing it out to us ;)

#4 — Perishable

John, rest assured that there are absolutely no googlebots mentioned in the blacklist. We are targeting the bad guys here, not legitimate agents such as Google, Yahoo!, and MSN. No worries! ;)

#6 — Perishable

Lisa you are wearing me out! :)

Anyway, to answer your question, yes, you can list multiple IP addresses in one line when using a “deny from ip” directive. Here is the syntax that you would use:

deny from 123.123.123 456.456.456 789.789.789

..etc. Notice that we are using spaces to separate the individual IPs — no commas allowed ;)

#8 — Perishable

Hei bjarbj78,

RewriteBase specifies the default URL for a directory. It is often used to remove the local directory prefix in order for the rewrite rules to act upon the correct file path. Don’t worry, if it sounds confusing, it is — I still find myself referring to the source when trying to explain it to people :)

Regards,
Jeff

#10 — Perishable

The question is, even if you could use htaccess to block over 9,000 domains, would you really want to? If you consider the potential performance hit and excessive load on server resources associated with the perpetual processing of such a monstrous list, it may inspire you to seek a healthier, perhaps more effective alternative..

For example, here is the code that I use for stopping 99% of the proxies that attempt to access one of my sites:

# prevent proxy access
RewriteEngine on
RewriteCond %{HTTP:VIA} !^$ [OR]
RewriteCond %{HTTP:FORWARDED} !^$ [OR]
RewriteCond %{HTTP:USERAGENT_VIA} !^$ [OR]
RewriteCond %{HTTP:X_FORWARDED_FOR} !^$ [OR]
RewriteCond %{HTTP:PROXY_CONNECTION} !^$ [OR]
RewriteCond %{HTTP:XPROXY_CONNECTION} !^$ [OR]
RewriteCond %{HTTP:HTTP_PC_REMOTE_ADDR} !^$ [OR]
RewriteCond %{HTTP:HTTP_CLIENT_IP} !^$
RewriteRule .* - [F]

Place in root htaccess and take ’er for a spin via any proxy service(s) of your choice.. - it may not be perfect, but it’s lightweight, concise, and very effective ;)

#13 — Perishable

Hi Bjørn,

Glad it worked for you! I have found it to be much more efficient than trying to block the endless swarm of proxy servers individually — things change waay too quickly for that. As for the additional rewrite conditions you mention, I have not used them.. perhaps you could test their effectiveness and share the results with us? ;)

Regards,
Jeff

#15 — Perishable

Hi Lisa :)

Using htaccess, wildcards are not necessary when specifying specific ranges of IP addresses. Simply truncate the address to the target range, for example:

deny from 74.54.

..would block every IP beginning with the 74.54. prefix:

74.54.1
   .
   .
   .
74.54.255
74.54.255.1
   .
   .
   .
74.54.255.255

..etc. Of course, you could also use wildcards if so desired:

deny from 74.54.*

..which would block the exact same range as before. It’s totally your choice! :)

#16 — Perishable

Ah, one more thing that I should point out.. Be careful when using the “dot” (.) in your blocked IP ranges. If we omit the dot from the example in the previous comment, we would block a different set of IP addresses:

74.540
   .
   .
   .
74.549
74.540.1
   .
   .
   .
74.549.255

..etc. Inclusion or exclusion of the dot can make all of the difference!

#20 — Perishable

Pádraig Brady,

Yes, that is an excellent idea. I am currently in the process of building the next version of the Ultimate htaccess Blacklist (v.3), and will definitely post an article once it is finished. As you can see, this post features the second version of the Blacklist, and I continue to post many useful tips and tricks concerning all things htaccess, PHP, XHTML, CSS, and all of the other web-related acronyms ;) Thus, my advice to people who want to stay current with the Ultimate htaccess Blacklist as well as other security and anti-spam information is to subscribe to my feed — I guarantee you won’t regret it!

Regards,
Jeff_

#22 — Perishable

Hi Peter, you are correct — you must remove the RewriteBase / directive when using the blacklist in your httpd.conf file. It is meant for per-directory htaccess files and is not needed when working with httpd.conf.

#25 — Perishable

Thanks for the tip, Max. I will definitely include this modification in the next iteration of the blacklist. Cheers! ;)

#27 — Perishable

Sure, I am more than willing to add it the next incarnation, provided some sort of explanation as to why “anonymouse” (or any other user agent for that matter) deserves to be blacklisted. Drop a link and we’ll go from there.. ;)

#29 — Perishable

Yes, I have also read elsewhere that blocking anonymouse is a wise move. I will add the directive to the next incarnation of the blacklist. Thanks for helping to improve the list! :)

#31 — Perishable

Hi John, I’m glad you enjoy the article. As for the blacklist, are you sure that it is the cause of the problem? It may very well be, however, many people use the list without any similar issues. Nonetheless, if you have determined that the source of the conflict involves the blacklist rules, a bit of further testing should help you target the problematic user agents. First, remove the first half of the user agents and test if things are working. If so, replace half of the removed agents and test again. Likewise, if the issue did not resolve after removing the first half, repeat the process with the second half. By repeating this process a few times (it shouldn’t take too long), you should be able to identify the conflicting agents. I hope that makes sense. I would do this exercise myself, but I am literally swamped with a million other tasks and chores that I have to get done. In any case, I hope this information is useful, as I definitely want to help you implement successfully the ultimate htaccess blacklist. Finally, if you do find problematic user agent(s), please drop a comment and share with the community here at Perishable Press.

Cheers,
Jeff

#33 — Perishable

Hi John, thank you for taking the time to share this information. I have removed the DA character string from the list and updated the article so that others may benefit from your work (and generosity). Thank you for helping to improve the quality of the blacklist! :)

Kind regards,
Jeff

#35 — Perishable

Doh! I completely forgot about Max’s comment concerning the DA string! Apparently age is directly proportional to forgetfulness (I should have scrolled thru the comments before responding) — would’ve saved everyone some time. Max’s solution also enables us to continue blocking the DA agent while allowing access to the various validation services. Once I return to the computer (this weekend), I will update the article and the blacklist with this information. Huge apologies to Max — thanks for the gentle reminder ;)

#36 — Perishable

The blacklist and article have been updated to (re)include the DA$ character string. Many thanks to Max for pointing this out. Cheers!

#39 — Perishable

Yes, there are many, many such programs. The blacklist attempts to block a decent number of them, but there are still many more to identify and add to the list. If you happen to notice a few particular, repeat mass-downloaders in your access logs, I recommend targeting them directly and/or adding their associated user-agent(s) to the blacklist. I hope this helps!

#41 — Perishable

Absolutely, Dave — apathetic and/or ignorant webmasters are exactly what spammers, crackers and other scumbags are targeting. The bad news is that such vulnerable sites become an even larger threat once compromised. What’s that old saying? “An ounce of prevention is worth a pound of spam..” ;)

#43 — Perishable

@flash arcade games: that is the first time I have heard of that particular type of site “hijacking.” The PHP code itself is fairly simple, but implemented sitewide on a busy site, it could reduce performance. Every request for every page on your site would be running the script. Thus, unless you have a small site and/or lite traffic, I would only use that technique if I discovered that my site was being ripped. In any case, it is very interesting — thanks for pointing it out!

#45 — Perishable

@ronaquin496: Unfortunately, there is no way to block all web proxies. If a page exists publicly on the Web, it is accessible by anyone (depending on skill level, etc.) who desires access. Of course, the blacklist presented above won’t block hidemyass.com because it’s not on the list.

#49 — Jeff Starr

Let’s try to keep it on-topic, Pablo. Thanks! :)

#51 — Jeff Starr

Alright, I suppose if you look at previous comments, your question does seem relevant. Sorry for jumping at you, just trying to keep the noise level down on these ongoing comment threads. Hopefully no hard feelings..

Looking at your question, it seems as if your site may have been Google-jacked or else compromised with some sort of redirection/proxy script. I wish I could say for sure, but there may be many things that you will need to check before identifying the culprit.

The first thing I would do is verify that all of the files on my server are intact and operating as intended. During that check, I would also keep an eye open for any files that may have been added, especially in higher-level directories. Also, a sitewide search for the term “tshake.com” and other related terms/URLs may prove very enlightening.

After verifying site integrity, I would then search for information on “Google-jacking” and follow any leads that are generated there. I would also check my site on other search engines as well; if the redirect is only happening for Google, it is likely due to hijacking.

I hope that provides some help, Pablo. My apologies for my previous response.

Regards,
Jeff

#53 — Jeff Starr

@Json: Careful with that HTAccess, Eugene! If you are actually contemplating using both versions of this blacklist, it may be best if you do a little more research on the subject to understand what you are doing. So to answer your question: “no”, by all means use the compressed version of the blacklist and call it good. Both versions would be around 90% redundant and thus wasteful of resources.

#56 — Jeff Starr

Json, you are the man! Thanks for finding and sharing that excellent resource with us. I didn’t realize that something like that was available on the Web. Many thanks!
Cheers,
Jeff

#58 — Jeff Starr

Are you referring to the code presented in this article, the no-referrers article, or the eight ways to blacklist article? You have commented on all three, and I am unable to locate any such code in the article above this particular comment thread. Please try to be specific — there are hundreds of posts and thousands of comments here at Perishable Press! ;)

#60 — Jeff Starr

I thought I answered this question already on the no-referrer comment thread.. But anyway, in a nutshell, the folder isn’t required at all unless you need to specify that particular version of the file. If you decide you need the folder, use a forward slash after its name. The backslash ( \ ) is an escape character that tells Apache to take the following character as a literal entity. Thus, in your second example, you would be escaping the letter “w”, which is pointless because it is already being taken literally. Let me know if this (and the other response on the no-referrer article) is not clear.