Ultimate htaccess Blacklist

Published Thursday, June 28, 2007 @ 10:46 am • 62 Responses

[ Image: Solar Eclipse ] For those of us running Apache, htaccess rewrite rules provide an excellent way to block spammers, scrapers, and other scumbags easily and effectively. While there are many htaccess tricks involving blocking domains, preventing access, and redirecting traffic, Apache’s mod_rewrite module enables us to target bad agents by testing the user-agent string against a predefined blacklist of unwanted visitors. Any matches are immediately and quietly denied access.

There are many ways to obtain an effective htaccess blacklist. There are several excellent forums around the web that provide a plethora of priceless htaccess advice. Highly suggested. Additionally, after copying and pasting your favorite forum blacklist examples to your domain’s root htaccess file, you will want to continue with its development by tracking bandwidth thieves, comment spammers, and site scrapers and adding them to the list. Or, you may wish to skip the tedious grunt work and simply grab a copy of the Ultimate htaccess Blacklist!

The Ultimate htaccess Blacklist began as a short list of only the most heinous offenders. Blocking scum was such an enjoyable activity that we soon added to the list the identity of every nasty agent we could find. The result has been a very low-stress, spam-free site with virtually zero stolen bandwidth. The list is fairly comprehensive and attempts to blacklist as many site rippers, grabbers, spammers and bad bots as possible. While no blacklist could ever block them all (nor would they want to using this method)1, an elaborate htaccess blacklist can do wonders to improve overall performance, decrease site maintenance, and reduce server expense. Overall, we consider this blacklist a great foundation on which to build and customize your own ultimate htaccess blacklist!2

So without further ado, here is our version of the ultimate htaccess blacklist, as promised. Simply copy and paste the following code into the root htaccess file of your site to enjoy a serious reduction in wasted bandwidth, stolen resources, and comment spam. Don’t forget to backup your data and test everything, etc. — After that, you’re good to go!

The Ultimate htaccess Blacklist

# Ultimate htaccess Blacklist from Perishable Press
# Deny domain access to spammers and other scumbags
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} almaden [OR]
RewriteCond %{HTTP_USER_AGENT} ^Anarchie [OR]
RewriteCond %{HTTP_USER_AGENT} ^ASPSeek [OR]
RewriteCond %{HTTP_USER_AGENT} ^attach [OR]
RewriteCond %{HTTP_USER_AGENT} ^autoemailspider [OR]
RewriteCond %{HTTP_USER_AGENT} ^BackWeb [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bandit [OR]
RewriteCond %{HTTP_USER_AGENT} ^BatchFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^Buddy [OR]
RewriteCond %{HTTP_USER_AGENT} ^bumblebee [OR]
RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^CICC [OR]
RewriteCond %{HTTP_USER_AGENT} ^Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Copier [OR]
RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR]
RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
RewriteCond %{HTTP_USER_AGENT} ^DA [OR]
RewriteCond %{HTTP_USER_AGENT} ^DIIbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Wonder [OR]
RewriteCond %{HTTP_USER_AGENT} ^Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Drip [OR]
RewriteCond %{HTTP_USER_AGENT} ^DSurf15a [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^EasyDL/2.99 [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} email [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailCollector [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FileHound [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} FrontPage [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetSmart [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]
RewriteCond %{HTTP_USER_AGENT} ^gigabaz [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go\!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^gotit [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^grub-client [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} ^HTTrack [OR]
RewriteCond %{HTTP_USER_AGENT} ^httpdown [OR]
RewriteCond %{HTTP_USER_AGENT} .*httrack.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Indy*Library [OR]
RewriteCond %{HTTP_USER_AGENT} Indy\ Library [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^InternetLinkagent [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^InternetSeer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^Iria [OR]
RewriteCond %{HTTP_USER_AGENT} ^JBH*agent [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]
RewriteCond %{HTTP_USER_AGENT} ^JustView [OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^LexiBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^lftp [OR]
RewriteCond %{HTTP_USER_AGENT} ^Link*Sleuth [OR]
RewriteCond %{HTTP_USER_AGENT} ^likse [OR]
RewriteCond %{HTTP_USER_AGENT} ^Link [OR]
RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mag-Net [OR]
RewriteCond %{HTTP_USER_AGENT} ^Magnet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Memo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mirror [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*Indy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla*MSIECrawler [OR]
RewriteCond %{HTTP_USER_AGENT} ^MS\ FrontPage* [OR]
RewriteCond %{HTTP_USER_AGENT} ^MSFrontPage [OR]
RewriteCond %{HTTP_USER_AGENT} ^MSIECrawler [OR]
RewriteCond %{HTTP_USER_AGENT} ^MSProxy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetMechanic [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR]
RewriteCond %{HTTP_USER_AGENT} ^Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]
RewriteCond %{HTTP_USER_AGENT} ^Openfind [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^Ping [OR]
RewriteCond %{HTTP_USER_AGENT} ^PingALink [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pockey [OR]
RewriteCond %{HTTP_USER_AGENT} ^psbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^QRVA [OR]
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^Reaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Recorder [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Scooter [OR]
RewriteCond %{HTTP_USER_AGENT} ^Seeker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^sitecheck.internetseer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^SlySearch [OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snake [OR]
RewriteCond %{HTTP_USER_AGENT} ^SpaceBison [OR]
RewriteCond %{HTTP_USER_AGENT} ^sproose [OR]
RewriteCond %{HTTP_USER_AGENT} ^Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^Szukacz [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]
RewriteCond %{HTTP_USER_AGENT} ^URLSpiderPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^Vacuum [OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [OR]
RewriteCond %{HTTP_USER_AGENT} ^webcollage [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebEMailExtrac.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebHook [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebMiner [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebMirror [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]
RewriteCond %{HTTP_USER_AGENT} ^Webster [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
RewriteCond %{HTTP_USER_AGENT} WebWhacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Whacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]
RewriteCond %{HTTP_USER_AGENT} ^x-Tractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xenu [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus.*Webster [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus
RewriteRule ^.* - [F,L]

Footnotes

  • 1 Note: although this blacklist is highly effective at eliminating unwanted scum, its immense length requires extra processing and may affect the performance of your server. In our experience, employing this list (along with several other htaccess directives) for over two years has resulted in zero noticeable performance issues. Nonetheless, this may not be an ideal solution for sites with extreme levels of visitor traffic. [ ^ ]
  • 2 To begin building your own customized blacklist, you may want to check out the excellent list offered at joemaller.com. Thanks, Joe! [ ^ ]
  • 3 Update (October 14,2007): To reduce confusion and consolidate htaccess rules, the last two lines have been removed from the blacklist. These two lines are not required for the blacklist to work as intended:
    RewriteCond %{HTTP_REFERER} ^http://www.iaea.org$
    RewriteRule !^http://[^/.]\.perishablepress.com.* - [F,L]

Dialogue

62 Responses Jump to comment form

1Phil

July 15, 2007 at 3:32 pm

Probably just being dim, but I don’t get the final two lines (where the condition is that the referrer is http://www.iaea.org). Is there some spambot that pretends to be an incoming link from the IAEA?

2Perishable

July 15, 2007 at 4:24 pm

Yes, apparently there is a nasty bot from somewhere in Southeast Asia that uses iaea.org as the referrer. Not sure if it is still crawling the web — you may be safe removing it from the list..

3Phil

July 16, 2007 at 2:01 pm

OK, thanks.

I ran into a problem with the blacklist provided, however, and think there might be a typo.

The following line causes an internal error on my server. If I add a space after the DISCo slash, then it works fine again:
RewriteCond %{HTTP_USER_AGENT} ^DISCo\Pump [OR]

Thanks for the great blacklist. I’m now using it in place of the less comprehensive version I obtained from here:
http://www.javascriptkit.com/howto/htaccess13.shtml

Phil.

4Perishable

July 16, 2007 at 2:31 pm

Phil,

Yes, good catch. The DISCo condition should definitely have a space before the Pump:

RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Pump [OR]

I have corrected the blacklist so others should not experience any problems.
Thank you for your help!

Regards,
Jeff

5WillMacc

July 22, 2007 at 3:52 pm

Excellent blacklist for the htaccess file!
It even had a few on there I haven’t seen before! :)
I rant on this subject on my blog all the time:
http://www.a-daily-rant.com/

6Perishable

July 22, 2007 at 4:16 pm

WillMacc,

We are honored by your presence! Your work at http://www.a-daily-rant.com/ is excellent and much appreciated. I highly recommend your site for anyone serious about winning the war against spam, scrapers, and other online scum!

Many thanks!

7Greg

October 6, 2007 at 2:41 am

Thanks so much, with any doubt, the best list ever.Your site is fantastic !!
I’m wondering if you could privude the same list but in a “compress version” ?
Likethis for exemple: RewriteCond %{HTTP_USER_AGENT} ADSARobot|Anarchie|ASPSeek|Atomz|BackWeb|Bandit|…and so much more… with the [OR] and/or the [NC,OR] at the ends of lines.
Thank you agnain !

8Greg

October 6, 2007 at 2:46 am

You can see, wht I’m talking about here: http://www.toulouse-renaissance.net/c_outils/c_htaccess_compact.htm

;-)

9Perishable

October 6, 2007 at 8:21 pm

Greg,

By all means, that is a great idea. I will rewrite the blacklist and post a complete, compressed version featuring even more agents within the next week or so. Stay tuned..

10Robs

October 15, 2007 at 12:09 am

Thanks - just a heads up that this line causes the whole list to fail on my server (it ignores everything and lets every bot through). Comment it out and there is no problem

RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]

11Perishable

October 15, 2007 at 7:58 am

Robs,

Thanks for the heads up — may I ask what type of server/software are you using?

Thanks,
Jeff

12Perishable

October 15, 2007 at 12:26 pm

Greg,
Thanks again for the suggestion to release a compressed version of the ultimate htaccess blacklist. I finally managed to find the time to rewrite the list, and decided to add another fifty or so agents. Even with the new entries, the compressed blacklist is almost half the size of the original. Here is a link to the new article.
Cheers,
Jeff

13Greg

October 15, 2007 at 1:15 pm

Well done !

Your Welcome :-)

14Sanstenarios

October 25, 2007 at 6:19 pm

Hi,

glad to find up to date info about referer spam. If i get the point, referer spam is created using forged referer info in the http request, but can’t the user agent be forged also ?

15Perishable

October 25, 2007 at 7:37 pm

Sanstenarios,
Thanks for the comments. Unfortunately, any blacklist will only stop only those agents that admit to being on the list. Of course, forging identifying data such as referrer info and user agent is indeed possible, rendering virtually all such lists at least partially ineffective. Nonetheless, extensive blacklists such as the one provided in this article remain quite effective in denying access to spam whores and other scumbags. So, until that perfect solution manifests, we continue to employ any and all tools at our disposal.

16Sanstenarios

October 27, 2007 at 3:08 am

I set up the compress list and gonna check my logs ;)

Thanx

17khalifa

October 27, 2007 at 8:19 pm

great list thnx

18Perishable

October 28, 2007 at 3:27 pm

My pleasure!

19Lisa

December 11, 2007 at 6:04 pm

Wow! This is one big list :) By default, my .htaccess file already has the line “RewriteEngine on” and
“RewriteBase /”. I don’t need to rewrite them right?

I’m a little confused with how .htaccess works.

20Perishable

December 11, 2007 at 10:14 pm

Yes, Lisa, that is correct — you only need to specify each of those directives once per directory. If your htaccess file is in the root of your site, then you only need to declare them in that file, even if you place additional rewrite rules in subdirectories further downstream. The RewriteEngine on is simply telling the Apache server software to enable the rewrite module so that it can process rewrite rules, while the RewriteBase / declaration explicitly sets the base URL for per-directory rewrites.

21Proximuz

January 21, 2008 at 4:38 am

Hi.

Thanks for your list :) It’s your list up to date?

22Perishable

January 21, 2008 at 8:18 am

Hi Proximuz,

I last updated this list on October 15, 2007. Blacklists such as this are difficult to keep updated because the bad bots are constantly changing — there are new bots popping up every day, and old ones that simply disappear. Currently, I am constructing a fresh blacklist that I will post later this year.. until then, you may want to check out the Ultimate Blacklist 2 — last updated on November 5th, 2007, and also compressed for easier handling :)

23Proximuz

January 21, 2008 at 10:56 pm

Hi Perishable :)

Thanks for your answers:)
I’m gonna take a look at your link.

Thanks

24Perishable

January 22, 2008 at 9:00 am

My pleasure, Proximuz :)
Best of luck!! ;)

25Rasheed

February 24, 2008 at 11:03 pm

Hello Jef,

I suffer a lot from site scrappers.

I have this list in my site.

For test purposes i tried to (scrap) my site using Offline Explorer but it was not blocked.

I also tried the same thing with your site (just for 30 sec) and did not get Offline Explorer blocked.

What should i do to block it ?

How can i determine which software the scrapper is using ? The raw log file shows that ?

Thanks,

Rasheed.

26Perishable

February 26, 2008 at 6:07 pm

Hi Rasheed,

I feel your pain!
Perhaps this will work:

# block Offline Explorer
SetEnvIfNoCase User-Agent "Offline Explorer" keep_out
<Limit GET POST>
   order allow,deny
   allow from all
   deny from env=keep_out
</Limit>

It takes a different approach by using SetEnvIfNoCase, but I am not certain that “Offline Explorer” is the actual user agent for that client. If it doesn’t work, you may also want to try alternate names for the user agent.

I hope it helps!

27Rasheed

February 26, 2008 at 7:12 pm

Hi Jeff,

I found they changed the agent name to IE.

If i block it also Internet Explorer will be blocked (403).

Frustrating !

28Perishable

February 27, 2008 at 10:35 am

Ouch — That sucks! I guess I can remove that line from the blacklist now.. ;)

29Jeff Mez

February 28, 2008 at 12:24 pm

If you have RSS feeds, something in this list blocks Google and Yahoo from being able to read your RSS XML. Feedburner was still able to update. I put this on my site on Feb-16 and just today checked my feeds on My-Google and Yahoo. Both stopped updating on Feb-16. I commented out just this blacklist from my .htaccess file, waited about 30 minutes and sure enough, Google and Yahoo both updated to today. :\

30Perishable

March 2, 2008 at 9:17 am

Thanks for the input, Jeff. I will be looking into this. In the meantime, you may want to check out a lighter, “friendlier” type of blacklist. :)

31Peter

March 3, 2008 at 9:43 am

Hi
Does this even have to go into a .htaccess file? Can’t it be used more globally by having it in a apache config file?

32Perishable

March 3, 2008 at 9:47 am

Yes, the blacklist may definitely be placed directly into Apache’s httpd.conf file.

33Peter

March 3, 2008 at 10:01 am

Thanks for the follow up. With several domains its much easier to maintain doing this globally.

I log all activity to a database and the amount of 403 I’m getting is impressive between this and the 2G Blacklist.

:)

34Perishable

March 3, 2008 at 11:10 am

Absolutely! I love to watch the idiot bots bounce off the walls ;)

35HR Blog

July 9, 2008 at 2:16 pm

Thanks for this list and the updated list. I love watching all the denials in the logs.. Is that creepy?

36TechJammer

September 12, 2008 at 6:46 am

I have noticed that comment spammers are bypassing this security step, and just forging my sites own referer ID. Has anyone discovered a way to detect a forged referer ID?

37Mark

January 9, 2009 at 7:00 am

My ISP just upgraded to PHP5, and I discovered that an “old” version of this list was caused 500 server errors. I replaced it with the new list, and it’s smiles all around again. I’m just leaving this note as a a thank you and as a “heads up” to anyone else running into a PHP5 panic attack.
Thanks again !!!

38Json

March 11, 2009 at 2:26 am

Hi Mark

I get the same with both lists.
Which list did you use?

Cheers

39Json

March 11, 2009 at 2:45 am

Fixed, it was a syntax issue!

“Order allow,deny” not “order allow,deny” Case sensitive.

Cheers

40Mark

March 11, 2009 at 6:09 am

Hello !

I can’t remember now, but it seems you located the error already anyway,
so it’s all good now, I suppose :)

Thanks again,

Mark

41Jeff Starr

March 11, 2009 at 7:07 am

@Json: It might be because I just woke up, but I am not seeing where there are any instances of order allow,deny in the blacklist.. what am I missing here?

42Json

March 11, 2009 at 7:16 am

@Jeff Starr: Sorry I didn’t post any code. My 500 error was from the wrong case in “order - should be Order

Order allow,deny
allow from all
deny from eudora.com
deny from bravenet.com
deny from tripod.com
deny from lethadinan.pib.ir
deny from xanga.com
deny from iblogme.com

43Jeff Starr

March 11, 2009 at 8:23 am

@Json: Okay thanks, now I understand. But keep in mind that by posting that comment on this thread, you insinuate that the faulty code was obtained from, or otherwise related to, this article. As far as I can tell, no such directives are provided on either this post or in the related (compressed version) blacklist. In any case, I am glad that you have resolved the issue with your code. :)

44Json

March 11, 2009 at 8:33 am

Cool will do for next time. I still can’t figure out how to test it though.
I have searched for a .htaccess testing system but couldn’t find one.

Cheers

45John

June 9, 2009 at 8:55 pm

Silly Question: Couldn’t the spammers just set an user-agent of mozilla? or are you assuming they are not that smart?

46Jeff Starr

June 9, 2009 at 11:44 pm

@John: Not silly at all, really. As with any group of people, there are those with intelligence and those lacking in it. In my experience, there are many spammers who declare mozilla and other common user-agents to bypass filters, but the vast majority do not. Unfortunately, user-agent blacklists are based on assumptions, but they continue to prove useful nonetheless.

47John

June 10, 2009 at 2:54 pm

Hi Jeff,

Good deal, makes sense.

Thanks!
-John

48Milan

June 13, 2009 at 9:25 am

This seems very useful, thanks.

Unfortunately, my hosting provider (GoDaddy) doesn’t provide detailed logs, so it is hard to know how much difference this is making.

Are there any disadvantages to having a long .htaccess file, in terms of site performance?

49Jeff Starr

June 15, 2009 at 9:57 pm

@Milan: I have not tested this list of directives specifically (in terms of performance), but have seen much bigger lists (and htaccess files) in play that don’t seem to have much of a negative impact on performance. But then again, I’m not going to sit here and tell you that it doesn’t have an effect - the server has to process all of those matches for every valid page request, so probably not advisable for high-volume traffic sites and/or slow servers. One thing you can do to improve performance in general is to add the following line to your root htaccess file:

AllowOverride None

For more info on this method, see my article Stupid htaccess Tricks.

Subscribe to comments on this post


Share your thoughts..

TopRead official comment policy

Contact Perishable Press

  • Contact Jeff via form

Search Perishable Press

About Perishable Press

Perishable Press is the virtual playground of Jeff Starr — visionary, founder and lead developer of Monzilla Media, a small web and graphic design company in the lush desert oasis of Moses Lake, Washington. Perishable Press features articles and tutorials on many aspects of digital design..

Read more..

Perishable on Twitter

heading out of town on a photographic excursion. great way to break up the routine.

Perishable on Tumblr

Office painted a warmish/neutral off-white color. New 2.1...

Tue, 02 Feb 2010


New office digs..


Night view..


Potential..

Office painted a warmish/neutral off-white color. New 2.1 Altec/Lansing speakers for crisp, clear sound. Two new executive desks: one for computer biz and another workspace for drawing, painting, and other offline pursuits. And of course, a big, cushy black office chair to make it all sweet.

Quick Photoshop Reset

Tue, 19 Jan 2010

Very good Photoshop trick to know: If you hold down ALT+CTRL+SHIFT (Mac: CMD+OPT+SHIFT) while starting Photoshop you can reset all settings back to factory default. Very useful if you have problems with some tools or the interface. – http://bit.ly/4A5LJ5

Insane October

Sun, 01 Nov 2009

By far the most insane month of 2009, October included the following activities:

1st week: Trip to the East Coast, beginning with some business in Connecticut.

2nd week: East Coast trip continues with much pleasure in downtown Manhattan.

3rd week: Photo and art excursions with good friend visiting from Portland, OR.

4th week: Marathon book-editing and fine-tuning for Digging into WordPress.

Now that November is here, things remain busy, but I am hoping to get a chance to restore some balance and regain my equilibrium. Of course, the holidays are right around the corner..

Import Feeds to Facebook

Mon, 07 Sep 2009

Seems like a lot of misinformation and confusion out there on how to import and display your feeds on Facebook. Here is what worked for me:

1. In the lower left-hand corner of your Facebook account, click on “Applications” > “Notes”.

2. In the upper mid-right column, click on “Import a blog” in the “Notes Settings” panel.

3. In the “Import an External Blog” panel, enter your feed URL and check the little box.

4. Click the “Start importing” button and then click on “Confirm Import” on the preview page.

That’s all there is to it. Don’t forget to edit your “Notes Privacy” settings to ensure that people can see and comment on your imported feed items.

Once you successfully import your feed(s), they will appear by clicking on the “Notes” button in the left sidebar of your Home page. Also, your timeline or “Wall” will also display the most recent post from each of your feeds as they are published and pulled into Facebook. This makes it easy for your “Friends” to see what you have been up to elsewhere on the Web.

help me in plain english

Mon, 31 Aug 2009

This has got to be the most ironic comment I have ever read:

“hi i dun a stupid noooby mistake and dint think about encrytion i just put a pass in the change pass box and now when i attempt to see my main.php or index.php its sayin password no and error how can i reset back to having no password or were can i edit the bit so that a pass is automattically seen or if not posable how can i make it so i can put in the pass i made at some point so i can login this way? the 3rd is most prefered as this will help me with other projects i am planning as i am a php noob :s plz sum1 hu is clever help me in plain english”

Thanks, “jay” — you made my week with that one.

Read more on Tumblr..

Subscribe to Comments Recent Dialogue

  • kn33ch41: "But for good browsers such as Opera and Safari, most users are quite savvy, understanding the game and always keeping their browsers...
  • Paul: First off great list. Probably the one of the best I've ever seen. I enjoyed your limiting of The Wall, and Dark Side of the Moon. He...
  • kn33ch41: I use method three, and use method four when a selector only requires one property....
  • r-a-y: Great code. Just wondering if, in certain scenarios, a permalink can be longer than 255 characters. I can think of a weird exam...
  • Jeff Starr: Thanks Oliver! Glad you enjoy the book. Thanks for the great feedback :)...
  • Perry: I've read a number of these Image Preloading techniques today and would like to know the best approach to use. I use PHP 5.3 on th...
  • Infographiste PAO: I didn't know that Jeff, that's a kind of great news, usually ppl don't know about no or dofollow so now it's gonna be alright withou...
  • Bill Brown: @Jakub: Of course, there are many ways to bypass the XML declaration issue when you have PHP short tags enabled. Since the intenti...
  • bucabay: AT Jakub ...
  • Jakub Lédl: Oh sh*t, I came back. What happened to that code I posted? Does this CMS remove PHP from comments or what? I now look like an idiot :...

Read more recent comments..

Attention: Do NOT follow this link!