In a previous article, I explain how to redirect your WordPress feeds to Feedburner. Redirecting your WordPress feeds to Feedburner enables you to take advantage of their many freely provided, highly useful tracking and statistical services. Although there are a few important things to considerbeforeoptimizing your feeds and switching to Feedburner, many WordPress users redirect their blog’s two main feeds — “main content” and “all comments” — using either a plugin or directly via htaccess. Here is the htaccess code as previously provided here at Perishable Press:
We all know how important it is to deliver sensible, helpful 404 error pages to our visitors. There are many ways of achieving this functionality, including the well-known htaccess trick used to locally redirect users to custom error pages:
..and so on. These directives basically tell Apache to deliver the designated documents for their associated error types. Many webmasters and developers employ this trick to ensure that visitors receive customized error pages that are generally more user-friendly or design-specific than the rather unfriendly Apache defaults. Serving custom error pages is an excellent way to enhance overall site usability and accessibility, but there are several other useful techniques to consider as well.
Welcome to the Perishable Press “Blacklist Candidate” series. In this post, we continue our new tradition of exposing, humiliating and banishing spammers, crackers and other worthless scumbags..
Imagine, if you will, an overly caffeinated Bob Barker, hunched over his favorite laptop, feverishly scanning his server access files. Like some underpaid factory worker pruning defective bobble heads from a Taiwanese assembly line, Bob rapidly identifies and isolates suspicious log entries with laser focus. Upon further investigation, affirmed spammers, scrapers and crackers are swiftly blacklisted from future access. For the most heinous offenders, we suddenly hear Rod Roddy’s guzzling voice echo throughout the room:
“Candidate number 2008-03-09, COME ON DOWN!! — you’re the next contestant to get blacklisted from the site!”
Every week, I dig through my access and errorlogs to learn from the spammers, scrapers, and other cracker whores. Typically, attempts to exploit potential security vulnerabilities demonstrate the following characteristics:
Since posting the Ultimate htaccess Blacklist and then the Ultimate htaccess Blacklist 2, I find myself dealing with a new breed of maliciousattacks. It is no longer useful to simply block nefarious user agents because they are frequently faked. Likewise, blocking individual IP addresses is generally a waste of time because the attacks are coming from a decentralized network of zombie machines. Watching my error and access logs very closely, I have observed the following trends in current attacks:
User agents are faked, typically using something generic like “Mozilla/5.0”
Each attack may involve hundreds of compromised IP addresses
Attacks generally target a large number of indexed (i.e., known) pages, posts, etc.
Frequently, attacks utilize query strings appended to variously named PHP files
The target URLs often include a secondary URL appended to the end of a permalink
An increasing number of attacks employ random character strings to probe for holes
Yet despite the apparent complexity of such attacks, they tend to look remarkably similar. Specifically, notice the trends in the following examples of (nonexistent) target URLs, or “attack strings,” as I like to call them:
Update 2010/07/07: Please visit the 2010 IP Blacklist for more current information.
Over the course of each year, I blacklist a considerable number of individual IP addresses. Every day, Perishable Press is hit with countless numbers of spammers, scrapers, crackers and all sorts of other hapless turds. Weekly examinations of my site’s error logs enable me to filter through the chaff and cherry-pick only the most heinous, nefarious attackers for blacklisting. Minor offenses are generally dismissed, but the evil bastards that insist on wasting resources running redundant automated scripts are immediately investigated via IP lookup and denied access via simple htaccess directive:
<Limit GET POST PUT>
order allow,deny
allow from all
deny from 123.456.789
</LIMIT>
Although many of the worst attacks happen in randomized, zombie-like fashion, I have found that individual IPs that are not blacklisted will return repeatedly until finally blocked. Yet, despite the short-term success enjoyed by denying access to the most malicious IPs, the long-term futility of such blacklisting reflects the temporary nature of this solution. In other words, I have found that blocking individual IPs is useful only for limited periods of time. Thus, every year, I gather my code and flush the blacklist of all individually blocked IP addresses. I then start fresh, adding the worst villains to the list, blocking entire IP ranges if necessary, and referring to previous versions of my htaccess files to cross-check suspiciously familiar entities. It is within this context, then, that I present the following manually assembled collection of over 150 of the worst spammers, scrapers, and crackers to hit my site in 2007.
During the recent ASO server debacle, I raced frantically to restore functionality to Perishable Press. Along the way, one of the many tricks that I tried while trying to fix the dreaded “white screen of death” syndrome involved increasing the amount of PHP memory available to WordPress. This fix worked for me, but may not prove effective on every installation of WordPress. If you are unsure as to whether or not you need to increase your PHP memory, consult with your host concerning current available memory 1 and overall compatibility with a localized increase. Note that if your blog is running fine, there is probably no need to employ this solution. I recommend this change only if you are experiencing PHP-related and/or PHP memory issues associated with any of the following:
This trick isn’t guaranteed to prevent all WordPress-generated PHP memory errors, but it certainly seems to help reduce their overall occurrence. For some reason, after my host upgraded their servers to Apache 1.3.41, I began logging an extremely high number of fatal PHP “memory exhausted” errors resulting from the WordPress cache.php script. Here is an example of the countless errors that are generated:
Welcome to the Perishable Press “Blacklist Candidate” series. In this post, we continue our new tradition of exposing, humiliating and banishing spammers, crackers and other worthless scumbags..
Scumbag number 2008-02-10, “COME ON DOWN!!” — you’re the next baboon to get banished from the site!
Like many bloggers, I like to spend a little quality time each week examining my site’s error logs. The data contained in Apache, 404, and even PHP error logs is always enlightening. In addition to suspicious behavior, spam nonsense, and cracker mischief, this site frequently endures automated and even manual attacks targeting various XSS exploits, WordPress vulnerabilities, and other potential security holes. Although the number of successful attacks remains relatively small, the very nature of some of the attacks serves to threaten site performance, security and stability. Such is the case of blacklist candidate number 2008-02-10: IP address 128.111.48.138.
How to streamline and maximize the effectiveness of your WordPress URLs by using htaccess to remove extraneous post-date information: years, months, and days..
Recently, there has been much discussion about whether or not to remove the post-date information from WordPress permalinks 1. Way back during the WordPress 1.2/1.5 days, URL post-date inclusion had become very popular, in part due to reports of potential conflicts with post-name-only permalinks. Throw in the inevitable “monkey-see, monkey-do” mentality typical of many bloggers, and suddenly an entire wave of WordPressers had adopted the following permalink structure:
/%year%/%monthnum%/%day%/%postname%/
The benefits of using this format are primarily organizational in nature. Post-date information that is “built-in” to every URL provides immediate, “at-a-glance” knowledge of post “freshness”. Looking ahead ten, twenty or even a hundred years into the future of the blogosphere, there will be trillions of posts and articles, each with their own unique URL. Archived copies of content may or may not include creation date: dynamically archived pages require deliberate database queries, while those archived statically may no longer have access to post-date data. Including post dates in permalinks provides permanent, facilitative record of content origination. Needless to say, most adopters of dated permalinks probably jump on board because the WordPress Admin makes it super-easy to follow the crowd.
In my previousarticles on PHP error handling, I explain the process whereby PHP error handling may be achieved using htaccess. Handling (logging, reporting) PHP errors via htaccess requires the following:
Access/editing privileges for htaccess files
A server running PHP via Apache, not CGI (e.g., phpSuExec) 1
Ability to edit/change permissions for files on your server
If you are having trouble handling PHP errors using htaccess, these three items are the first things to check. If it turns out that you are unable to use htaccess to work with PHP errors, don’t despair — this article explains how to achieve the same goals using local php.ini files. To implement this strategy, the following is required:
Permalink URL canonicalization is automated via PHP in WordPress 2.3+, however, for those of us running sites on pre-2.3 versions or preferring to deal with rewrites directly via Apache, comprehensive WordPress URL canonicalization via htaccess may seem impossible. While there are several common methods that are partially effective, there has not yet been available a complete, user-friendly solution designed specifically for WordPress. Until now..
In this article, I share my “secret” htaccess URL canonicalization formula. I originally developed this method in July of 2007, and have been using it successfully on a variety of WordPress-powered sites since that time. Thus, having verified the effective performance of this technique, I feel confident in sharing it with the open-source community. But first, a bit of context..
In my previous article on logging PHP errors, How to Enable PHP Error Logging via htaccess, we observed three fundamental aspects of preventing, preserving, and protecting your site’s PHP errors:
Prevent public display of PHP errors via htaccess
# supress php errors
php_flag display_startup_errors off
php_flag display_errors off
php_flag html_errors off
php_value docref_root 0
php_value docref_ext 0
Preserve (log) your site’s PHP errors via htaccess
# prevent access to PHP error log
<Files PHP_errors.log>
Order allow,deny
Deny from all
Satisfy All
</Files>
Now, in this article, we will explore these operations 2 in greater depth, provide additional functionality, and examine various implications. First we will explore PHP error handling for production environments (i.e., for websites and applications that are online, active, and public), then we will consider error handling for development environments (i.e., for projects that are under development, testing, private, etc.).
Come one, come all — today we officially begin a new series of posts here at Perishable Press: the public exposure, humiliation, and banishment of spammers, crackers, and other site attackers. Kicking things off for 2008: blacklist candidate number 2008-01-02!
Every Wednesday, I take a little time to investigate my 404errorlogs. In addition to spam, crack attacks, and other deliberate mischief, the 404 logs for Perishable Press contain errors due to missing resources, mistyped URLs, and the occasional bizarre or even suspiciousbehavior of the search-engine robots. Whenever possible, I attempt to resolve a majority of the “fixable” errors, either by restoring missing resources, adding an htaccess redirect, or by any other means available.
Having exercised this rigorous maintenance practice for well over a year now, my 404 error logs are almost completely devoid of all “fixable” 404 errors, and are filled almost exclusively with spam attacks, XSS attempts, and other miscellaneous cracker nonsense. Fortunately, my site has only fallen victim to such espionage on one occasion, and on a different server.
In this article, my goal is to help you optimize WordPress by replacing a few common plugins with their correspondingly effective code equivalents. As we all know, WordPress can be a very resource-hungry piece of software, especially when running a million extraneous plugins. Often, many common plugins are designed to perform relatively simple tasks, such as redirect a feed, display a random image, or return a database value. For those of us comfortable with editing PHP and htaccess code, there is no need to bloat WordPress with additional plugins for the sake of a few routine tasks. For each of the “pure code” alternatives presented below, we are able to drop an unnecessary plugin without editing the WordPress core. In fact, all of the plugin replacements presented here affect only theme files, thereby keeping WordPress updates nice and easy. Well, okay, one or two methods require editing your root htaccess file, but we are all okay with that, right?
In this brief tutorial, I will show Apache users how to suppress PHP errors from visitors and enable PHP error logging via htaccess.
Tracking your site’s PHP errors is an excellent way to manage and troubleshoot unexpected issues related to plugins and themes. Even better, monitoring PHP errors behind the scenes via private log is far better than trying to catch them as they appear at random visits. Thanks to the magical powers of htaccess, there is an easy way to implement this effective strategy.
Hide PHP errors from visitors
In our article, , we discuss a technique whereby PHP errors are suppressed via htaccess. This is done by including the following htaccess directives to your domain’s httpd.conf or to your site’s root (or other target directory) htaccess file:
..And we’re back. After an insane week spent shopping for a new host, dealing with some Bad Behavior, and transferring Perishable Press to its new home on a virtual private server (VPS), everything is slowly falling back into place. Along the way, there have been some interesting challenges and many lessons learned. Here are a few of the highlights..
The tide may be turning for A Small Orange
I am certainly not alone when I say that shopping for a new hosting provider and transferring websites is one of my least favorite aspects of web development. In my experience, switching hosts requires waay too much time and rarely unfolds without significant problems. Nonetheless, when service and/or support turns sour, upgrading to a better host is well worth the effort. In my case, A Small Orange just wasn’t working out.
Everything was going fine for the first several months — excellent service, fantastic support, and consistent, reliable server uptime. However, during the last several months, server uptime frequently dipped below the 98% level, an unacceptable amount of downtime, especially since it generally happened during critical times: peak hours or while I was trying to work on the site. When I finally submitted a support ticket addressing the “unacceptable levels of downtime,” ASO support staff put my mind at ease by moving my site to a “a more stable server.”
After implementing any of the hotlink-prevention techniques described in our previous article, you may find it necessary to disable hotlink-protection for a specific directory. By default, htaccess rules apply to the directory in which it is located, as well as all subdirectories contained therein. There are (at least) three ways to enable selective hotlinking:
Place hotlink images in an alternate directory
This method works great if your hotlink-protection rules are located in a directory other than the site root. Simply create another directory outside of the directory containing the htaccess rules and place your hotlink-allowed images into that directory.
Create a pseudo-rewrite rule for the target directory
If your images are located within the influence of your hotlink-protection rules, and you need to enable hotlinking for a specific directory, it is possible to circumvent the anti-hotlink rewriting for that directory. In the htaccess file for the target directory, add the following, pseudo-rewrite rules 1:
In this brief tutorial, we are going to enable users to access any file or directory of a site that is password-protected via htaccess. There are many reasons for wanting to employ this technique, including:
Share public resources from an otherwise private site
Enable visitors to access content during site maintenance
Testing and formatting of layout and design during development
As a webmaster, I have used this technique on several occasions. This trick works great for allowing access to any number of files, directories, and/or combination of both. We will begin with a generalized example, proceed with an explanatory discussion, and wrap things up with a couple of useful modifications.
When I wrote my article, Stupid htaccess Tricks, a couple of years ago, hotlink-protection via htaccess was becoming very popular. Many webmasters and bloggers were getting tired of wasting bandwidth on hotlinked resources, and therefore turned to the power of htaccess to protect their content. At that time, there were only a couple of different hotlink-protection methods available on the internet, and the functional difference between them was virtually insignificant. All that was necessary for up-and-coming bloggers-slash-site-administrators to eliminate leaking bandwidth and stolen resources was a relatively straightforward copy-&-paste procedure. Implementing the de facto htaccess hotlink protection code required a simple binary decision: “do you want hotlink-protection or not?”
These days, however, things are not so simple. Today, if you do a search on the internet for “htaccess hotlink protection”, you’ll get the phone book. There are countless mutations of the once tried-and-true htaccess code. There have been significant improvements, but there is also plenty of noise and error lurking among the countless techniques. These days, protecting your site’s assets and conserving bandwidth requires either trusting the first htaccess trick you find, or performing a mind-numbing and time-consuming amount of research to find the htaccess anti-hotlinking strategy that’s perfect for you and your domain. With this article, I do the grunt work for you — analyzing, deconstructing, and cannibalizing a contemporary collection of hotlink-protection methods to create the ultimate htaccess Anti-Hotlinking Strategy.
In our original htaccess blacklist article, we provide an extensive list of bad user agents. This so-called “Ultimate htaccess Blacklist” works great at blocking many different online villains: spammers, scammers, scrapers, scrappers, rippers, leechers — you name it. Yet, despite its usefulness, there is always room for improvement. For example, as reader Greg suggests, a compressed version of the blacklist would be very useful. In this post, we present a compressed version of our Ultimate htaccess Blacklist that features around 50 new agents. Whereas the original blacklist is approximately 8.6KB in size, the compressed version is only 3.4KB, even with the additional agents. Overall, the compressed version requires fewer system resources to block a greater number of bad agents.
Keeping track of your access and error logs is a critical component of any serious security strategy. Many times, you will see a recorded entry that looks legitimate, such that it may easily be dismissed as genuine Google fare, only to discover upon closer investigation a fraudulent agent. There are many such cloaked or disguised agents crawling around these days, mimicking various search engines to hide beneath the radar. Thus, it is a good idea to implement a procedure for scanning and checking select agents for authenticity. In general, the verification process involves a “forward/reverse” DNS lookup, which is then cross-verified with the search engine in question. Let’s have a quick look at how to do this..
First, visit and bookmark the following articles (and/or this article). These resources explain how to identify and verify the agents for each of the four major search engines: Google, Yahoo!, MSN/Live, and Ask.
WordPress users employing permalinks via htaccess to optimize their dynamic URLs transform complicated-looking links such as:
http://example.com/blog/index.php?page=33
..into search-engine friendly links such as:
http://example.com/blog/post-title/
Every rewritten URL relies on a common set of htaccess rules to transform the links. The htaccess rules for all WordPress permalinks look like this for root WP installations:
# BEGIN WordPress
<ifmodule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</ifmodule>
# END WordPress
..and like this for subdirectory WP installations:
Recently, I discussed the suspicious behavior recently observed by the Yahoo! Slurp crawler. As revealed by the site’s closely watched 404-error logs, Yahoo! had been requesting a series of nonexistent resources. Although a majority of the 404 errors were exclusive to the Slurp crawler, there were several instances of requests that were also coming from Google, Live, and even Ask. Initially, these distinct errors were misdiagnosed as existing URLs appended with various JavaScript functions. Here are a few typical examples of these frequently observed log entries:
Fortunately, an insightful reader named Bas pointed out that the errors were actually PHP functions. Bas explains:
The two functions (array_rand and opendir) you define as javascript functions are PHP functions. Some servers generate clickable links to the php manual (which uses function.NAMEOFFUNCTION in their URL’s) in php scripting error messages. Maybe that’s also the cause of these problems.
In our article Stupid htaccess Tricks, we present the htaccess code required for redirecting visitors temporarily during periods of site maintenance. Although the article provides everything needed to implement the temporary redirect, I think readers would benefit from a more thorough examination of the process — nothing too serious, just enough to get it right. After discussing temporary redirects via htaccess, I’ll also explain how to accomplish the same thing using only PHP.
![[ ~:{*}:~ ]](http://perishablepress.com/press/wp-content/images/2008/misc-chunks/redirect-deco.gif)
In a previous article, I explain how to redirect your WordPress feeds to Feedburner. Redirecting your WordPress feeds to Feedburner enables you to take advantage of their many freely provided, highly useful tracking and statistical services. Although there are a few important things to consider before optimizing your feeds and switching to Feedburner, many WordPress users redirect their blog’s two main feeds — “main content” and “all comments” — using either a plugin or directly via htaccess. Here is the htaccess code as previously provided here at Perishable Press:![[ Photo: Bob Barker Looking Sharp ]](http://perishablepress.com/press/wp-content/images/2008/blacklisted/2008-03-09.jpg "Come on down! You're the next contestant on the 'Blacklist is Right'!")
Imagine, if you will, an overly caffeinated Bob Barker, hunched over his favorite laptop, feverishly scanning his server access files. Like some underpaid factory worker pruning defective bobble heads from a Taiwanese assembly line, Bob rapidly identifies and isolates suspicious log entries with laser focus. Upon further investigation, affirmed spammers, scrapers and crackers are swiftly blacklisted from future access. For the most heinous offenders, we suddenly hear Rod Roddy’s guzzling voice echo throughout the room: ![[ ~{+}~ ]](http://perishablepress.com/press/wp-content/images/2008/misc-chunks/blacklist-deco.gif)
Since posting the ![[ ~:{*}:~ ]](http://perishablepress.com/press/wp-content/images/2008/misc-chunks/php-memory-deco.gif)
During the recent ASO ![[ Photo: Bob Barker points a finger ]](http://perishablepress.com/press/wp-content/images/2008/blacklisted/2008-02-10.jpg "Come on down! You're the next cracker whore to get blacklisted!")
Scumbag number 2008-02-10, “COME ON DOWN!!” — you’re the next baboon to get banished from the site!![[ ~*+*~ ]](http://perishablepress.com/press/wp-content/images/2008/misc-chunks/permalink-deco.gif)
Recently, there has been ![[ ~{*}~ ]](http://perishablepress.com/press/wp-content/images/2008/misc-chunks/php-error-deco.gif)
In my ![[ Image: Decorative Icon ]](http://perishablepress.com/press/wp-content/images/2008/misc-chunks/canonical-deco.gif)
Permalink URL canonicalization is automated via PHP in WordPress 2.3+, however, for those of us running sites on pre-2.3 versions or preferring to deal with rewrites directly via Apache, comprehensive WordPress URL canonicalization via ![[ Photo: Bob Barker makes a fist ]](http://perishablepress.com/press/wp-content/images/2008/blacklisted/2008-01-02.jpg "Come on down! You're the next cracker whore to get blacklisted!")
Every Wednesday, I take a little time to investigate my ![[ Photo: Macro shot of a Yttrium claw ]](http://perishablepress.com/press/wp-content/images/2007/misc-chunks/yttrium-plugin.jpg "Unplug Unnecessary Plugins")
In this article, my goal is to help you optimize WordPress by replacing a few common plugins with their correspondingly effective code equivalents. As we all know, ![[ Image: Illustration of two hands holding a glowing object ]](http://perishablepress.com/press/wp-content/images/2007/misc-chunks/hotlink-hands.jpg "Creation of the ultimate htaccess hotlink-protection strategy")
When I wrote my article, ![[ Image: Lunar Eclipse ]](http://perishablepress.com/press/wp-content/images/2007/misc-chunks/blacklist2-eclipse.jpg)
In our original ![[ Image: Abstract Mathematical Diagram ]](http://perishablepress.com/press/wp-content/images/2007/misc-chunks/redirect-diagram.gif)
In our article