Plugin Sale! Save 15% on pro plugins with discount code: FALL2020
Web Dev + WordPress + Security

Best Practices for Error Monitoring

Given my propensity to discuss matters involving error log data (e.g., monitoring malicious behavior, setting up error logs, and creating extensive blacklists), I am often asked about the best way to go about monitoring 404 and other types of server errors. While I consider myself to be a novice in this arena (there are far brighter people with much greater experience), I do spend a lot of time digging through log entries and analyzing data. So, when asked recently about my error monitoring practices, I decided to share my response here at Perishable Press, and hopefully get some good feedback concerning best practices for error monitoring. Here is my email response to the question:

Hi Michael,

I don’t know if there is a “best” way to monitor error logs. There are many different ways to do it, and you just have to pick the method(s) that works best for you and provides the information you need. For me, I take the process to be a bit of an art form, with the end result being an improved understanding of the traffic patterns relating to my domains.

That said, the methods I use to track errors are three-fold:

  • First, I keep a close eye on PHP errors with a few lines in my HTAccess file (Linux/Apache)
  • I also like to watch my Apache error log, which is available through cPanel on a shared host
  • Most importantly (to me), I keep a tight watch on all 404 errors via a custom 404 script

If you would like more information on any of these methods, let me know and I will do my best to share them with you.

As for monitoring and analyzing, I do everything manually, line by line. I manage a large number of sites and spend around 2 hours per week checking things out, taking notes, and fixing or locking things down as needed. I began this process around four years ago with a single site. As I got better analyzing the 404 error log — which includes URL, referrer, query string, host, IP, and several other metrics — the number of sites increased as well. Many people prefer to automate the process using Excel or some other software, but I take great pleasure in a more organic approach. The key here is to remember that practice improves skill (as I am sure you already know), and will enable you to scan thousands of lines of log entries in very short periods of time.

[Editor’s Note: Michael had also inquired about a sequence of 404 errors that he had sent..]

As for the error log entries you sent, I have the following observations:

First of all, the pattern suggests some sort of an automatic/mass downloader or crawler of some sort (obviously). It doesn’t look like a typical crawler following links, as there are other types of resources that were requested (e.g., JS and RSS files). Whatever it is, it isn’t friendly and should be blocked. Before doing so, I would reverse-lookup the IP/Host information and see if you can gain any insight into what it is or what it’s doing. I see entries like this all the time, and would certainly feel justified blacklisting the related IP address.

I hope this helps, Michael — feel free to ask any additional questions about this — it is one of my favorite activities!


Jeff Starr
About the Author
Jeff Starr = Web Developer. Security Specialist. WordPress Buff.
Blackhole Pro: Trap bad bots in a virtual black hole.

2 responses to “Best Practices for Error Monitoring”

  1. Hi Jeff,

    First of all, i like to thank you for the detail notes o htaccess, its very useful to me as a newbie.

    I have disable the errors in production environment, but it still showing the mysql error. this is the line i have made in htaccess file.

    # disable display of all other errors
    php_flag display_errors off

    thank You

  2. Jeff Starr

    Hi mady, please check this article for more information about this. Also check the “Related Posts” section at the end of the article for some additional help.

Comments are closed for this post. Something to add? Let me know.
Perishable Press is operated by Jeff Starr, a professional web developer and book author with two decades of experience. Here you will find posts about web development, WordPress, security, and more »
Banhammer: Protect your WordPress site against threats.
Air finally clearing here in WA. Feeling grateful to breathe again. #oxygenmatters
Past week here in WA state has been hellish. So much smoke, like living in a chimney.
Now in September, I’m where I wanted to be in March.
Spent some time updating my article on unsafe characters, once again current with latest IETF specification.
Just realized that “Neo” is an anagram for “One”. As in, “he is the One” (The Matrix).
To get VLC app to load all songs (including subfolders), go to Preferences ▸ Show All ▸ Playlist ▸ Subdirectory behavior ▸ Expand.
Switching from PhotoShop to Affinity Photo is one of the most liberating work-related things I've done in 20 years.
Get news, updates, deals & tips via email.
Email kept private. Easy unsubscribe anytime.