Super Plugin Sale! Your Choice: BOGO or 30% Off »
Web Dev + WordPress + Security

Best Practices for Error Monitoring

Given my propensity to discuss matters involving error log data (e.g., monitoring malicious behavior, setting up error logs, and creating extensive blacklists), I am often asked about the best way to go about monitoring 404 and other types of server errors. While I consider myself to be a novice in this arena (there are far brighter people with much greater experience), I do spend a lot of time digging through log entries and analyzing data. So, when asked recently about my error monitoring practices, I decided to share my response here at Perishable Press, and hopefully get some good feedback concerning best practices for error monitoring. Here is my email response to the question:

Hi Michael,

I don’t know if there is a “best” way to monitor error logs. There are many different ways to do it, and you just have to pick the method(s) that works best for you and provides the information you need. For me, I take the process to be a bit of an art form, with the end result being an improved understanding of the traffic patterns relating to my domains.

That said, the methods I use to track errors are three-fold:

  • First, I keep a close eye on PHP errors with a few lines in my HTAccess file (Linux/Apache)
  • I also like to watch my Apache error log, which is available through cPanel on a shared host
  • Most importantly (to me), I keep a tight watch on all 404 errors via a custom 404 script

If you would like more information on any of these methods, let me know and I will do my best to share them with you.

As for monitoring and analyzing, I do everything manually, line by line. I manage a large number of sites and spend around 2 hours per week checking things out, taking notes, and fixing or locking things down as needed. I began this process around four years ago with a single site. As I got better analyzing the 404 error log — which includes URL, referrer, query string, host, IP, and several other metrics — the number of sites increased as well. Many people prefer to automate the process using Excel or some other software, but I take great pleasure in a more organic approach. The key here is to remember that practice improves skill (as I am sure you already know), and will enable you to scan thousands of lines of log entries in very short periods of time.

[Editor’s Note: Michael had also inquired about a sequence of 404 errors that he had sent..]

As for the error log entries you sent, I have the following observations:

First of all, the pattern suggests some sort of an automatic/mass downloader or crawler of some sort (obviously). It doesn’t look like a typical crawler following links, as there are other types of resources that were requested (e.g., JS and RSS files). Whatever it is, it isn’t friendly and should be blocked. Before doing so, I would reverse-lookup the IP/Host information and see if you can gain any insight into what it is or what it’s doing. I see entries like this all the time, and would certainly feel justified blacklisting the related IP address.

I hope this helps, Michael — feel free to ask any additional questions about this — it is one of my favorite activities!


About the Author
Jeff Starr = Fullstack Developer. Book Author. Teacher. Human Being.
WP Themes In Depth: Build and sell awesome WordPress themes.

2 responses to “Best Practices for Error Monitoring”

  1. Hi Jeff,

    First of all, i like to thank you for the detail notes o htaccess, its very useful to me as a newbie.

    I have disable the errors in production environment, but it still showing the mysql error. this is the line i have made in htaccess file.

    # disable display of all other errors
    php_flag display_errors off

    thank You

  2. Hi mady, please check this article for more information about this. Also check the “Related Posts” section at the end of the article for some additional help.

Comments are closed for this post. Something to add? Let me know.
Perishable Press is operated by Jeff Starr, a professional web developer and book author with two decades of experience. Here you will find posts about web development, WordPress, security, and more »
Blackhole Pro: Trap bad bots in a virtual black hole.
All free plugins updated and ready for WordPress 6.6 dropping next week. Pro plugin updates in the works :)
99% of video thumbnail/previews are pure cringe. Goofy faces = Clickbait.
Crazy that we’re almost halfway thru 2024.
I live right next door to the absolute loudest car in town. And the owner loves to drive it.
8G Firewall now out of beta testing, ready for use on production sites.
It's all about that ad revenue baby.
Get news, updates, deals & tips via email.
Email kept private. Easy unsubscribe anytime.