Spring Sale! Save 30% on all books w/ code: PLANET24
Web Dev + WordPress + Security

Ajax-Powered Error Logs

[ Ajax Error Log - File Structure ] As an obsessive website administrator, I like to keep a keen eye on my error logs. Once a week, I download my PHP, 404, and other error logs, and analyze them as closely as time allows. Monitoring site errors and other traffic patterns leads to many of the security-related articles I post here at Perishable Press, including resources such as the 5G Blacklist, Ultimate HTAccess Blacklist, and the Blacklist Candidate Series. Easily, one of the best ways to protect your site is to understand the different types of errors that are happening behind the scenes. This is why monitoring your error logs is so important. In this article, you’ll learn how to make it a little easier and more dynamic with MySQL, PHP, and the magic of Ajax.

Update: Check out the new and improved Ajax Error Log Version 2.0!

Overview

I tend to over-explain things, so here is an overview of this tutorial:

Dynamic Error Monitoring with Ajax

For the past couple of years, I have wanted to develop a way to easily monitor my sites’ errors in “real time.” My idea is to use Ajax and jQuery to display different types of errors dynamically on the web page. If you think of a typical Ajax chat script, where the page is refreshed at short intervals to display new entries, you’ve got a good idea of what I’m doing here. The process goes something like this:

  1. 404 errors are redirected to a custom PHP script
  2. The script then logs the different errors into a database
  3. The contents of the database are displayed on a web page
  4. jQuery/Ajax is used to refresh the page at short intervals

Is this for everyone? Probably not, but if you’re into logging visitor information, recording errors in a database, and keeping an eye on your site in general, you’ll probably benefit from the technique. Keep in mind that the tutorial focuses on setting up and putting things together, so I don’t spend a lot of time getting into the technical aspects of what the actual code is doing. There’s actually a lot of neat stuff happening behind the scenes, such as looking up geographical IP data, encoding image content for the database, and displaying results dynamically with Ajax.

How is this useful?

[ Ajax: Asynchronous Javascript and XML ] By examining static error logs, we’re catching problems after they happen. In my mind, it would be better to have the ability to keep an eye on things in real time and resolve issues as they happen. For example, Perishable Press is frequently scanned for vulnerabilities by teh evil scriptz, but the Ajax Error Log enables me to take immediate action and protect my site. Plus, catching 404 errors helps you spot legitimate errors for things that should exist, enabling a more responsive maintenance strategy.

I honestly don’t think any of this is really necessary, but more like an enhancement over boring static log files.

Sure, you can’t sit there and just watch your error log all day, but you can keep a tab open and keep an eye on things while you’re working online. For many of us, that’s nearly all the time, but even if you’re only watching your errors for a few minutes a day, you’re still going to benefit from observing errors as they happen. Even that much will help you better understand what’s happening (or not happening) on your server, which is the key to good security.

Now that I’ve got this script working for Perishable Press, I like to open a browser window, resize it so it fits in the corner of my screen, and then just sort of keep an eye on things as I work. It’s extremely satisfying stopping the bad guys dead in their tracks. It’s good times ;)

Demonstration

This may all sound rather abstract, so here is a Live Demo of my Ajax Error Log technique. The Demo displays the 404 errors just moments after they occur at my test site. The site gets very little traffic, so if no new errors are appearing, try the following to watch the Dynamic Error Log do its thing:

  1. Open the Demo in new window (not tab), and position it next to the current window (so you can see both at the same time).
  2. Then trigger a 404 error by clicking here *
  3. Watch the 404 error automagically appear on the Demo page!

* Notes:

Technically the “404” error that we’re triggering here is a faux error used for the demo. As explained in the tutorial, the Ajax Error Log is designed for actual 404 errors.

Please only click the error link once or twice. Obviously, I’m heavy into monitoring things, so if you don’t want to get banned, please click only a few times to get the idea. Then if you like the technique and want to continue experimenting, follow the tutorial and set it up at your own site. Thanks in advance for being cool :)

While we’re on the subject, it’s also important to note that you SHOULD NOT display your error log to the general public. This technique is designed for use by a single user from a secure and private web page. Making it public reveals potentially sensitive information and increases the load on your server.

Implementation

Here are the three files you will need to implement your own dynamic error log:

  • 404.php – contains the PHP script that logs the errors
  • AjaxErrorLog.php – connects to the database and displays its contents
  • AjaxErrorLog.html – contains the HTML & jQuery used to display the results

We’ll create and configure each of these files in this tutorial, but you can also just download the files and follow along that way.

In addition to these 3 files, you’ll need to create a table in your MySQL database. This can either be an existing database or a new one.

Step 1: Create the database table

Once you’ve got the database in place, execute the following SQL query (via phpMyAdmin or similar):

CREATE TABLE `error_log` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `logtime` varchar(200) NOT NULL default '',
  `request` varchar(200) NOT NULL default '',
  `referer` varchar(200) NOT NULL default '',
  `qstring` varchar(200) NOT NULL default '',
  `address` varchar(200) NOT NULL default '',
  `agent` varchar(200) NOT NULL default '',
  `error` varchar(200) NOT NULL default '',
  PRIMARY KEY  (`id`),
  cur_timestamp TIMESTAMP(8)
) AUTO_INCREMENT = 1

That should create a table named “error_log” in your database. This is where we will be storing the data collected from our 404.php file, which serves as our site’s actual “404-Error” page that also logs each 404 error into the database.

Step 2: Create the custom error page

Create a blank file named 404.php, and fill it with the following PHP code:

<?php
/*
Dynamic Error Logging by Jeff Starr @ Perishable Press
Project URL: https://perishablepress.com/ajax-error-log/
License: GNU General Public License
Version: 1.1
*/

// database credentials
$hostname = "hostname";
$database = "database";
$username = "username";
$password = "password";

// site URL (no trailing slash!)
$website = "http://example.com";

// send proper headers for 404
header("HTTP/1.1 404 Not Found");
header("Status: 404 Not Found");

// set some server variables
$logtime = date("F jS Y, h:ia", time() - 10800);
$request = $site . $_SERVER['REQUEST_URI'];
$referer = $_SERVER['HTTP_REFERER'];
$qstring = $_SERVER['QUERY_STRING'];
$address = $_SERVER['REMOTE_ADDR'];
$agent   = $_SERVER['HTTP_USER_AGENT'];
$error   = "404"; // for other errors

// connect to database
$dbcon = mysql_connect($hostname, $username, $password)
 or die("Couldn't connect to SQL Server on $dbhost");
$selected = mysql_select_db($database, $dbcon)
 or die("Couldn't open database $database");

// sanitize server variables
$logtime_esc = mysql_real_escape_string($logtime);
$request_esc = mysql_real_escape_string($request);
$referer_esc = mysql_real_escape_string($referer);
$qstring_esc = mysql_real_escape_string($qstring);
$address_esc = mysql_real_escape_string($address);
$agent_esc   = mysql_real_escape_string($agent);

// insert record and close database
$query  = "INSERT INTO `error_log` (logtime,request,referer,qstring,address,agent,error) ";
$query .= "VALUES ('$logtime_esc','$request_esc','$referer_esc','$qstring_esc','$address_esc','$agent_esc','$error')";
mysql_query($query);
mysql_close($dbcon);

// and finally output the html
?><html>
	<head><title>404 - Not Found</title></head>
	<body id="error-404">
		<div class="left">
			<div class="post">
				<h1>Error 404 — Not Found</h1>
				<p>The requested resource was not found on this server.</p>
				<div class="content">
					<p><?php echo "Requested URL: " . $website . $request_esc; ?></p>
				</div>
			</div>
		</div>
	</body>
</html>

This script basically declares some server variables and then sanitizes them before recording them in the database. It also send the correct headers and performs a Geographical IP Lookup to capture key attributes of the request. Thanks to geoPlugin.com for their awesome lookup service, and also to this forum thread for the help with forwarded URLs.

For the 404.php file to work, you’ll need to edit the first four variables – $hostname, $database, $username, and $password – to enable the script to connect to your database. Once the file is configured and in place, the server needs to know to use it when responding to 404 errors. We set this up in the next step..

Step 3: Setup the 404 redirect

The next step is to instruct the server to use the custom 404.php file instead of the default error page. With Apache, this is possible with a single line added to your site’s root .htaccess file:

ErrorDocument 404 /ajax-error-log/404.php

..and then edit the path to match the file’s actual location on your server. Once that’s in place on your server, it’s time to check that things are working.

Step 4: Verify functionality

Once you get the 404.php file uploaded and the redirect set up, trigger a few 404 errors on your site and check the results. For each error, you should see the custom 404 page displayed in your browser and a recorded entry in your error_log table. Give it a few spins and ensure that everything is working. You will probably want to customize the appearance of your new error page to fit your site’s design (or not). It’s pretty basic by default, but there’s a ton of cool things you can do. To help with customizing, check out my post on pimping out your 404 error page.

Step 5: Create the dynamic log script

At this point, you have something very useful: a custom 404 page that records all 404 errors in the database. Now let’s get dynamic with it and display the information in “real-time” on a web page. Create a file called AjaxErrorLog.php and insert the following PHP code:

<?php
/*
Dynamic Error Logging by Jeff Starr @ Perishable Press
Project URL: https://perishablepress.com/ajax-error-log/
License: GNU General Public License
Version: 1.1
*/

// database credentials
$hostname = "hostname";
$database = "database";
$username = "username";
$password = "password";

// site URL (no trailing slash!)
$website = "http://example.com";

// connect to server
$dbhandle = mysql_connect($hostname, $username, $password)
 or die('Could not connect to SQL Server on $hostname');

// select the database
$selected = mysql_select_db($database, $dbhandle)
 or die('Could not open database $database');

// create the sql query
$query = 'SELECT id,logtime,request,referer,qstring,address,agent,error
 FROM error_log ORDER BY id DESC LIMIT 50'; // defaults to 50 entries

// execute query and return records
$result = mysql_query($query);
$number = mysql_num_rows($result);

// display the results
while($row = mysql_fetch_array($result)) {
	echo '<div class="log-entry"><pre>';
	echo 'Error: '.htmlspecialchars($row['error']);
	echo ' - '.htmlspecialchars($row['logtime'])."\n";
	echo 'Request: '.$website.htmlspecialchars($row['request'])."\n";
	echo 'User Agent: '.htmlspecialchars($row['agent'])."\n";
	echo 'IP Address: '.htmlspecialchars($row['address']);
	echo '</pre></div>';
}

//close db connection
mysql_close($dbhandle);
?>

Let’s see what’s happening with this script. First we specify our database credentials, which should be the same as those specified in the 404.php file. Using this information, the script then connects to the server (or dies trying) and selects the specified database. From there, it creates and executes an SQL query and returns the newest ten results (configurable). It then uses that information to output chunks of HTML markup to the web page. Finally, the database connection is closed.

After configuring the AjaxErrorLog.php file, it’s time to bring it all together by including the output in an actual web page and displaying the log results dynamically via jQuery-flavored Ajax.

Step 6: Create the dynamic log page

Finally, create a file named “AjaxErrorLog.html” and insert the following code:

<!DOCTYPE HTML>
<html>
	<head>
		<title>Ajax Error Log</title>
		<!-- Ajax Error Log @ https://perishablepress.com/ajax-error-log/ -->
		<meta http-equiv="content-type" content="text/html; charset=UTF-8">
		<style>
			pre {
				font: 10px/1.5 Courier, "Courier New", mono;
				background-color: #efefef; border: 1px solid #ccc;
				width: 700px; margin: 7px; padding: 10px;
				white-space: pre-wrap;
				}
		</style>
		<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.0/jquery.min.js"></script>
		<script>
			$(document).ready(function() {
				$("#results").load("AjaxErrorLog.php");
				var refreshId = setInterval(function() {
					$("#results").load("AjaxErrorLog.php").fadeIn("slow");
				}, 2000); // refresh time (default = 2000 ms = 2 seconds)
			});
		</script>
	</head>
	<body>
		<noscript><div id="response"><h1>JavaScript is required for this demo.</h1></div></noscript>
		<div id="results"></div>
	</body>
</html>

This is the page that will display the results dynamically in a browser. It’s a basic web page that uses a little jQuery/Ajax magic to load the output of the AjaxErrorLog.php file and refresh the results every two seconds (as specified in the code as “2000”). Feel free to use any refresh interval you wish, just keep in mind that these are server requests, so don’t go too crazy. ;)

We’re also using some CSS to add a little style to the markup. Currently, the styling is pretty minimal, so you’ll probably want to customize things to suit your own design. Also note that, for the sake of simplicity, we are using Google’s ajax.googleapis.com for the jQuery script.

Once you get all the files configured and in place, open the AjaxErrorLog.html file in your browser and watch the results scroll across your screen. If the database doesn’t contain any errors yet, go ahead and make up some crazy URLs and request them in another tab. Things should be working quite beautifully.

Download the files

Here you may download the source files for Ajax Error Log.

Download Ajax Error LogVersion 2.0 ( 2.80 KB ZIP )

Changelog

  • v1.1: Replaced mysql_escape_string with mysql_real_escape_string
  • v1.1: Added htmlspecialchars to escape output SQL
  • v1.1: Added $website variable for better accuracy
  • v1.1: Changed the default number of displayed results to 50

With a glass eye on you..

So that’s how to build a Dynamic Error Log using PHP, HTML, CSS, and jQuery. The Ajax Error Log that we build in this tutorial is pretty basic, but it’s also fairly easy to customize once you understand how everything works. To see an example of the possibilities, check out the Ajax Error Log Ultra, which is a highly customized version of the Ajax-error-log script provided in this tutorial.

I would eventually like to expand the functionality of this Dynamic Error Log and translate it into a WordPress plugin. Until then, I want to share the technique and get as much feedback as possible. If you have any ideas for improving the security, functionality, or usefulness of the technique, please let me know, either in the comments or via email. As always, Thank you for reading Perishable Press =)

About the Author
Jeff Starr = Creative thinker. Passionate about free and open Web.
USP Pro: Unlimited front-end forms for user-submitted posts and more.

16 responses to “Ajax-Powered Error Logs”

  1. Russell Heimlich 2011/07/08 11:06 am

    Or you could use a desktop tool like SurvLog http://grass.co.ve/

  2. Kroc Camen 2011/07/08 12:05 pm

    Quite an idea!

    But, MySQL, really? Really, really? Isn’t that a bit over kill. Wouldn’t SQLite do just fine, and no having to setup and configure an external database, and the db is just a file you can download and examine locally too?

    • Ha! You make it sound hard.

      I like to use MySQL, but dude you can use whatever you want, it’s totally up to you.

      Setting up a MySQL database isn’t that difficult and you can even examine the database files locally, with a text editor or whatever floats your boat.

  3. Norik Davtian 2011/07/08 3:24 pm

    I love the opening statement “As an obsessive website administrator” lol

    Thanks for sharing the code :)

  4. Nice Jeff. I set up something similar a couple months ago. It just logs to the db then I just monitor the db table remotely. Definitely great for finding broken links/images and baddies.

  5. Anonymous 2011/07/09 1:38 am

    Great stuff, Jeff. I took your idea, and extended it out to an all out “business log”. With a bit of customizing, I’ll use this to not only track errors, but sales, customers, support tickets, affiliate commissions, etc., across several websites and services, all from one interface.

  6. WOW Jeff that’s pretty amazing!

    hmmm here is an idea for you to make a little $$ money with, well that is if it isn’t already available….

    Turn this in to an add-on/plug-in for the Magento eCommerce platform. The company I work for just rolled out a new site on Magento and this would be very very useful.

    Thanks again for sharing your work!

    I know it takes a hell of a lot of time to build the system to start with but when you add an awesome detailed tutorial on top of it, dam do you sleep much?

    mad props my man.

  7. I tested the AJAX Error Log on two sites. In addition to your directions, I also password-protected the ajax-error-log directory using .htaccess/.htpasswd (Yes, I’m obsessive about security too)

    The first site was my empty test site and everything worked perfectly.

    The second site was a WordPress installation, with WordPress installed in a subdirectory but in control of the root as detailed in Chapter 2.1.2 of “Digging into WordPress”. (Yes, I bought the ebook)

    This is where I encountered a problem. When I try to access a file I know is missing, WordPress catches the 404 error and notifies the visitor. The same thing happens when I try to visit http://mydomain.com/ajax-error-log/AjaxErrorLog.html which I know exists. It still throws a 404 error caught by WordPress. Just for testing purposes, I put a favicon.ico file in the password-protected ajax-error-log directory and tried to view it in the browser. I was able to view it and it never asked for a username/password. I thought it was odd that the password protection didn’t seem to work.

    After some research, it seems the problem is related to a conflict between password-protected directories and the WordPress-generated .htaccess code when permalinks is turned on.

    In the root directory’s .htaccess, I did add the 404 redirect line as given in Step 3. This was added above the WordPress-generated directives. From what I’ve found online, a solution may involve a 401 redirect and maybe a 403 redirect. Sorry, but I’m not knowledgeable enough with .htaccess trickery to solve this alone.

    Can you help with a possible solution?

  8. Same problem here. I can’t get it working on WordPress in a sub-directory.
    A direct call for the 404.php does the job; it shows what it needs to show, but just hxxp://mydomain/sffs is caught by WordPress.

    I tried adding the ajax-error-log files in the sub-directory but no luck.

    For the redirect in the htaccess, I used ErrorDocument 404 /ajax-error-log/404.php and in the sub-directory I used ErrorDocument 404 /blog/ajax-error-log/404.php

    What could cause it?

  9. Jeff Starr 2011/07/09 8:52 pm

    Re: those trying to integrate the Ajax Log with WordPress, I’ll try to get a quick tutorial posted soon on how to set that up.

  10. This is a great little tool that I hope to utilise soon. Looks fun and a good way of checking persistent 404 creators.

    Cheers
    I

Comments are closed for this post. Something to add? Let me know.
Welcome
Perishable Press is operated by Jeff Starr, a professional web developer and book author with two decades of experience. Here you will find posts about web development, WordPress, security, and more »
.htaccess made easy: Improve site performance and security.
Thoughts
I live right next door to the absolute loudest car in town. And the owner loves to drive it.
8G Firewall now out of beta testing, ready for use on production sites.
It's all about that ad revenue baby.
Note to self: encrypting 500 GB of data on my iMac takes around 8 hours.
Getting back into things after a bit of a break. Currently 7° F outside. Chillz.
2024 is going to make 2020 look like a vacation. Prepare accordingly.
First snow of the year :)
Newsletter
Get news, updates, deals & tips via email.
Email kept private. Easy unsubscribe anytime.