Latest TweetsDisplay Your WordPress Site Statistics: Complete Guide digwp.com/2019/02/display-word…
Perishable Press

7G Firewall : Log Blocked Requests

[ 7G Firewall (Beta) ] This tutorial explains how to log requests that are blocked by the 7G Firewall. This is useful for testing, debugging, and just keeping an eye on things. Learn how to log requests from Apache mod_rewrite and download my custom 7G logging script. It’s a complete example that shows how to log rewrite requests via PHP. All open source and free :)

If you’re using 7G, you can find any errors using the free logging script.

Contents

How it works

The 7G Firewall includes built-in logging directives. Once logging is enabled (as explained below), mod_rewrite will send request data to our handy little PHP script. The logging script then parses the data and writes it to the log file. The result is a log file that looks very similar to Apache defaults.

Under the hood, here is what happens for any requests blocked by 7G:

  1. 7G checks the request
  2. If not blocked, the request continues normally
  3. If blocked, the request data is sent to the logging script
  4. The logging script then writes the data to the log file
  5. The script exits with a simple message (configurable)

So regular visitors and legit traffic will continue normally, while any blocked requests are redirected to the logging script. Conceptually simple.

Log Example

Here is an example showing what the log entries look like:

012.012.012.01 - 2019/01/24 01:47:12 - GET - HTTP/1.0 - /www.php - boot.ini [boot.ini] - - - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6)
187.255.215.91 - 2019/01/24 21:19:01 - GET - HTTP/1.0 - / - - - - Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) [Exabot] 
222.333.444.00 - 2019/01/24 00:15:33 - POST - HTTP/1.0 - /xertive.php [xertive] - - - - curl/7.54.0

In log entries, matching firewall patterns are indicated via brackets like [this]. In the above example, notice the following blocked patterns:

  • boot.ini — from 7G QUERY STRING rules
  • Exabot — from 7G USER AGENT rules
  • xertive — from 7G REQUEST URI rules

So for each request, the location of the bracketed/match string tells us the related section of 7G. More details on this later in the post.

Usage

To log any/all requests blocked by 7G, follow the steps:

  1. Add 7G Firewall to root .htaccess (or Apache config)
  2. Configure 7G Firewall for logging (see next section)
  3. Unzip 7G Logging script and upload files to root web directory
  4. Make log.txt writable (and protect via .htaccess)

Test well and leave feedback below in the comments or direct via my contact form. Please do not report bugs in the comment section, best to communicate via email/contact form, thank you for understanding.

Configure 7G for logging

The 7G Firewall comprises the following sections:

# 7G:[CORE]
# 7G:[QUERY STRING]
# 7G:[REQUEST URI]
# 7G:[USER AGENT]
# 7G:[REMOTE HOST]
# 7G:[HTTP REFERRER]
# 7G:[REQUEST METHOD]

Each of these sections contains a set of mod_rewrite rules. For each section (except CORE rules), you want to disable (comment out) the default RewriteRule, and enable (un-comment) the logging RewriteRule. For example, in the QUERY STRING section:

# 7G:[QUERY STRING]
<IfModule mod_rewrite.c>
	
	.
	.
	.
	
	RewriteRule . - [F,L]
	
	# RewriteRule . /7g_log.php?log [L,NE,E=7G_QUERY_STRING:%1___%2___%3]
	
</IfModule>

We want to change that to this:

# 7G:[QUERY STRING]
<IfModule mod_rewrite.c>
	
	.
	.
	.
	
	# RewriteRule . - [F,L]
	
	RewriteRule . /7g_log.php?log [L,NE,E=7G_QUERY_STRING:%1___%2___%3]
	
</IfModule>

So now the QUERY STRING rules are sending request data to our 7g_log.php log file. Repeat this same “rule swap” for each section (except CORE rules) in 7G. Once this is done, 7G is configured for logging and will send all blocked requests and data to the 7G Logging script.

Preparing the Log file

To prepare the log file, you want to make sure it is writable by the server (e.g., change CHMOD permissions on the file). Then you also want to make sure the file is protected from outside access. This is prevent sensitive information from falling into public domain. As we’re already working with .htaccess, here is a simple code snippet that protects our log file from all outside access:

# Apache < 2.4
<IfModule !mod_authz_core.c>
	<Files ~ "7g_log\.txt">
		Deny from all
	</Files>
</IfModule>

# Apache 2.4+
<IfModule mod_authz_core.c>
	<Files ~ "7g_log\.txt">
		Require all denied
	</Files>
</IfModule>

Add that to your root .htaccess file and done. No changes need to be made. Note however that both sections are not necessary; if you know your version of Apache, it is safe to use only the relevant code.

Reading the log file

For each log entry, the following request data is recorded (if available):

  • IP Address
  • Date/Time
  • Request Method
  • Request Protocol
  • Request URI
  • Query String
  • Remote Host
  • Referrer
  • User Agent

So each log entry/line records these fields in the following order:

IP Address - Date/Time - Request Method - Request Protocol - Request URI - Query String - Remote Host - Referrer - User Agent

Note that empty/blank values are simply left as-is. So when you see stuff like this in your log file it’s totally normal:

- / - - - - User agent 1234..

It just means that no data was available for each of the blank items. It is common especially for Remote Host and Referrer fields to be blank.

Preflight Check

Here is a checklist of requirements for logging to work:

  • 7G Firewall installed and configured for logging
  • 7G_log.php and 7G_log.txt both added to root web directory
  • 7G_log.txt is writable by the server
  • 7G_log.txt is protected against outside access

Once everything is in place and ready, 7G logging should be enabled on your site. Note that the 7G_log.php script includes several variables/options that you can modify if desired:

  • SEVENGLOGPATH = Path to the log file
  • SEVENGLOGFILE = Name of the log file
  • SEVENGUALENGTH = Length of User Agent log entry
  • SEVENGEXIT = Message displayed to blocked visitors
  • date_default_timezone_set('UTC') = Default Timezone

These variables are predefined to work according to this tutorial, out of the box. So no changes need to be made to the logging script, but there are few things that you can tweak as needed.

Download 7G Logging script

Here is the latest version of the 7G Firewall Logging script. License: GPL v3.

7G Firewall: Log Blocked Requests – Version 1.0 (2KB zip)
Important: 7G Logging is recommended for testing/development purposes only. Not recommended for live/production sites.
Disclaimer: Capturing HTTP requests via mod_rewrite can be challenging, because each host/server/setup/config is unique. The 7G Logging script is developed in several different Apache/server environments, but it is not guaranteed to work out of the box for everyone.

Troubleshooting 7G Firewall

Once logging is enabled on your site, troubleshooting and testing the 7G Firewall becomes quick and easy. This section gives some ideas and tips that should be useful for anyone wanting to debug, test, and so forth.

While using 7G, if any legitimate page or resource fails to load, or if some feature stops working, simply take a look at the 7G log file. Locate the most recent (and/or relevant) entries. For each related entry, the offending firewall pattern will be indicated with [brackets]. So you will know immediately which 7G rules/pattern(s) are responsible for any issues.

If you discover any offending patterns, simply disable (comment out) or remove. Also please report any bugs or patterns that are causing problems. That way I can update the firewall and keep things as error-free as possible.

Troubleshooting Tips

Here is a working list of things to check when troubleshooting 7G.

Locating patterns
Some sections of 7G contain a lot of “tightly packed” rules. This can make locating offending patterns difficult, even when you know the matching pattern. In such cases, the quickest way to identify the pattern is to use the the halving method.
Code placement
In your .htaccess file, Make sure that 7G code before any existing mod_rewrite rules (e.g., WordPress Permalinks).
Server error
If you get a server error after installing 7G, double-check that your site meets the requirements.
Encoded Characters
By default, special characters are passed to the log file unencoded. To instead pass the encoded equivalents, remove the “No Escape” flag NE from each of the 7G rewrite rules.
Live Demo
You can check out a live demo of 7G Firewall here at Perishable Press.

More tips will be added as they arrive.

Bonus: IP Logging with 7G

Users familiar with nG Firewall/Blacklist may have noticed that IP blocking is removed from 7G. As covered in previous posts, there are better, more effective ways to prevent site access based on IP address. Nonetheless, blocking by IP remains useful in a variety of scenarios. So with that in mind, here is a bonus IP ADDRESS section that may be added to 7G (or any .htaccess file):

# 7G:[IP ADDRESS]
<IfModule mod_rewrite.c>
	
	# RewriteCond %{REQUEST_URI} !(7g_log.php) [NC]
	
	# RewriteCond %{REMOTE_ADDR} ^(000\.000\.000\.000)$ [OR]
	# RewriteCond %{REMOTE_ADDR} ^(100\.100\.100\.100)$ [OR]
	# RewriteCond %{REMOTE_ADDR} ^(200\.200\.200\.200)$ 
	
	# RewriteRule . /7g_log.php?log [L,NE,E=7G_IP_ADDRESS:%1]
	
</IfModule>

As written, this code does nothing, because each directive is disabled (commented out with a hash/pound sign #). To enable these rules, remove the hash from each line. Then replace the “dummy” placeholder IP values with any that you would like to block and log. Once implemented, these rules will result in any blocked IP addresses indicated in the log file with [brackets].

Tip: For the above IP ADDRESS section, each RewriteCond should end with an [OR] flag, except the last RewriteCond. If you only want to block one IP address, use only one RewriteCond and omit the [OR] flag.

Faster Way to Block IP Addresses

One last thing came to mind: for anyone who may be new to all of this, a much better way of blocking some IP address, is to just use core Apache functionality:

# Apache < 2.4
<IfModule !mod_authz_core.c>
	Deny from 123.123.123.123
</IfModule>

# Apache 2.4+
<IfModule mod_authz_core.c>
	Require not ip 123.123.123.123
</IfModule>

You can drop that puppy in wholesale, or just use whichever snippet applies to your Apache version. Going this route is much faster than invoking the majesty of mod_rewrite. And if you need to add more IPs to block just add more lines:

# Apache < 2.4
<IfModule !mod_authz_core.c>
	Deny from 123.123.123.123
	Deny from 111.111.111.111
	Deny from 222.222.222.222
</IfModule>

# Apache 2.4+
<IfModule mod_authz_core.c>
	Require not ip 123.123.123.123
	Require not ip 111.111.111.111
	Require not ip 222.222.222.222
</IfModule>

Etc.! :)

May the log files be with you!

Jeff Starr
About the Author Jeff Starr = Fullstack Developer. Book Author. Teacher. Human Being.
Archives
2 responses
  1. I checked out the PHP coding, looks pretty good and simple. I didn’t know about “parse_url()”, at first, because my PHP documentation apparently did not have it! – Oops! Pretty darn useful function.

    Anyway,

    I changed the log-file name to something like: 7g_log.log, because I already disallow access to all “log” files.

    Some of these filtering rules (namely those for User-Agent, Host, Referer, and even Request_URI [YES – uses underline separator instead!] ) also work very well with SetEnvIfNoCase directives. I set an environment variable for any positives (to something like “bad_bot”) – which results in setting $_SERVER[‘bad_bot’] – OR – $_SERVER[‘REDIRECT_bad_bot’] (as in the case of trapping a bad “Request_URI”).

    In using this method,

    I can use a module, from within an “errordoc” program, to trap out mischief, by reading the custom environment variable that was set, and doing whatever with that visit, IP, or whichever.

    BTW: Pretty awesome REGEX’s in the lot! I have a few others I have been using to trap bad “THE_REQUEST” header field values.

    Thank you for the quality info you publish. I have learned to be a bit more creative with my solutions, on account of some of the fine solutions you have presented.

    – Jim S.

    • Jeff Starr

      Thanks Jim, always appreciate your feedback and thoughts. I did initially write 7G using SetEnvIfNoCase directives, but there were certain sections (I forget which ones) that were not picking up. So I went with mod_rewrite as it is more universal in effectiveness. Hopefully you find some benefit with all of it, and thanks again for all of your inputs and ideas.

Drop a Comment  ]
RSS