7G Firewall : Log Blocked Requests

♦ Posted by Jeff Starr in .htaccess, PHP, Security

Updated June 26, 2024 • 20 comments

[ 7G Firewall (Beta) ] This tutorial explains how to log requests that are blocked by the 7G Firewall. This is useful for testing, debugging, and just keeping an eye on things. Learn how to log requests from Apache mod_rewrite and download my custom 7G logging script. It’s a complete example that shows how to log rewrite requests via PHP. All open source and free :)

Note: The following article applies to 7G Firewall. To enable logging for 8G (or better), check out Enable Logging for nG Firewall.

How it works
Log Example
Usage
Configure 7G for logging
Preparing the Log file
Reading the log file
Preflight Check
Download 7G Logging script
Troubleshooting 7G Firewall
Bonus: IP Logging with 7G

How it works

The 7G Firewall includes built-in logging directives. Once logging is enabled (as explained below), mod_rewrite will send request data to our handy little PHP script. The logging script then parses the data and writes it to the log file. The result is a log file that looks very similar to Apache defaults.

Under the hood, here is what happens for any requests blocked by 7G:

7G checks the request
If not blocked, the request continues normally
If blocked, the request data is sent to the logging script
The logging script then writes the data to the log file
The script exits with a simple message (configurable)

So regular visitors and legit traffic will continue normally, while any blocked requests are redirected to the logging script. Conceptually simple.

Log Example

Here is an example showing what the log entries look like:

012.012.012.01 - 2019/01/24 01:47:12 - GET - HTTP/1.0 - /www.php - boot.ini [boot.ini] - - - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6)
187.255.215.91 - 2019/01/24 21:19:01 - GET - HTTP/1.0 - / - - - - Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) [Exabot] 
222.333.444.00 - 2019/01/24 00:15:33 - POST - HTTP/1.0 - /xertive.php [xertive] - - - - curl/7.54.0

In log entries, matching firewall patterns are indicated via brackets like [this]. In the above example, notice the following blocked patterns:

boot.ini — from 7G QUERY STRING rules
Exabot — from 7G USER AGENT rules
xertive — from 7G REQUEST URI rules

So for each request, the location of the bracketed/match string tells us the related section of 7G. More details on this later in the post.

Usage

To log any/all requests blocked by 7G, follow the steps:

Add 7G Firewall to root .htaccess (or Apache config)
Configure 7G Firewall for logging (see next section)
Unzip 7G Logging script and upload files to root web directory
Make 7G_log.txt writable (and protect via .htaccess)

Test well and leave feedback below in the comments or direct via my contact form. Please do not report bugs in the comment section, best to communicate via email/contact form, thank you for understanding.

Configure 7G for logging

The 7G Firewall comprises the following sections:

# 7G:[CORE]
# 7G:[QUERY STRING]
# 7G:[REQUEST URI]
# 7G:[USER AGENT]
# 7G:[REMOTE HOST]
# 7G:[HTTP REFERRER]
# 7G:[REQUEST METHOD]

Each of these sections contains a set of mod_rewrite rules. For each section (except CORE rules), you want to disable (comment out) the default RewriteRule, and enable (un-comment) the logging RewriteRule. For example, in the QUERY STRING section:

# 7G:[QUERY STRING]
<IfModule mod_rewrite.c>
	
	.
	.
	.
	
	RewriteRule . - [F,L]
	
	# RewriteRule .* /7G_log.php?log [END,NE,E=7G_REQUEST_URI:%1___%2___%3]
	
</IfModule>

We want to change that to this:

# 7G:[QUERY STRING]
<IfModule mod_rewrite.c>
	
	.
	.
	.
	
	# RewriteRule . - [F,L]
	
	RewriteRule .* /7G_log.php?log [END,NE,E=7G_REQUEST_URI:%1___%2___%3]
	
</IfModule>

So now the QUERY STRING rules are sending request data to our 7G_log.php log file. Repeat this same “rule swap” for each section (except CORE rules) in 7G. Once this is done, 7G is configured for logging and will send all blocked requests and data to the 7G Logging script.

Preparing the Log file

To prepare the log file, you want to make sure it is writable by the server (e.g., change CHMOD permissions on the file). Then you also want to make sure the file is protected from outside access. This is prevent sensitive information from falling into public domain. As we’re already working with .htaccess, here is a simple code snippet that protects our log file from all outside access:

# Apache < 2.4
<IfModule !mod_authz_core.c>
	<Files ~ "7G_log\.txt">
		Deny from all
	</Files>
</IfModule>

# Apache 2.4+
<IfModule mod_authz_core.c>
	<Files ~ "7G_log\.txt">
		Require all denied
	</Files>
</IfModule>

Add that to your root .htaccess file and done. No changes need to be made. Note however that both sections are not necessary; if you know your version of Apache, it is safe to use only the relevant code.

Reading the log file

For each log entry, the following request data is recorded (if available):

IP Address
Date/Time
Request Method
Request Protocol
Request URI
Query String
Remote Host
Referrer
User Agent

So each log entry/line records these fields in the following order:

IP Address - Date/Time - Request Method - Request Protocol - Request URI - Query String - Remote Host - Referrer - User Agent

Note that empty/blank values are simply left as-is. So when you see stuff like this in your log file it’s totally normal:

- / - - - - User agent 1234..

It just means that no data was available for each of the blank items. It is common especially for Remote Host and Referrer fields to be blank.

Preflight Check

Here is a checklist of requirements for logging to work:

Running PHP version >= 5.4.0
7G Firewall installed and configured for logging
7G_log.php and 7G_log.txt both added to root web directory
7G_log.txt is writable by the server
7G_log.txt is protected against outside access

Once everything is in place and ready, 7G logging should be enabled on your site. Note that the 7G_log.php script includes several variables/options that you can modify if desired:

SEVENGLOGPATH = Path to the log file
SEVENGSTATUS = Status Code for blocked requests
SEVENGLOGFILE = Name of the log file
SEVENGUALENGTH = Length of User Agent log entry
SEVENGEXIT = Message displayed to blocked visitors
date_default_timezone_set('UTC') = Default Timezone

These variables are predefined to work according to this tutorial, out of the box. So no changes need to be made to the logging script, but there are few things that you can tweak as needed.

Download 7G Logging script

Here is the latest version of the 7G Firewall Logging script. License: GPL v3.

Download 7G Firewall: Log Blocked RequestsVersion 1.2 ( 2.27 KB ZIP )

Important: 7G Logging is recommended for testing/development purposes only. Not recommended for live/production sites.

Disclaimer: Capturing HTTP requests via mod_rewrite can be challenging, because each host/server/setup/config is unique. The 7G Logging script is developed in several different Apache/server environments, but it is not guaranteed to work out of the box for everyone.

Download Notes

For some unknown reason, downloads on Windows 10 are not working very well. I’ve been troubleshooting, but so far have been unable to resolve the issue. If you have any infos on this, please let me know via the comments below or send a quick email via my contact form. Until then, Windows 10 users best advice is to clear browser cache and try again, or try downloading with a different browser.

Troubleshooting 7G Firewall

Once logging is enabled on your site, troubleshooting and testing the 7G Firewall becomes quick and easy. This section gives some ideas and tips that should be useful for anyone wanting to debug, test, and so forth.

While using 7G, if any legitimate page or resource fails to load, or if some feature stops working, simply take a look at the 7G log file. Locate the most recent (and/or relevant) entries. For each related entry, the offending firewall pattern will be indicated with [brackets]. So you will know immediately which 7G rules/pattern(s) are responsible for any issues.

If you discover any offending patterns, simply disable (comment out) or remove. Also please report any bugs or patterns that are causing problems. That way I can update the firewall and keep things as error-free as possible.

Troubleshooting Tips

Here is a working list of things to check when troubleshooting 7G.

Locating patterns: Some sections of 7G contain a lot of “tightly packed” rules. This can make locating offending patterns difficult, even when you know the matching pattern. In such cases, the quickest way to identify the pattern is to use the the halving method.
Code placement: In your .htaccess file, Make sure that 7G code before any existing mod_rewrite rules (e.g., WordPress Permalinks).
Server error: If you get a server error after installing 7G, double-check that your site meets the requirements.
Encoded Characters: By default, special characters are passed to the log file unencoded. To instead pass the encoded equivalents, remove the “No Escape” flag NE from each of the 7G rewrite rules.
Live Demo: You can check out a live demo of 7G Firewall here at Perishable Press.

More tips will be added as they arrive.

Bonus: IP Logging with 7G

Users familiar with nG Firewall/Blacklist may have noticed that IP blocking is removed from 7G. As covered in previous posts, there are better, more effective ways to prevent site access based on IP address. Nonetheless, blocking by IP remains useful in a variety of scenarios. So with that in mind, here is a bonus IP ADDRESS section that may be added to 7G (or any .htaccess file):

# 7G:[IP ADDRESS]
<IfModule mod_rewrite.c>

	# RewriteCond %{REMOTE_ADDR} ^(000\.000\.000\.000)$ [OR]
	# RewriteCond %{REMOTE_ADDR} ^(100\.100\.100\.100)$ [OR]
	# RewriteCond %{REMOTE_ADDR} ^(200\.200\.200\.200)$ 

	# RewriteRule .* /7G_log.php?log [END,NE,E=7G_IP_ADDRESS:%1]

</IfModule>

As written, this code does nothing, because each directive is disabled (commented out with a hash/pound sign #). To enable these rules, remove the hash from each line. Then replace the “dummy” placeholder IP values with any that you would like to block and log. Once implemented, these rules will result in any blocked IP addresses indicated in the log file with [brackets].

Tip: For the above IP ADDRESS section, each RewriteCond should end with an [OR] flag, except the last RewriteCond. If you only want to block one IP address, use only one RewriteCond and omit the [OR] flag.

Faster Way to Block IP Addresses

One last thing came to mind: for anyone who may be new to all of this, a much better way of blocking some IP address, is to just use core Apache functionality:

# Apache < 2.4
<IfModule !mod_authz_core.c>
	Deny from 123.123.123.123
</IfModule>

# Apache 2.4+
<IfModule mod_authz_core.c>
	Require not ip 123.123.123.123
</IfModule>

You can drop that puppy in wholesale, or just use whichever snippet applies to your Apache version. Going this route is much faster than invoking the majesty of mod_rewrite. And if you need to add more IPs to block just add more lines:

# Apache < 2.4
<IfModule !mod_authz_core.c>
	Deny from 123.123.123.123
	Deny from 111.111.111.111
	Deny from 222.222.222.222
</IfModule>

# Apache 2.4+
<IfModule mod_authz_core.c>
	Require not ip 123.123.123.123
	Require not ip 111.111.111.111
	Require not ip 222.222.222.222
</IfModule>

For more information, check out How to Block IPs with 6G Firewall.

May the log files be with you!

About the Author

Jeff Starr = Designer. Developer. Producer. Writer. Editor. Etc.

20 responses to “7G Firewall : Log Blocked Requests”

Jim S Smith 2019/02/16 9:14 pm • Reply

I checked out the PHP coding, looks pretty good and simple. I didn’t know about “parse_url()”, at first, because my PHP documentation apparently did not have it! – Oops! Pretty darn useful function.

Anyway,

I changed the log-file name to something like: 7g_log.log, because I already disallow access to all “log” files.

Some of these filtering rules (namely those for User-Agent, Host, Referer, and even Request_URI [YES – uses underline separator instead!] ) also work very well with SetEnvIfNoCase directives. I set an environment variable for any positives (to something like “bad_bot”) – which results in setting $_SERVER[‘bad_bot’] – OR – $_SERVER[‘REDIRECT_bad_bot’] (as in the case of trapping a bad “Request_URI”).

In using this method,

I can use a module, from within an “errordoc” program, to trap out mischief, by reading the custom environment variable that was set, and doing whatever with that visit, IP, or whichever.

BTW: Pretty awesome REGEX’s in the lot! I have a few others I have been using to trap bad “THE_REQUEST” header field values.

Thank you for the quality info you publish. I have learned to be a bit more creative with my solutions, on account of some of the fine solutions you have presented.

– Jim S.

Jeff Starr 2019/02/17 10:07 am • Post Author • Reply

Thanks Jim, always appreciate your feedback and thoughts. I did initially write 7G using SetEnvIfNoCase directives, but there were certain sections (I forget which ones) that were not picking up. So I went with mod_rewrite as it is more universal in effectiveness. Hopefully you find some benefit with all of it, and thanks again for all of your inputs and ideas.

David 2019/04/09 12:08 pm • Reply

First of all id like to let you know how much technical ability i have ……… 0.3 :)

I recently built a website using DIVI. I hosted it with site-ground and they are the reason i find myself here. They recommend your 6G script and have it set out on their help pages so that it can be easily copied and pasted to the .htaccess file (which i have done)

Obviously, id like to set uo the 7G but when it comes to the set up here im lost. Is there anyway to get it so i can copy and paste directly with out getting blindsided with technical instructions.

Apologies for troubling you in advance!

Jeff Starr 2019/04/09 3:31 pm • Post Author • Reply

I wrote the instructions to be as clear as possible, but they do assume a certain level of understanding. If anything is not clear in the installation steps let me know and I will try to expand/clarify. Otherwise it should be recommended to get an experienced developer to implement properly.

Bernd 2020/01/05 2:35 pm • Reply

Hi Jeff, concerns 7g_log.php My server was telling me that there is a problem in line 171

$string = (!empty(SEVENGUALENGTH)) ? substr($string, 0, SEVENGUALENGTH) : $string;

using your defaults

define('SEVENGUALENGTH', 0);

For troubleshooting I commented this line out and every works fine now.

Any idea what’ wrong?

Bernd.

Jeff Starr 2020/01/05 3:58 pm • Post Author • Reply

Hi Bernd, if you can be more specific and provide the actual PHP error/warning, I will take a look asap. Thank you.
- Bernd 2020/01/06 5:54 am
  
  Jeff, the server errorlog file says this
  [Mon Jan 06 14:49:17 2020] [warn] mod_fcgid: stderr: PHP Parse error: syntax error, unexpected ')', expecting :: (T_PAAMAYIM_NEKUDOTAYIM) in xxxx/7g_log.php on line 171
- Jeff Starr 2020/01/06 8:57 am • Post Author
  
  Thanks I am looking into this now. Just glancing at line 171 I cannot find any error with the default download script. Will run full tests later and report back. In the meantime, can you let me know the version of PHP and Apache you are using, or send via email if preferred. Thank you Bernd.
- Bernd 2020/01/06 9:24 am
  
  well Sys Info is telling this
  Server OS Apache
  PHP Version 5.4.45
- Jeff Starr 2020/01/06 11:17 am • Post Author
  
  Thanks. It could be an issue with the old/outdated PHP version (5.4 is pretty old). So you may want to try re-downloading the script and/or upgrading PHP to something newer (like v7+). The script runs fine (no errors or warnings) on PHP 7 and better.
- Bernd 2020/01/06 12:12 pm
  
  this is a service provider and I have no chance to ask for an update.
  Jeff, did you made any update on your 7G_log.php file?
  Best Regards Bernd.
- Jeff Starr 2020/01/07 1:34 am • Post Author
  
  No because I did not find any error (running PHP 7.3). I’ll try to set up an old PHP version and retest at some point, likely for the next 7G update.

Alexey 2023/02/06 5:16 pm • Reply

Since the site is located behind the CND, I corrected the log function. Maybe something like this will be useful to you in your projects.

function perishablePress_7G_ip_address() {
	
	$string = isset($_SERVER['REMOTE_ADDR']) ? $_SERVER['REMOTE_ADDR'] : '';
	
	// If connecting through Cloudflare, rely on Cloudflare's connecting-IP header to get IP, otherwise check server header
	
	if(isset($_SERVER['HTTP_CF_CONNECTING_IP'])) {
		
		$string = '[Cloudflare reports] '. $_SERVER['HTTP_CF_CONNECTING_IP'] .' / '. $string;
		
	} else {
		
		//
	}
	
	$match = isset($_SERVER['REDIRECT_7G_IP_ADDRESS']) ? $_SERVER['REDIRECT_7G_IP_ADDRESS'] : '';
	
	return perishablePress_7G_get_patterns($string, $match);
	
}

Anonymous User 2023/02/06 5:27 pm • Reply

Editor’s note: This information was sent in by a 7G user. I am unable to locate the original email/name. If it’s you, let me know and I’ll be glad to update the comment with your infos.</note>

I was continuing to have issues with 7G log not [bracketing] correctly. I traced it down to Apache configuration differences. I searched for “apache adds REDIRECT_ to environment variables”. One article which explains the issue.

In short, my config of Open Litespeed does not prepend REDIRECT_ to some environment vars, and I think that is the case for some apache configs. To fix, I replaced all REDIRECT_ with empty in 7G_log.php. I’d imagine there is some rewrite you could do to account for both possibilities depending on varying configs in the next update.

Update: I tracked it down to the use of REDIRECT_QUERY_STRING vs QUERY_STRING in perishablePress_7G_check(). Below is part of the $_SERVER array. I changed the code in 7G_log.php to check against QUERY_STRING and it now works fine. I don’t know if this is a Litespeed thing, but I thought you might want to know in case it’s a larger bug. If you need more of the $_SERVER array, let me know.

[REQUEST_URI] => /?browsersploit
[REDIRECT_URL] => /
[REDIRECT_QUERY_STRING] => browsersploit
[HTTPS] => on
[7G_QUERY_STRING] => browsersploit______
[REDIRECT_STATUS] => 200
[SSL_PROTOCOL] => TLSv1.3
[SSL_CIPHER] => TLS_CHACHA20_POLY1305_SHA256
[SSL_CIPHER_USEKEYSIZE] => 256
[SSL_CIPHER_ALGKEYSIZE] => 256
[LSWS_EDITION] => Openlitespeed 1.6.21
[X-LSCACHE] => on,crawler
[QUERY_STRING] => log
[SCRIPT_NAME] => /7G_log.php
[SERVER_PROTOCOL] => HTTP/1.1
[SERVER_SOFTWARE] => LiteSpeed
[REQUEST_METHOD] => GET
[PHP_SELF] => /7G_log.php

George 2023/02/11 1:59 pm • Reply

Also just fyi, on some servers, user-declared environment variables must start with HTTP_ for security purposes, e.g.: SetEnv HTTP_MY_VARIABLE "my value"

7G User 2023/02/06 5:38 pm • Reply

In order to get logging-script working, I have to set in htaccess:

RewriteRule .* - [E=REDIRECT_QUERY_STRING:log]

Seems to be undocumented or did I miss something to read?

Jeff Starr 2023/02/06 5:41 pm • Post Author • Reply
Thanks for sharing this, also note that there are additional environmental variables for each set of firewall rules, for example:
```
$_SERVER['REDIRECT_7G_QUERY_STRING']
$_SERVER['REDIRECT_7G_REQUEST_URI']
$_SERVER['REDIRECT_7G_USER_AGENT']
.
.
.
```
So you would want to add similar rewrite rules for each.

Luca 2023/05/24 1:33 am • Reply

Hi, I have a website with many spam pages in Excluded by “noindex” tags like these: “?s=%E5%A4%A7%E5%A5%96%E5%A8%B1%E4%B9%90% E6%98%AF%E8%80%81%E5%93%81%E7%89%8C%E4%B9%88+Q82019309.com.com” when I installed 7G firewall I noticed that the spam pages in Excluded by “noindex” tag decreased but increased on 404 pages in Google Serach Console.

Is it possible to set the code by sending these spam pages to 410?
Thanks

Jeff Starr 2023/05/24 1:06 pm • Post Author • Reply

Yes it’s possible. As-is, the firewall sends a 403 Forbidden response to all blocked requests. To change that, replace each instance of [F] with [R=410,L] or similar.

Marios 2023/12/01 10:46 am • Reply

Just a suggestion… space dash space isn’t a great delimiter if you’re going to use something like Excel to put things into nice columns for filtering/searching. I’ve modified the script to use a pipe.

Thanks for all you do and the information you’ve provided.