Prevent Duplicate Content in cPanel

This is an exclusive guest post by , posted on July 3rd, 2012.

[ cPanel Addon-Domain Fix ] In this guest-post, Jon Brown shares a solution to the age-old problem of preventing duplicate content from addon-domains in cPanel. Jon explains the issue and shares his methodology in crafting an elegant solution applied via .htaccess. If you’re using cPanel and want to improve your SEO, this will help. Here is the table of contents:

The Problem

For those of you who do not know this tail of woe read on, for those that do, you will find the code you seek at the end of the article for both www and non-www versions.

The outcome of this article is a htaccess rewrite rule that completely removes any possibility of duplicate content issues caused by having 3 points of entry to your Addon domain, along with several variations when using cPanel hosting environment.

When you create an Addon domain the 3 possibly entry points are as follows:

  1. Folder on the main account domain
  2. Subdomain
  3. The Addon Domain’s URL

For the following instructions we will use the following cPanel example

  • Domain Namemytestdomain.co.uk
  • Subdomain/FTP Usernameukgroup
  • Document Root/Subfolder/mytestdomain.co.uk

The following is a complete list of the variations of the entry points:

  1. mytestdomain.co.uk
  2. www.mytestdomain.co.uk
  3. mytestdomain.com/mytestdomain.co.uk
  4. www.mytestdomain.com/mytestdomain.co.uk
  5. ukgroup.mytestdomain.com
  6. www.ukgroup.mytestdomain.com

And then if you want to be 100% correct all these variations:

  1. mytestdomain.co.uk/
  2. www.mytestdomain.co.uk/
  3. mytestdomain.com/mytestdomain.co.uk/
  4. www.mytestdomain.com/mytestdomain.co.uk/
  5. ukgroup.mytestdomain.com/
  6. www.ukgroup.mytestdomain.com/

When creating links to your website google can see these variations as two different sites, one with the trailing slash and one without. According to Matt Cutts, Apache webservers perform a 301 redirect from domains with no trailing slash to one with a trailing slash, this redirect adds a little bit of time on and will have a small effect. Matt in this video said Google can work out that they are the same site but it is better not to have the redirect. Also, later on when we cover rel="canonical" elements, Matt explains how URLs with slashes can be classed as entirely different websites, which could possibly lead to duplicate content.

The Research

I started off with using the basic redirect wizard in cPanel. This process just added rewrite rules into the main domain and the subdomain as noted below.

  • When you create redirects for the folder of the Addon domain, cPanel adds it to the main domains root htaccess.
  • When you redirects for the the subdomain, cPanel adds these rules in to the subdomains folder .htaccess file.

If you have quite a few Addon domains, you do not want to be going in to each one to set the redirect either by FTP or going through the cPanel procedure as this could take some time, so I progressively went through the rewrite rules cPanel creates until eventually I got one rule to manage both the folder and subdomain redirect and could run from the account’s main .htaccess file. This means you can add and manage duplicate content prevention for multiple Addon domains in one place with ease.

robots.txt

Now before we continue, I must mention the robots.txt method: dont use it. Firstly this will not stop Bad Bots scanning your site, not to mention the fact that if you advertise the folder in your robots.txt file, search-engines and bots will know the folder exists.

Google and other good bots will obey robots.txt directives, which will prevent the folder from being indexed by good bots; however, this will not prevent the subdomain iterations of your Addon domain being indexed if/when they are discovered. Further, the robots.txt-method can not compensate for www/non-www canonicalization.

Basically there is no benefit to use robots.txt to fix this issue, but there is at least one potential downside.

rel="canonical" Tag

I am not sure of the merits of the rel="canonical" in preventing duplicate content as the pages have already been displayed and loaded. In an excellent blog post, Matt Cutts explains in a video about the causes of duplicate content, and then elaborates on the canonical tag, which looks like this:

<link rel="canonical" href="http://example.com/page.html/">

The rel="canonical" tag is a hint, not a mandatory rule. When a bot sees this element, it does not have to obey. Matt Cutts says fixing the correct page location upstream is better than using this element, so the use of rel="canonical" is not a complete solution. This is similar to the situation with robots.txt: the canonical tag does not appear to have any downsides except having to add it to every page you want to redirect.

Methodology

In the following steps, I am going to show you how I went about building my cPanel duplicate content prevention for Addon domains rewrite rules. The reason I have outlined all the steps is so people can see how I got to the end code and might help for possibly improvements, which is always nice.

Step 1

Redirect folder and subdomains to Addon domain with cPanel wizard using default locations of htaccess

These are copies of what you get when you use the cPanel wizard to create redirects and is a good place to start (When in Rome!). I will not go through the process of how to create redirects in cPanel as one of the objectives of this article is to avoid that.

I knew what the problem was but was not sure of my syntax so I decided let cPanel do it for me and see what I get. What I got was two groups of rewrite code in two separate htaccess files, which is a good start. This code works but I want the code in a single rewrite group.

Folder Redirect

The folder redirect is placed within the domain’s root .htaccess file.

This method redirects all requests for the following URLs to the Addon domain (www.mytestdomain.co.uk used for example):

  • http://mytestdomain.com/mytestdomain.co.uk
  • http://www.mytestdomain.com/mytestdomain.co.uk
## Folder Redirect - Put this in the main .htaccess
RewriteCond %{HTTP_HOST} ^mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.mytestdomain.com$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/www\.mytestdomain\.co\.uk\/$1" [R=301,L]

Subdomain Redirect

The redirects for any subdomains should be placed into the .htaccess file of the subfolder of your main domain, which is also the root of your Addon domain. Once in place, the following ruleset will redirect both iterations of the subdomain (ukgroup. and www.ukgroup.) to the primary domain, www.mytestdomain.co.uk:

## Subdomain Redirect - Put this in the Addon domain's folder
RewriteCond %{HTTP_HOST} ^ukgroup.mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.ukgroup.mytestdomain.com$
RewriteRule ^(.*)$ "http\:\/\/www\.mytestdomain\.co\.uk\/$1" [R=301,L]

NB: when you run this from ukgroup.lancastriangroup.com it redirects properly, but if you move the code to the main .htaccess file in your root (for your main domain) it adds mytestdomain.co.uk to the end of the URL.

So we can prevent duplicate content with these two rules. Setting up these rules for each of your Addon domains can be time consuming and possibly hard to manage, so on to the next step to further improve the method.

Step 2

Change subdomain rule so the redirect can be performed from the main directory

In this section I have altered the subdomain code created by cPanel so I can place the code into the main .htaccess file.

When this htaccess rule is called the domain name of the request is checked against ukgroup.mytestdomain.com and www.ukgroup.mytestdomain.com; if it matches, the URL is rewritten and then the session is 301-redirected to www.mytestdomain.co.uk.

  • Put this in the main domain’s root folder htaccess (remove any redirects you have made in your Addon domain’s .htaccess previously)
  • Note it is still seeing the Addon domain as a folder not as a subdomain
## Subdomain Redirect - Put this in the main .htaccess
RewriteCond %{HTTP_HOST} ^ukgroup.mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.ukgroup.mytestdomain.com$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/www.mytestdomain\.co\.uk\/$1" [R=301,L]

Step 3

Try combining both rule sets in to one

We have now got to a stage where we have both rules (subdomain and folder redirect) are in the main .htacccess file, but why have two rules when you can have one.

## Domains redirect AIO - Put this in the main .htaccess
RewriteCond %{HTTP_HOST} ^mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^ukgroup.mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.ukgroup.mytestdomain.com$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/www\.mytestdomain\.co\.uk\/$1" [R=301,L]

This can be reduced to :

## Domains redirect AIO (reduced) - Put this in the main .htaccess
RewriteCond %{HTTP_HOST} ^(www\.|ukgroup\.|www\.ukgroup\.)?mytestdomain.com$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/www\.mytestdomain\.co\.uk\/$1" [R=301,L]

Here, I’ve introduced the variable %{REQUEST_URI} to allow detection of the folder. This redirects all the required subdomains (www & non-www) and subfolder, and works from the main htaccess file:

# Subdomain and subfolder redirect -  Put this in the main .htaccess
RewriteCond %{HTTP_HOST} ^(www\.|ukgroup\.|www\.ukgroup\.)?mytestdomain.com$ [OR]
RewriteCond %{REQUEST_URI} ^mytestdomain\.co\.uk\/?(.*)$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/www\.mytestdomain\.co\.uk\/$1" [R=301,L]

My testing must have been flawed as the %{REQUEST_URI} statement is not needed as the rule works fine without it. The rewrite rule is very specific and is how the cPanel wizard created it in the beginning of this article — I keep forgetting that the rewrite rule will act upon the subfolder (this may be a quirk in cPanel). So we return to the previous code knowing that we do not need to involve %{REQUEST_URI} for the folder.

## Domains redirect AIO (reduced) - Put this in the main .htaccess
RewriteCond %{HTTP_HOST} ^(www\.|ukgroup\.|www\.ukgroup\.)?mytestdomain.com$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/www\.mytestdomain\.co\.uk\/$1" [R=301,L]

Step 4

Different named subdomain and folder — testing how cPanel writes

I have throughout the article maintained the default folder name that is given to the Addon domain (usually the domain name itself), and I just wanted to see what happens when I change it because some people might not want to use the domain name for the folder and this section is to compensate for that.. will things stop working as expected!

So I will now use green as the subfolder to store the Addon domain in, and keep using ukgroup for the subdomain as the subdomain rule is very easy to follow.

#redirect all subdomain/Addon traffic to its correct www
RewriteCond %{HTTP_HOST} ^(www\.|ukgroup\.|www\.ukgroup\.)?mytestdomain\.com$
RewriteRule ^green\/?(.*)$ http://www.mytestdomain.co.uk/$1 [R=301,L]
  • After testing I came to the conclusion it is the rewrite rule that holds the key, by this I mean that the rewrite rule is acting on the folder name of the hosted sub domain and by this mechanism allows for a unified rule for duplicate content.
  • All test links correctly redirect.

So what this little section proves is that the rule rewrites the folder name and as a consequence you can use any folder name as long as you alter the last line in the rewrite rule to match the folder as in the example above.

Step 5

Adding Canonical Redirect

The code so far that I have developed will 301-redirect any of the entry points to a specified domain. The issue of duplicate content on the Addon domain is not limited to entry points from outside of the Addon domain but can also suffer from a traditional duplicate content issue caused by www or non-www endpoint.

What I mean by this is that in my examples I have forwarded all the content to www.mytestdomain.co.uk but mytestdomain.co.uk is also a valid domain for the same content and does not have a 301 to redirect it to the www-version. This is a normal problem for web developers. You could now just add a canonical redirect to either the main htaccess or the subfolder htaccess to solve this problem but I prefer to add it to my code to make evrything neater and in one place. Why have two when one will do.

This redirects all traffic to the www version of the Addon domain

RewriteCond %{HTTP_HOST} ^(www\.|ukgroup\.|www\.ukgroup\.)?mytestdomain\.com$ [OR]
RewriteCond %{HTTP_HOST} ^mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ http://www.mytestdomain.co.uk/$1 [R=301,L]

This redirects all traffic to the non-www version of the Addon domain

RewriteCond %{HTTP_HOST} ^(www\.|ukgroup\.|www\.ukgroup\.)?mytestdomain\.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ http://mytestdomain.co.uk/$1 [R=301,L]

Step 6

Improving the Code

In this section I will now take the code I have and process it so it is more streamline after taking advice from Jeff. I will concentrate on the version that redirects to the www-version

  • Domain Namemytestdomain.co.uk
  • Subdomain/FTP Usernameukgroup
  • Document Root/Subfolder/mytestdomain.co.uk

Start here; this does not have the fully escaped endpoint URL like the one cPanel creates

RewriteCond %{HTTP_HOST} ^(www\.|ukgroup\.|www\.ukgroup\.)?mytestdomain\.com$ [OR]
RewriteCond %{HTTP_HOST} ^mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ http://www.mytestdomain.co.uk/$1 [R=301,L]

Now compress the subdomain condition

RewriteCond %{HTTP_HOST} ^(www\.)?(ukgroup\.)?mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ http://www.mytestdomain.co.uk/$1 [R=301,L]

Now remove unneeded escaped slash on the last line:

RewriteCond %{HTTP_HOST} ^(www\.)?(ukgroup\.)?mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk/?(.*)$ http://www.mytestdomain.co.uk/$1 [R=301,L]

..almost there..

Finished Code

Well here it is, the working code. All you have to do is alter the following details to match your domain etc.. and then put this in your main htaccess file. Notes:

  1. On the first condition change ukgroup to match you subdomain/FTP user
  2. Change mytestdomain.com to match your domain
  3. Change the domain name on the second condition to match your Addon domain (do not modify the www)
  4. The first part of the rewrite rule should be your folder name
  5. The second part of the rewrite rule should be your Addon domain

I used the following variables for the final code below.

  • Domain Namemytestdomain.co.uk
  • Subdomain/FTP Usernameukgroup
  • Document Root/Subfolder/mytestdomain.co.uk

WWW Version

## cPanel Prevent Addon Domain Duplicate Content  –  Redirect all to www domain
RewriteCond %{HTTP_HOST} ^(www\.)?(ukgroup\.)?mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk/?(.*)$ http://www.mytestdomain.co.uk/$1 [R=301,L]

Non-WWW Version

## cPanel Prevent Addon Domain Duplicate Content  –  Redirect all to non-www domain
RewriteCond %{HTTP_HOST} ^(www\.)?(ukgroup\.)?mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^www\.mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk/?(.*)$ http://mytestdomain.co.uk/$1 [R=301,L]

Notes..

Just for reference — here are the fully escaped codes using the cPanel paradigm.

## with cannonical www redirect
RewriteCond %{HTTP_HOST} ^(www\.)?(ukgroup\.)?mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/www\.mytestdomain\.co\.uk\/$1" [R=301,L]
## with cannonical non-www redirect
RewriteCond %{HTTP_HOST} ^(www\.)?(ukgroup\.)?mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^www\.mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/mytestdomain\.co\.uk\/$1" [R=301,L]

The cPanel virtualisation of the Addon domain must be separate. That is, when the incoming request comes in for the Addon domain it serves it correctly but where the Addon domain name is not used it assumes the document root as the URL to be rewritten.

  • The handling when Addon domain is present in the request, act normally like the main domain
  • When the Addon domain name is not present in the request, the cPanel uses the document root as if the Addon domain does not exist

Testing domains and links work

These links allows us to test the domain redirects are working as they should be, both the main and Addon domains:

mytestdomain.com
www.mytestdomain.com
mytestdomain.co.uk
www.mytestdomain.co.uk
ukgroup.mytestdomain.com
www.ukgroup.mytestdomain.com
mytestdomain.com/mytestdomain.co.uk
mytestdomain.com/mytestdomain.co.uk/
www.mytestdomain.com/mytestdomain.co.uk
www.mytestdomain.com/mytestdomain.co.uk/

Now with files

mytestdomain.com/test1234
www.mytestdomain.com/test1234
mytestdomain.co.uk/test1234
www.mytestdomain.co.uk/test1234
ukgroup.mytestdomain.com/test1234
www.ukgroup.mytestdomain.com/test1234
mytestdomain.com/mytestdomain.co.uk/test1234
www.mytestdomain.com/mytestdomain.co.uk/test1234