Latest Tweets404 Fix: Block Nuisance Requests for Non-Existent Files: perishablepress.com/block-nuis…
Perishable Press

Prevent Duplicate Content in cPanel

Jon Brown Exclusive guest post by .

[ cPanel Addon-Domain Fix ] In this guest-post, Jon Brown shares a solution to the age-old problem of preventing duplicate content from addon-domains in cPanel. Jon explains the issue and shares his methodology in crafting an elegant solution applied via .htaccess. If you’re using cPanel and want to improve your SEO, this will help. Here is the table of contents:

The Problem

For those of you who do not know this tail of woe read on, for those that do, you will find the code you seek at the end of the article for both www and non-www versions.

The outcome of this article is a htaccess rewrite rule that completely removes any possibility of duplicate content issues caused by having 3 points of entry to your Addon domain, along with several variations when using cPanel hosting environment.

When you create an Addon domain the 3 possibly entry points are as follows:

  1. Folder on the main account domain
  2. Subdomain
  3. The Addon Domain’s URL

For the following instructions we will use the following cPanel example

  • Domain Namemytestdomain.co.uk
  • Subdomain/FTP Usernameukgroup
  • Document Root/Subfolder/mytestdomain.co.uk

The following is a complete list of the variations of the entry points:

  1. mytestdomain.co.uk
  2. www.mytestdomain.co.uk
  3. mytestdomain.com/mytestdomain.co.uk
  4. www.mytestdomain.com/mytestdomain.co.uk
  5. ukgroup.mytestdomain.com
  6. www.ukgroup.mytestdomain.com

And then if you want to be 100% correct all these variations:

  1. mytestdomain.co.uk/
  2. www.mytestdomain.co.uk/
  3. mytestdomain.com/mytestdomain.co.uk/
  4. www.mytestdomain.com/mytestdomain.co.uk/
  5. ukgroup.mytestdomain.com/
  6. www.ukgroup.mytestdomain.com/

When creating links to your website google can see these variations as two different sites, one with the trailing slash and one without. According to Matt Cutts, Apache webservers perform a 301 redirect from domains with no trailing slash to one with a trailing slash, this redirect adds a little bit of time on and will have a small effect. Matt in this video said Google can work out that they are the same site but it is better not to have the redirect. Also, later on when we cover rel="canonical" elements, Matt explains how URLs with slashes can be classed as entirely different websites, which could possibly lead to duplicate content.

The Research

I started off with using the basic redirect wizard in cPanel. This process just added rewrite rules into the main domain and the subdomain as noted below.

  • When you create redirects for the folder of the Addon domain, cPanel adds it to the main domains root htaccess.
  • When you redirects for the the subdomain, cPanel adds these rules in to the subdomains folder .htaccess file.

If you have quite a few Addon domains, you do not want to be going in to each one to set the redirect either by FTP or going through the cPanel procedure as this could take some time, so I progressively went through the rewrite rules cPanel creates until eventually I got one rule to manage both the folder and subdomain redirect and could run from the account’s main .htaccess file. This means you can add and manage duplicate content prevention for multiple Addon domains in one place with ease.

robots.txt

Now before we continue, I must mention the robots.txt method: dont use it. Firstly this will not stop Bad Bots scanning your site, not to mention the fact that if you advertise the folder in your robots.txt file, search-engines and bots will know the folder exists.

Google and other good bots will obey robots.txt directives, which will prevent the folder from being indexed by good bots; however, this will not prevent the subdomain iterations of your Addon domain being indexed if/when they are discovered. Further, the robots.txt-method can not compensate for www/non-www canonicalization.

Basically there is no benefit to use robots.txt to fix this issue, but there is at least one potential downside.

rel="canonical" Tag

I am not sure of the merits of the rel="canonical" in preventing duplicate content as the pages have already been displayed and loaded. In an excellent blog post, Matt Cutts explains in a video about the causes of duplicate content, and then elaborates on the canonical tag, which looks like this:

<link rel="canonical" href="http://example.com/page.html/">

The rel="canonical" tag is a hint, not a mandatory rule. When a bot sees this element, it does not have to obey. Matt Cutts says fixing the correct page location upstream is better than using this element, so the use of rel="canonical" is not a complete solution. This is similar to the situation with robots.txt: the canonical tag does not appear to have any downsides except having to add it to every page you want to redirect.

Methodology

In the following steps, I am going to show you how I went about building my cPanel duplicate content prevention for Addon domains rewrite rules. The reason I have outlined all the steps is so people can see how I got to the end code and might help for possibly improvements, which is always nice.

Step 1

Redirect folder and subdomains to Addon domain with cPanel wizard using default locations of htaccess

These are copies of what you get when you use the cPanel wizard to create redirects and is a good place to start (When in Rome!). I will not go through the process of how to create redirects in cPanel as one of the objectives of this article is to avoid that.

I knew what the problem was but was not sure of my syntax so I decided let cPanel do it for me and see what I get. What I got was two groups of rewrite code in two separate htaccess files, which is a good start. This code works but I want the code in a single rewrite group.

Folder Redirect

The folder redirect is placed within the domain’s root .htaccess file.

This method redirects all requests for the following URLs to the Addon domain (www.mytestdomain.co.uk used for example):

  • http://mytestdomain.com/mytestdomain.co.uk
  • http://www.mytestdomain.com/mytestdomain.co.uk
## Folder Redirect - Put this in the main .htaccess
RewriteCond %{HTTP_HOST} ^mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.mytestdomain.com$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/www\.mytestdomain\.co\.uk\/$1" [R=301,L]

Subdomain Redirect

The redirects for any subdomains should be placed into the .htaccess file of the subfolder of your main domain, which is also the root of your Addon domain. Once in place, the following ruleset will redirect both iterations of the subdomain (ukgroup. and www.ukgroup.) to the primary domain, www.mytestdomain.co.uk:

## Subdomain Redirect - Put this in the Addon domain's folder
RewriteCond %{HTTP_HOST} ^ukgroup.mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.ukgroup.mytestdomain.com$
RewriteRule ^(.*)$ "http\:\/\/www\.mytestdomain\.co\.uk\/$1" [R=301,L]

NB: when you run this from ukgroup.lancastriangroup.com it redirects properly, but if you move the code to the main .htaccess file in your root (for your main domain) it adds mytestdomain.co.uk to the end of the URL.

So we can prevent duplicate content with these two rules. Setting up these rules for each of your Addon domains can be time consuming and possibly hard to manage, so on to the next step to further improve the method.

Step 2

Change subdomain rule so the redirect can be performed from the main directory

In this section I have altered the subdomain code created by cPanel so I can place the code into the main .htaccess file.

When this htaccess rule is called the domain name of the request is checked against ukgroup.mytestdomain.com and www.ukgroup.mytestdomain.com; if it matches, the URL is rewritten and then the session is 301-redirected to www.mytestdomain.co.uk.

  • Put this in the main domain’s root folder htaccess (remove any redirects you have made in your Addon domain’s .htaccess previously)
  • Note it is still seeing the Addon domain as a folder not as a subdomain
## Subdomain Redirect - Put this in the main .htaccess
RewriteCond %{HTTP_HOST} ^ukgroup.mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.ukgroup.mytestdomain.com$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/www.mytestdomain\.co\.uk\/$1" [R=301,L]

Step 3

Try combining both rule sets in to one

We have now got to a stage where we have both rules (subdomain and folder redirect) are in the main .htacccess file, but why have two rules when you can have one.

## Domains redirect AIO - Put this in the main .htaccess
RewriteCond %{HTTP_HOST} ^mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^ukgroup.mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.ukgroup.mytestdomain.com$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/www\.mytestdomain\.co\.uk\/$1" [R=301,L]

This can be reduced to :

## Domains redirect AIO (reduced) - Put this in the main .htaccess
RewriteCond %{HTTP_HOST} ^(www\.|ukgroup\.|www\.ukgroup\.)?mytestdomain.com$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/www\.mytestdomain\.co\.uk\/$1" [R=301,L]

Here, I’ve introduced the variable %{REQUEST_URI} to allow detection of the folder. This redirects all the required subdomains (www & non-www) and subfolder, and works from the main htaccess file:

# Subdomain and subfolder redirect -  Put this in the main .htaccess
RewriteCond %{HTTP_HOST} ^(www\.|ukgroup\.|www\.ukgroup\.)?mytestdomain.com$ [OR]
RewriteCond %{REQUEST_URI} ^mytestdomain\.co\.uk\/?(.*)$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/www\.mytestdomain\.co\.uk\/$1" [R=301,L]

My testing must have been flawed as the %{REQUEST_URI} statement is not needed as the rule works fine without it. The rewrite rule is very specific and is how the cPanel wizard created it in the beginning of this article — I keep forgetting that the rewrite rule will act upon the subfolder (this may be a quirk in cPanel). So we return to the previous code knowing that we do not need to involve %{REQUEST_URI} for the folder.

## Domains redirect AIO (reduced) - Put this in the main .htaccess
RewriteCond %{HTTP_HOST} ^(www\.|ukgroup\.|www\.ukgroup\.)?mytestdomain.com$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/www\.mytestdomain\.co\.uk\/$1" [R=301,L]

Step 4

Different named subdomain and folder — testing how cPanel writes

I have throughout the article maintained the default folder name that is given to the Addon domain (usually the domain name itself), and I just wanted to see what happens when I change it because some people might not want to use the domain name for the folder and this section is to compensate for that.. will things stop working as expected!

So I will now use green as the subfolder to store the Addon domain in, and keep using ukgroup for the subdomain as the subdomain rule is very easy to follow.

#redirect all subdomain/Addon traffic to its correct www
RewriteCond %{HTTP_HOST} ^(www\.|ukgroup\.|www\.ukgroup\.)?mytestdomain\.com$
RewriteRule ^green\/?(.*)$ http://www.mytestdomain.co.uk/$1 [R=301,L]
  • After testing I came to the conclusion it is the rewrite rule that holds the key, by this I mean that the rewrite rule is acting on the folder name of the hosted sub domain and by this mechanism allows for a unified rule for duplicate content.
  • All test links correctly redirect.

So what this little section proves is that the rule rewrites the folder name and as a consequence you can use any folder name as long as you alter the last line in the rewrite rule to match the folder as in the example above.

Step 5

Adding Canonical Redirect

The code so far that I have developed will 301-redirect any of the entry points to a specified domain. The issue of duplicate content on the Addon domain is not limited to entry points from outside of the Addon domain but can also suffer from a traditional duplicate content issue caused by www or non-www endpoint.

What I mean by this is that in my examples I have forwarded all the content to www.mytestdomain.co.uk but mytestdomain.co.uk is also a valid domain for the same content and does not have a 301 to redirect it to the www-version. This is a normal problem for web developers. You could now just add a canonical redirect to either the main htaccess or the subfolder htaccess to solve this problem but I prefer to add it to my code to make evrything neater and in one place. Why have two when one will do.

This redirects all traffic to the www version of the Addon domain

RewriteCond %{HTTP_HOST} ^(www\.|ukgroup\.|www\.ukgroup\.)?mytestdomain\.com$ [OR]
RewriteCond %{HTTP_HOST} ^mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ http://www.mytestdomain.co.uk/$1 [R=301,L]

This redirects all traffic to the non-www version of the Addon domain

RewriteCond %{HTTP_HOST} ^(www\.|ukgroup\.|www\.ukgroup\.)?mytestdomain\.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ http://mytestdomain.co.uk/$1 [R=301,L]

Step 6

Improving the Code

In this section I will now take the code I have and process it so it is more streamline after taking advice from Jeff. I will concentrate on the version that redirects to the www-version

  • Domain Namemytestdomain.co.uk
  • Subdomain/FTP Usernameukgroup
  • Document Root/Subfolder/mytestdomain.co.uk

Start here; this does not have the fully escaped endpoint URL like the one cPanel creates

RewriteCond %{HTTP_HOST} ^(www\.|ukgroup\.|www\.ukgroup\.)?mytestdomain\.com$ [OR]
RewriteCond %{HTTP_HOST} ^mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ http://www.mytestdomain.co.uk/$1 [R=301,L]

Now compress the subdomain condition

RewriteCond %{HTTP_HOST} ^(www\.)?(ukgroup\.)?mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ http://www.mytestdomain.co.uk/$1 [R=301,L]

Now remove unneeded escaped slash on the last line:

RewriteCond %{HTTP_HOST} ^(www\.)?(ukgroup\.)?mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk/?(.*)$ http://www.mytestdomain.co.uk/$1 [R=301,L]

..almost there..

Finished Code

Well here it is, the working code. All you have to do is alter the following details to match your domain etc.. and then put this in your main htaccess file. Notes:

  1. On the first condition change ukgroup to match you subdomain/FTP user
  2. Change mytestdomain.com to match your domain
  3. Change the domain name on the second condition to match your Addon domain (do not modify the www)
  4. The first part of the rewrite rule should be your folder name
  5. The second part of the rewrite rule should be your Addon domain

I used the following variables for the final code below.

  • Domain Namemytestdomain.co.uk
  • Subdomain/FTP Usernameukgroup
  • Document Root/Subfolder/mytestdomain.co.uk

WWW Version

## cPanel Prevent Addon Domain Duplicate Content  –  Redirect all to www domain
RewriteCond %{HTTP_HOST} ^(www\.)?(ukgroup\.)?mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk/?(.*)$ http://www.mytestdomain.co.uk/$1 [R=301,L]

Non-WWW Version

## cPanel Prevent Addon Domain Duplicate Content  –  Redirect all to non-www domain
RewriteCond %{HTTP_HOST} ^(www\.)?(ukgroup\.)?mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^www\.mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk/?(.*)$ http://mytestdomain.co.uk/$1 [R=301,L]

Notes..

Just for reference — here are the fully escaped codes using the cPanel paradigm.

## with cannonical www redirect
RewriteCond %{HTTP_HOST} ^(www\.)?(ukgroup\.)?mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/www\.mytestdomain\.co\.uk\/$1" [R=301,L]
## with cannonical non-www redirect
RewriteCond %{HTTP_HOST} ^(www\.)?(ukgroup\.)?mytestdomain.com$ [OR]
RewriteCond %{HTTP_HOST} ^www\.mytestdomain\.co\.uk$
RewriteRule ^mytestdomain\.co\.uk\/?(.*)$ "http\:\/\/mytestdomain\.co\.uk\/$1" [R=301,L]

The cPanel virtualisation of the Addon domain must be separate. That is, when the incoming request comes in for the Addon domain it serves it correctly but where the Addon domain name is not used it assumes the document root as the URL to be rewritten.

  • The handling when Addon domain is present in the request, act normally like the main domain
  • When the Addon domain name is not present in the request, the cPanel uses the document root as if the Addon domain does not exist

Testing domains and links work

These links allows us to test the domain redirects are working as they should be, both the main and Addon domains:

mytestdomain.com
www.mytestdomain.com
mytestdomain.co.uk
www.mytestdomain.co.uk
ukgroup.mytestdomain.com
www.ukgroup.mytestdomain.com
mytestdomain.com/mytestdomain.co.uk
mytestdomain.com/mytestdomain.co.uk/
www.mytestdomain.com/mytestdomain.co.uk
www.mytestdomain.com/mytestdomain.co.uk/

Now with files

mytestdomain.com/test1234
www.mytestdomain.com/test1234
mytestdomain.co.uk/test1234
www.mytestdomain.co.uk/test1234
ukgroup.mytestdomain.com/test1234
www.ukgroup.mytestdomain.com/test1234
mytestdomain.com/mytestdomain.co.uk/test1234
www.mytestdomain.com/mytestdomain.co.uk/test1234

Jon Brown
About the Author My name is Jon Brown and I am an all-round IT guy from Lancashire in the North of England (UK). I have been self-employed and running my company, Lancastrian IT for about 10 years going to people's houses and small business fixing the large range of issues that these places can cause, but I have only really got in to web design and coding in the last 5 years. I am one of these people that if I find a problem I will keep going until I fix it, this almost certainly include late nights and usually some trial and error, maybe a few coffees as well!
Archives
12 responses
  1. airoschou July 3, 2012 @ 7:50 pm

    oh,my god
    i used robots.txt to Prevent Duplicate Content before…
    now i will try your this method

  2. Jon Brown July 11, 2012 @ 9:36 am

    HI Internet

    has anyone tried my code yet?

    i would appreciate some feedback.

    • Hi Jon,

      I tried out your code and am not able to get it to work.

      If both sites in question are WordPress sites, how do you fit the code in with the WordPress .htaccess file?

      What should your add-on domain .htaccess contain?

  3. I’m a bit confused about this as I never had any problem.

    1. If I create an add-on domain and take out public_html/ from the document root then it effectively sets it as a separate folder outside of the main site’s folder so there is no sub-folder problem.

    2. And for the subdomain you can just set it to redirect to the add-on domain directly from the cpanel subdomain section and it works fine..

    Maybe i’m missing the point of the article.. ?

    (slightly confused)

  4. David Sherwood July 18, 2012 @ 8:02 pm

    How could I use this in combination with a CDN service? The CDN purposely creates duplicate content on local servers to decrease page load time. However I want to “canonize” my original content (my website pages, files, pictures, and stuff) while avoiding the robots.txt on the CDN because it stops all of the possible “link juice” that helps with SEO. I was thinking of forcing the CDN files to and rel=canonical … with a 301 redirect? I’m not an IT guy, I’m a biologist diving into this new language and appreciate your work!

  5. Jon Brown July 24, 2012 @ 6:20 am

    I am not sure how this would work with a CDN setup but you should somewhere at least you should redirect traffic to just one site ie non-www to www.

    if you use a normal cpanel account with addon domains and your CDN just presents the user with the closest version the files (ie joomla addon) this code should work as the code just alters the URL not the content, this also assumes your cpanel account is the primary target of the traffic before content is served.

  6. Lawrence Harmen July 25, 2012 @ 6:28 pm

    So is there anything wrong with this code from my web site :http://www.abbaswatchman.com/NEWS%20MAIN%20PAGE%20WORLD.htm
    I had also found a web site anylyser that sugested I might have a tracking image an you take alook if you know what this means . Thank you lharmen.
    http://www.abbaswatchman.com/
    span.mark { background-position:
    center bottom; background-color:transparent; background-repeat:repeat-x; background-attachment:scroll }
    body.custom-background { background-color: #a7affb; background-repeat: repeat; background-position: left top; background-attachment: fixed }
    div.post{word-wrap:break-word;}div.post{display:block;margin:0 0 30px 0}.clearfix{min-width:0;display:inline-block;display:block}div.post-bodycopy{}* html .clearfix{height:1%;}.yog-bd{background-position: center top; background-color:#fff; background-repeat:no-repeat; background-attachment:scroll } div.wrapper { margin: 0 auto; width: 982px;padding:0;}
    .IPA {

  7. Absolutely took care of the subdomain!

    But the redirect for the folder doesn’t work for me :(

  8. This article was a bit abstract for me to wrap my head around but then I’m not always the sharpest knife in the drawer, so to speak ;)

    I WANT to use my subdomain name, it is there for SEO reasons and most people find it “catchy”

    I DO NOT WANT duplicate content being leaked out of my site to the googlebot. (obviously). I use a shared hosting CPanel environment, self-hosted WordPress, a CDN and two very popular plugins, WordPress SEO (Yoast) and W3TC. WordPress SEO establishes some canonical rules, I’m not sure W3TC has anything directly to do with anything here, but it might be worth mentioning that it controls the CDN and my CDN host, MAXCDN defaults to no-indexing hosted files and my cookie domain is set to my origin site, so the problem of CDN dupe content seems non-existent.

    …but I digress

    These are the rules in my domain root to redirect requests for both www and non-www root to the subdomain:

    RewriteCond %{HTTP_HOST} ^locationscout.us$ [OR] RewriteCond %{HTTP_HOST} ^www.locationscout.us$ RewriteRule ^/?$ "http://nyc.locationscout.us/" [R=301,L]

    I dont have any complaints from Google Webmaster Tools about dupe content, is there anything I should be doing differently?

    The reason for the two versions of your final code is mysterious to me. What is the “CPanel Paradigm”? A Google for that term produced no results. The main differences I see in the two versions are the quotation marks around the urls. Please explain.

    The potential consequences of experimenting on my production site with changes outlined in your article seem a bit dire without having a complete understanding.

    But thanks in any case, let’s say for now that “I am intriqued”.

  9. Austin@plagspotter November 5, 2012 @ 4:16 am

    Excellent guide to stop duplicate content flow, thank you:)

  10. Thank you, I’m using wordpress and it already fix the www to no-www so I deleted the second line to avoid any problem.

  11. Hi again, as I already said this works great, but I was testing with dns name servers and come up with a solution:

    I’m using hostgator shared hosting, so i dont know if this will work for you.

    When you create your account on the hosting provider, we have to use the hosting name servers, it automatically creates all of our dns zones. When we add an addon domain it creates the dns zones for our domain pointing to our IP, and also it creates a subdomain of our main domain that serves the same folder as our addon domain.

    To solve this I just copied the dns zones of my main domain to another service (I’m using cloud flare) but i didn’t copied the dns records that create a subdomain of the main domain that points to the addon domain folder, that way when I type addondomain.maindomain.com the page cannot be found, but when I type addondomian.com it works as it should. BTW, dont forget to change the nameservers of your domain with your registrar.

    I’m not sure if this will work directly on cpanel using the hosting name servers and dns zone editor, i didn’t test it because i have active sites. If someone wants to test it all you have to do is delete the dns records of the subdomain of the main domain that points to the addon domain folder, something like addondomain.maindomain.com A xx.xx.xx and http://www.addondomain.maindoain.com A xx.xx.xx but please be carefull, maybe cpanel might delete the addon domain folder, so backup before any change. Dont test it on live sites.

    Anyways I think its better to manage dns with another service. Sorry for my bad english.

[ Comments are closed for this post ]