Tell Google to Not Index Certain Parts of Your Page
by Jeff Starr on Sunday, August 23, 2009 – 27 Responses
There are several ways to instruct Google to stay away from various pages in your site:
- Robots.txt directives
- Nofollow attributes on links
- Meta noindex/nofollow directives
- X-Robots noindex/nofollow directives
..and so on. These directives all function in different ways, but they all serve the same basic purpose: control how Google crawls the various pages on your site. For example, you can use meta noindex to instruct Google not to index your sitemap, RSS feed, or any other page you wish. This level of control over which pages are crawled and indexed is helpful, but what if you need to control how Google crawls the contents of a specific page? Easy. Google enables us to do this with a set of googleon/googleoff tags.
About googleon and googleoff tags
Put simply, the googleon/googleoff tags tell GoogleBot Google Search Appliance when to start and stop indexing various parts of the web document. Consider the following example:
<p>This is normal (X)HTML content that will be indexed by Google.</p>
<!--googleoff: index-->
<p>This (X)HTML content will NOT be indexed by Google.</p>
<!--googleon: index>In this example, we see how the googleon/googleoff tags will prevent the second paragraph from being indexed by Google. Notice the “index” parameter, which may be set to any of the following:
- index — content surrounded by “
googleoff:index” will not be indexed by Google - anchor — anchor text for any links within a “
googleoff:anchor” area will not be associated with the target page - snippet — content surrounded by “
googleoff:snippet” will not be used to create snippets for search results - all — content surrounded by “
googleoff:all” are treated with all attributes: index, anchor, and snippet
Cool, eh? Let’s have a look at a specific usage example..
Using googleon and googleoff tags
Example 1: Blog Comments
Let’s say your comment threads tend to stray into off-topic conversation. Keeping your pages as tightly focused on the subject at hand is a great way to improve the on-page relevancy of your targeted keywords while improving the accuracy of matching search queries. Thus, to keep the superfluous banter out of the Google index, you could add googleon/googleoff tags as follows:
<!--googleoff: all-->
<div id="comments">
<p><strong>Nick Mason</strong> - August 2nd, 2009</p>
<p>From Her Majesty the queen. His boots were very clean.</p>
<p><strong>Rick Wright</strong> - August 3rd, 2009</p>
<p>Every year is getting shorter, never seem to find the time.</p>
<p><strong>David Gilmour</strong> - August 4th, 2009</p>
<p>By the river holding hands roll me up and lay me down.</p>
<p><strong>Roger Waters</strong> - August 5th, 2009</p>
<p>And after a while, you can work on points for style.</p>
</div>
<!--googleon: all>We definitely don’t want to see such a mindless thread in Google, and it will be interesting to see if the <pre> example gets dropped from the index..
May the Force be with You
While using this method to control how Google indexes your on-page content, there are a couple of things you should keep in mind. First, there is a difference between indexing and crawling. Google may crawl portions of your page that are demarcated with googleon/googleoff tags. So in case you were thinking that this technique may be a way to deal with Google’s new nofollow policy, forget about it — PageRank will continue to flow through any links contained within a googleoff zone.
Besides this, also keep in mind that this is a proprietary technique supported exclusively by Google Search Appliance. If you have content that you want to keep out of the index of all search engines, then you will need to find another way to do it. Eventually, Yahoo! and MSN/Live/Bing/Whatever may create proprietary “on/off tags” of their own, but chances are slim to none that they will obey the proprietary technology of the mighty Google.
For more information about googleon/googleoff tags, check out Google’s official documentation.
![[ Jeff Starr ]](http://images.monzilla.biz/serious/jeff-starr.png)
![[ Advertise Here ]](http://images.monzilla.biz/serious/adspot.png)
27 Responses
Add a comment
Donace – #1
That actually sounds pretty awesome! will need to add it to my ‘must needed improvements’
Jessi Hance – #2
It seems that these tags are for Google Search Appliance, whatever that is. Do you have any evidence that they work for Googlebot as well? It would be great if they do.
Jeff Starr – #3
*Sigh* - I figured it was too good to be true. In my excitement, I failed to realize that this technique only applies to the GSA. Ah well, life goes on.
I did update the post with the correct information, so thank you Jessi for pointing it out. Cheers.
Jessi Hance – #4
I got all excited with you, Jeff. Wonder why Googlebot doesn’t have something like this. It would be especially helpful for CMS sites.
My company’s site keeps getting weird Google results that have one title but go to a page with different content, all because we have sidebar modules that have teasers, like “latest reviews” and so on, that appear on many different pages in our site. I wish I could keep Googlebot from indexing those modules.
Jeff Starr – #5
Lol, yeah, sorry about the false alarm.
Google does need to implement something like this for their main index. If you think about it, the googleon/googleoff tags have been deemed worthy for enterprise use, so it seems logical that they would also benefit everyone else as well.
They are probably paranoid about potential “black-hat” abuse if it were available to the masses.
Michael Clark – #6
I got all excited for a moment. Alas. I asked for page section exclusion almost two years ago http://www.planetmike.com/goto/622 .
Jeff Starr – #7
Hi Michael, my apologies for the false hope. I have known about this technique for quite awhile, but always assumed that it targeted Googlebot. In fact, I had never even heard of the “Google Search Appliance” until today.
Enabling the googleon/googleoff tags only on GSA seems like a perfectly good technique wasted on a rather obscure aspect of search. Quite a disappointment imho.
fuzion – #8
There are a few other useful features that GSA has but the bot does not, such as meta tag searches. I think their main concern is the potential for black hat SEO abuse in these cases…
Jeff Starr – #9
Hi fuzion, that’s what I’m thinking too. Another case where a small group of people ruin it for the rest of us.
Also keep in mind that Google’s main purpose is to make money. Regardless of how useful a tool happens to be (as in the case of googleon/googleoff), if it threatens their bottom-line they’re not going to use it.
Jessi Hance – #10
Hi Jeff, I’m sure you’re right about this:
“if it threatens their bottom-line they’re not going to use it.”
And unfortunately, those of us adversely affected by lack of this tool need Google too much to try and organize a boycott.
I’ve actually been trying to convince my co-workers that we should design our pages with this problem in mind, and try not to put teaser content on a page we don’t want it associated with in Google results. Doesn’t seem like we should have to do that, but that’s life.
Jeff Starr – #11
Boycotting Google sounds futile, but it is something I have advocated in the past. Unfortunately, Google is too huge for anything like this to work. We will just have to wait until Google becomes fat and bloated and collapses from its own tremendous weight. Kinda like Microsoft.
In the meantime there’s always JavaScript to keep things away from teh Googlebot :)
Jessi Hance – #12
Jeff, I am a javascript ignoramus. How do you use it to hide stuff from Google? Is it worthy of a new how-to post from you? :)
Jeff Starr – #13
That’s an excellent question, and one that I recently covered while pondering a somewhat-related issue: Google’s new nofollow policy. One way to workaround the new nofollow dilemma is to use a slice of external JavaScript, which is mostly inaccessible to Googlebot at this time (may change in the future). Check out this post for a discussion of this along with an easy way to use JavaScript for non-indexed content.
Jessi Hance – #14
Thanks so much, Jeff! I’ll check out that article for sure. Thanks for being so hugely helpful, as always. You are a treasure!
fuzion – #15
Religious use of Scroogle is the extent of my own personal big G boycott.
https://addons.mozilla.org/en-US/firefox/addon/12506
http://userscripts.org/scripts/show/23529
Well, that and a few angry blog posts from back when Chrome came out and they banned my Adsense account, taking ~$500 with it. ;)
Jeff Starr – #16
Scroogle looks pretty sweet. I installed it and will be trying it out. Thanks!
Sorry to hear about your Adsense account — I had no idea they would do such a thing. Were you able to get the money back and/or get your account re-activated?
fuzion – #17
heh, no.. I joined the legions of other banned users that got no response whatsoever. I still email them once in a while, but they never reply.
Google’s in a sad state these days, doing what they can to keep the baddies from exploiting their services, while at the same time totally sacrificing their normal user experience. Did you know that you can’t create a gmail account these days without giving them your cell phone number? It only affects IP’s with an account already associated, but think about the impact on college or corporate networks… I guess data mining is yet another industry not affected by our financial depression.
Mark – #18
You can adjust the importance of content using section targeting:
“Section targeting allows you to suggest sections of your text and HTML content that you’d like us to emphasize or downplay when matching ads to your site’s content. By providing us with your suggestions, you can assist us in improving your ad targeting. We recommend that only those familiar with HTML attempt to implement section targeting.
To implement section targeting, you’ll need to add a set of special HTML comment tags to your code. These tags will mark the beginning and end of whichever section(s) you’d like to emphasize or de-emphasize for ad targeting.
The HTML tags to emphasize a page section take the following format:
<!-- google_ad_section_start --><!-- google_ad_section_end -->”
https://www.google.com/adsense/support/bin/answer.py?hl=en&answer=23168
I use this on Scamdex - I was unaware of the GSA options.
Jeff Starr – #19
@fuzion: yes, I sense that Google is peaking these days, meaning that things are about as good as they’re going to get, and that it’s only downhill from here. I have been reading many, many negative reports about Google lately (including yours), something which was pretty scarce even just a couple of years ago. From what I can tell, they are selling out to every big business deal they can get their hands on and have shifted from a proactive strategy to a reactive one. Instead of going out of their way to do their best, they simply make more rules, lock things down, and call it good. This is only my opinion, but this general trend has been seen in many corporations and even celebrity entities, actors, rock stars, etc. With social media, it wouldn’t take much for a new lean, mean search engine to catch fire and cleave a huge chunk out of Google’s pie.
Jeff Starr – #20
@Mark: Interesting, I was not aware of this either. So it looks like the ad_sections apply only to sites participating with Adsense? Have you seen any positive effects since using these tags? Do you think they are effective?
vertical media solutions – #21
This is such an amazing piece of information. I would definitely try my hands at this cheeky little tip of controlling the google’s search mechanism. Thanks a lot for sharing this stuff.
Ted Goas – #22
Wow, googleon and googleoff tags, I never knew!
Helen – #23
Hi Jeff,
I used googleoff to encumber certain expressions I don’t want to be indexed. For instance: In an article about the etymology of the word “w-h-o-r-e” I wanted to prevent people looking for professionals women entering my site.
Now I can see: It doesn’t work like everything Google does not work exactly.
I am turning back to span the expression and spell it backwards and adding “
unicode-bidi: bidi-override;” and “direction: rtl;” in CSS.Bye, Helen
Jeff Starr – #24
Excellent tip, Helen — thanks for sharing :)
Al Sefati – #25
That seems to be good only if you have Google Search appliance installed according to this doc.
http://static.googleusercontent.com/external_content/untrusted_dlcp/www.google.com/en/us/enterprise/pdf/gsa_datasheet.pdf
Trackbacks / Pingbacks