One way to prevent Google from crawling certain pages is to use <meta> elements in the <head> section of your web documents. For example, if I want to prevent Google from indexing and archiving a certain page, I would add the following code to the head of my document:
<meta name="googlebot" content="noindex,noarchive" />
I’m no SEO guru, but it is my general understanding that it is possible to manipulate the flow of page rank throughout a site through strategic implementation of <meta> directives.
Continue Reading
Controlling the spidering, indexing and caching of your (X)HTML-based web pages is possible with meta robots directives such as these:
<meta name="googlebot" content="index,archive,follow,noodp">
<meta name="robots" content="all,index,follow">
<meta name="msnbot" content="all,index,follow">
I use these directives here at Perishable Press and they continue to serve me well for controlling how the “big bots” 1 crawl and represent my (X)HTML-based content in search results.
For other, non-(X)HTML types of content, however, using meta robots directives to control indexing and caching is not an option. An excellent example of this involves directing Google to index and cache PDF documents. The last time I checked, meta tags can’t be added to PDFs, Word documents, Excel documents, text files, and other non-(X)HTML-based content. The solution, of course, is to take advantage of the relatively new 2 HTTP header, X-Robots-Tag.
Continue Reading
Want to make sure that your feeds are not indexed by Google and other compliant search engines? Add the following code to the channel element of your XML-based (RSS, etc.) feeds:
<xhtml:meta xmlns:xhtml="http://www.w3.org/1999/xhtml" name="robots" content="noindex" />
Here is an example of how I use this tag for Perishable Press feeds (vertical spacing added for emphasis):
Continue Reading
During the most recent Perishable Press redesign, I noticed that several of my WordPress admin pages had been assigned significant levels of PageRank. Not good. After some investigation, I realized that my ancient robots.txt rules were insufficient in preventing Google from indexing various WordPress admin pages. Specifically, the following pages have been indexed and subsequently assigned PageRank:
- WP Admin Login Page
- WP Lost Password Page
- WP Registration Page
- WP Admin Dashboard
Needless to say, it is important to stop WordPress from leaking PageRank to admin pages. Instead of wasting our hard-earned link-equity on non-ranking pages, let’s redirect it to more important pages and posts. In order to accomplish this, we will attack the problem on three different fronts: admin links, robots.txt rules, and meta tags. Let’s take a look at each of these methods..
Continue Reading
This XHTML header tags resource is a work in progress, perpetually expanding and evolving as new information is obtained, explored, and integrated. Hopefully, you will find it useful in some way. Even better, perhaps you will share any complimentary or critical information concerning the contents of this article.
Table of Contents
Continue Reading
Official Information & Download for the Head MetaData Plus WordPress Plugin
The Head MetaData Plus plugin is designed to complete a site’s head construct by including some of the more obscure <meta> tags, such as “author” and “copyright.” As a matter of practicality, the more widely used tags such as “description” and “keywords” have been omitted, as they are already present at most WP-powered sites. The metadata output via this plugin applies to the entire site.
Installation: Upload the “head-metadata-plus.php” file to your WordPress Plugins directory, activate via WP Admin, and enjoy.
Customization: Open the “head-metadata-plus.php” file and edit any of the meta tag attributes located in the “variables” section.
Adding/Removing Meta tags: Removing unwanted tags is easy. Just comment-out or delete the variable line AND its corresponding line in the echo function. Adding new tags involves emulating the existing pattern and adding a new variable and corresponding echo.
Please use the Press Contact Form for any questions, comments, criticisms, etc.
Download Head MetaData Plus [ version 1.0 | 3.3KB | .zip | 2067 downloads ]