|[an error occurred while processing this directive]
by Brian D. Chmielewski
The foundation of our business has been built upon helping people get
their web sites indexed better within the spider-based search engines.
Yet, today we are going to show you how to prevent search engines
from indexing certain pages within your web site.
If you remember from our earlier discussion on how the search engines
work, you know that the spider based search engines utilize "robots"
(also called robot spiders or crawlers) to tirelessly and continually
index sites they find from the web.
But what if you have certain pages within your site that you do not
want to be put on the web? There are many potential reasons to keep
some pages from being indexed. For instance, you may want to prevent
people from seeing pages that are under construction. You may have
areas within your site that possess secure or proprietary information
about your company. Another example is Real estate agents often have
temporary pages for properties that they are selling but once the
properties are sold the page need not be found in a search engine.
So, what can you do to exclude some segments of your site from public
view, while allowing other segments of your site to be spidered for
indexing in a search engine's database? The answer is to set up the
instruments of the Robots Exclusion Standard, which allows robots
to identify portions of a site that should not be visited.
Termed the Robots Exclusion Protocol, this is a method that allows web
site administrators to indicate to visiting robots which parts of
their site should not be visited. When a robot first visits a web
site, it confirms its walking orders from the robot text file. If it
can find this file, it will analyze its contents to see if it may
retrieve further information. You can customize the robots.txt file to
apply only to specific robots, and to disallow access to specific
directories or files. Following is a portion of WebPromote's
robots.txt file. The entire file can be viewed at
# robots.txt for http://upromote.com/
All robots.txt records start with one or more user-agent lines,
specifying which robot(s) the record applies to. What's a user agent
line? It is simply a line that indicates which agent - the technical
name for a spider, crawler, scout, etc. - should follow the
robots.txt. To evaluate if access to a URL is allowed, a robot must
attempt to match the paths in the Allow and Disallow lines against the
URL, in the order they occur in the record. The first match found is
used. If no match is found, the default assumption is that the URL is
The symbol, "*" in the user-agent title indicates that all search
engines should be excluded. As mentioned above, you can have multiple
user-agent lines and change the "*" to a robot's specific name token,
(ex. Gulliver for Northernlight, ArchitextSpider for Excite, or
Slurp/2.0 for HotBot) to alienate a single robot. The part following
the allow or disallow instructions indicates the directory the
spider's command should recognize. In the above example, all spiders
will be excluded from the directories titled /wpbla, /makes, and
/wpword and allowed to index all directories not mentioned in the
robots.txt and the directories titled /cgi-bin/se/t and
/upromo/temp/TEST. If you place "/" by itself after disallow, like
NASA did in its robots.txt, your entire site will NOT be indexed.
Many servers have a robots.txt file in place. You can view the public
allow and disallow orders of participating sites by entering any URL
in the location box on your browser and adding /robots.txt to the end
If you don't own your server and the administrator of your site is
unwelcome to the idea of robots.txt, you can use HTML robot META tags
to secure exclusion support. At this time, not all search engines
recognize the robot META tag as an exclusion indicator, but for the
majority of us, it is the easiest method for maintaining and
installing exclusion capabilities into desired web pages.
Just as META "description" and "keyword" tags provide descriptive
information about your site, the META "robot" tag indicates where not
to probe for that description. The basic idea is that if you include
a tag in the element of your HTML document, that document will
not be archived, and no links on the page will be followed. To
indicate to visiting robots that a document may not be indexed, place
the simple script in the of your HTML as follows.
TITLE "Your Site Title" /TITLE
META name="robots" content="noindex,nofollow"
META name="description" content="Your web site description"
META name="keywords" content="Your unique keywords"
The adherence to the Protocol for Robots Exclusion is proof positive
that we can play nicely on the Internet. Absolutely nobody is
required to participate in this particular method for exclusion, and
there are no guarantees that some robots will not visit restricted
parts of your URL's space. In the words on the official Protocol for
Robots Exclusion web site, this method "is not an official standard
backed by a standards body, or owned by any commercial organization.
It is not enforced by anybody, and there no guarantee that all current
and future robots will use it."
Therefore, you should take the proper measures to insure that
restricted information is really being protected from public display.
Failure to use proper authentication or other restriction may result
in exposure of restricted information. It is very likely that the
occurrence of paths through directories in your robots.txt file may
expose the existence of resources not otherwise linked to on the site,
which may aid people guessing for URLs. So, if someone can publicly
view server directories that are excluded, they may attempt to
infiltrate your site.
Alternatively, you can assimilate the Perl programming script for
Robot's Exclusion into each web page that you do not want to be
http://www.vperson.com/mlm/rnw.html to get the script. The Protocol
Robots Exclusion web site is at