|[an error occurred while processing this directive]
by Brian D. Chmielewski
Webster's defines the prefix, meta, as meaning "more comprehensive:
transcending -- used with the name of a discipline to designate a new
but related discipline designed to deal critically with the original
one." Generically relating this definition to Internet search tools,
one can envision the metacrawler as a comprehensive source for
searching many engines at once. Unlike search engines, metacrawlers
don't crawl the web themselves to build listings. Instead, they allow
searches to be sent to several search engines all at once. The
results are then blended together onto one page. Upon investigation,
you will find that the number of search engines that an individual
metacrawler uses is, on average, between four and eight. Every
metacrawler that we experienced also used the directory, Yahoo!, as a
part of the search.
Think about metacrawlers from the perspective of the auto mechanic.
For lack of better terminology, metacrawlers have the body of a normal
automobile with the power of many different racecar engines underneath
the hood. This type of search tool is versatile and time-friendly
because it allows you to search multiple search engines
simultaneously. But you cannot submit your web site to Metacrawlers.
You can be found in a metacrawler search only by being correctly
submitted in all of the search engines.
Why would search engines allow metacrawlers to use their
Metacrawlers were originally non-profit creations of the computer
departments of a few universities. Since metacrawlers presented no
direct competition in the marketplace, search engines knowingly and
openly provided the use of their technology behind the scenes to
encourage the educational growth in this field, and to get some free
visibility for themselves. As the Web has grown, so have
metacrawlers. Once low-traffic, non-profit, unknown sites, have
become traffic-heavy, revenue-generating players in the search
industry. Accordingly, search engines have begun to enter into
licensing agreements or other business arrangements, rather than
freely provide, the technology that is underneath the metacrawler
surface. In other words, without search engines, the metacrawler
could not exist.
How does the relevancy work on metacrawlers?
Each of the metacrawlers creates its relevancy based on the
preferences of those who designed the search tool itself. Most
metacrawlers admit that they created their search tool as a response
to not getting "focused results." Metacrawlers return the most
relevant searches at all of the major search engines that it utilizes
at once rather than returning every possible instance of your keyword
search. That is why you will only see a few returns from each search
engine rather than all of the returns that you would normally
experience if you made a search at the search engine itself. Beyond
personal preferences, relevancy ranking at metacrawlers is based on
how many search engines a link was found at and how those sites ranked
it. Keywords that are located in the title and/or description will
also rank higher in relevancy.
Searching with a metacrawler allows you to take advantage of all of
the different approaches to locating pages which contain all of the
closest references to your keyword search from each service. In
effect, you are receiving the most polished results from each of the
major search engines and Yahoo!.
Visit your favorite search engine and search for
"metasearch" or "metacrawler" under the power
search feature to locate a few metacrawlers. You may find the
PowerSearch Assistant Internet floating utility helpful.
uPromote Directory Listing
business into the major search tools so that Metacrawlers will give