KDnuggets : News : 2007 : n11 : item8 < PREVIOUS | NEXT >

Features

From: Gregory Piatetsky-Shapiro
Date: June 4, 2007
Subject: Other bots start using site local search

Following Googlebot, which was observed submitting queries to KDnuggets local search engine, other search bots are also starting to use the local search, in order to dive into the "deep web".

In the last week (May 27 - June 2), Googlebot used KDnuggets local search 386 times. Most popular queries were

  • mining (39 times)
  • market (36)
  • state (33)
  • web (32)
  • system (31)
  • share (20)
  • biology (8)
  • measure (5)
  • farm (3)
I can see why Googlebot would consider "mining" relevant to KDnuggets, but why "state" or "farm" ? Since heritrix bot from Stanford also submitted searches for 4 of the above 10 words, they clearly use a similar algorithm.

Other search bots which submitted queries to KDnuggets local search included

  • Yahoo Slurp (1 search for "biology")
  • heritrix (from Stanford), 5 searches ("market","share","mining","web").
Search bots also were able to parse the local search form and used different search "Realms" (search collections).

Bookmark using any bookmark manager! What's this?

KDnuggets : News : 2007 : n11 : item8 < PREVIOUS | NEXT >

Copyright © 2007 KDnuggets.   Subscribe to KDnuggets News!