Trawling by inktomisearch.com
david at bullmedia.com.au
Wed Jan 5 00:49:58 CST 2005
>I was looking through my apache logs after I noticed 'msnbot'
>trying to download a large pdf over my modem link...
>... and noticed another 'bot' called lj1140.inktomisearch.com
>doing some strange things.
That's normal behaviour - how else would google cache pages if they -did
not- download them at some stage?
>It was just looking for '/robots.txt' but seemed to do so every
>30 minutes or so.
Seems relatively normal too.
>Also, can anyone confirm whether MSN is at war with Google?
It would make business sense for MSN to be at war with anyone who is not
owned by them; monopolies are bad for end users but good for the company
who has the monopoly.
>MSN seems to be scanning my site, and grabbing the same stuff
>approximately 3 days after the googlebot comes by. They started
>doing this on the 18th of December.
Who knows? Maybe MSN has managed to implement the google algorithm
successfully; or maybe they do access google itself to work out what to
look for...all speculation really.
More information about the linuxsa