Trawling by inktomisearch.com

David Lloyd david at bullmedia.com.au
Wed Jan 5 00:49:58 CST 2005


Paul,

>I was looking through my apache logs after I noticed 'msnbot'
>trying to download a large pdf over my modem link...
>... and noticed another 'bot' called lj1140.inktomisearch.com
>doing some strange things.
>  
>
That's normal behaviour - how else would google cache pages if they -did 
not- download them at some stage?

>It was just looking for '/robots.txt' but seemed to do so every 
>30 minutes or so.
>  
>
Seems relatively normal too.

>Also, can anyone confirm whether MSN is at war with Google?
>  
>
It would make business sense for MSN to be at war with anyone who is not 
owned by them; monopolies are bad for end users but good for the company 
who has the monopoly.

>MSN seems to be scanning my site, and grabbing the same stuff
>approximately 3 days after the googlebot comes by. They started
>doing this on the 18th of December.
>  
>
Who knows? Maybe MSN has managed to implement the google algorithm 
successfully; or maybe they do access google itself to work out what to 
look for...all speculation really.

DSL


More information about the linuxsa mailing list