Trawling by

David Lloyd david at
Wed Jan 5 00:49:58 CST 2005


>I was looking through my apache logs after I noticed 'msnbot'
>trying to download a large pdf over my modem link...
>... and noticed another 'bot' called
>doing some strange things.
That's normal behaviour - how else would google cache pages if they -did 
not- download them at some stage?

>It was just looking for '/robots.txt' but seemed to do so every 
>30 minutes or so.
Seems relatively normal too.

>Also, can anyone confirm whether MSN is at war with Google?
It would make business sense for MSN to be at war with anyone who is not 
owned by them; monopolies are bad for end users but good for the company 
who has the monopoly.

>MSN seems to be scanning my site, and grabbing the same stuff
>approximately 3 days after the googlebot comes by. They started
>doing this on the 18th of December.
Who knows? Maybe MSN has managed to implement the google algorithm 
successfully; or maybe they do access google itself to work out what to 
look for...all speculation really.


More information about the linuxsa mailing list