Trawling by inktomisearch.com
quail.linux at gmail.com
Wed Jan 5 04:48:48 CST 2005
On Tue, 04 Jan 2005 18:11:52 +1030, Paul Schulz <paul at mawsonlakes.org> wrote:
> Also, can anyone confirm whether MSN is at war with Google?
> MSN seems to be scanning my site, and grabbing the same stuff
> approximately 3 days after the googlebot comes by. They started
> doing this on the 18th of December.
You are lucky you only got MSNbot coming every 3 days, until i blocked
MSNbot from crawling my site via robot.txt they were harrassing my
server daily. And the stupid thing is a done a search for my web site
in there seach engine and could not find myself.
Damn MSN was just harrassing my server cause it is linux based i guess :(
And googlebot visits my server about once a week and i am indexed in
there serch engine
More information about the linuxsa