MSNBot Ignores Robots.txt
I think I might have accidentally started the MSNBot Boycott when I made some comments on on JWZ's blog. I posted the exclusion rules to put in your robots.txt, to keep Microsoft from crawling your site. I set it up on my site immediately, but I discovered that the MSNBot ignores the robots.txt rules, they scanned my site anyway. MS claims it's a bug. Yeah right. As usual, Microsoft thinks the rules don't apply to them.
Comments
Actually, I'm *not* JWZ. He's more famous than I am. :-)
Posted by: Jeremy Zawodny | August 5, 2003 1:35 PM
Sorry for the inadvertent misattribution, Jeremy.
Posted by: Charles | August 5, 2003 7:01 PM