Sometimes a site running on ASP.NET and IIS6 and implement UrlRewriting can crash when crawled by GoogleBot.
This is due to a bug in Html32TextWriter used by the Adaptive Rendering feature in ASP.NET
You can read more about the error here :
It happened to my www.gurgle.com after we implemented friendly urls for SEO.
If you’re using Visual Studio 2008 Team System Edition, and you are using Webtest to Test and LoadTest your site, there’s something you can do to prevent your site to crash when crawled by GooleBot or yahoo bot.
You can easily create a custom Browser Template, with the GoogleBot User-Agent and test your site simulating that browser.
HowTo: create a custom Browser Template for LoadTesting
You can then create a loadtest file and execute the webtest using the GoogleBot crawler