How to prevent crash in your site when crawled by GoogleBot

Sometimes a site running on ASP.NET and IIS6 and implement UrlRewriting can crash when crawled by GoogleBot.

This is due to a bug in Html32TextWriter used by the Adaptive Rendering feature in ASP.NET

You can read more about the error here :

It happened to my after we implemented friendly urls for SEO.

If you’re using Visual Studio 2008 Team System Edition, and you are using Webtest to Test and LoadTest your site, there’s something you can do to prevent your site to crash when crawled by GooleBot or yahoo bot.
You can easily create a custom Browser Template, with the GoogleBot User-Agent and test your site simulating that browser.

HowTo: create a custom Browser Template for LoadTesting

You can then create a loadtest file and execute the webtest using the GoogleBot crawler


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s