How to prevent crash in your site when crawled by GoogleBot

Sometimes a site running on ASP.NET and IIS6 and implement UrlRewriting can crash when crawled by GoogleBot.

This is due to a bug in Html32TextWriter used by the Adaptive Rendering feature in ASP.NET

You can read more about the error here :
http://todotnet.com/archive/0001/01/01/7472.aspx
http://aspnet.4guysfromrolla.com/demos/printPage.aspx?path=/articles/050504-1.aspx

It happened to my www.gurgle.com after we implemented friendly urls for SEO.

If you’re using Visual Studio 2008 Team System Edition, and you are using Webtest to Test and LoadTest your site, there’s something you can do to prevent your site to crash when crawled by GooleBot or yahoo bot.
You can easily create a custom Browser Template, with the GoogleBot User-Agent and test your site simulating that browser.

HowTo: create a custom Browser Template for LoadTesting
http://blogs.msdn.com/slumley/pages/adding-an-ie7-browser-template-for-use-by-web-tests.aspx

You can then create a loadtest file and execute the webtest using the GoogleBot crawler

image

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s