Fiddler V2x beta bug and web crowler
Well, fiddler is very nice tool to monitor outgoing request from your IE or AspNet development server to any third party web.
Fiddler is good tool to analyse the request going through IE. After monitoring the requests parameters posted to any web
one can write a program to crowl that web. best way use C#.net for programming crowler.
You can download Fiddler from here http://www.fiddler2.com/Fiddler2/version.asp
Latest Fiddler beta version 2.x for .net framework has bug
After 25000 to 50000 request fiddler throws MemoryExceptions. I tested it with 1GB and 2Gb ram on windows XP
with .net Framework 2.0 with latest service pack.
If you are looking for writting web crowler using .Net technology, you can refer http://www.revenmerchantservices.com/category/HttpWebRequest.aspx
and use System.Net.HttpWebRequest or WebCleint and NameValueCollection.