Use an intercepting proxy to crawl your site. I personally recommend the Zed Attack Proxy (which is a fork of the famous paros proxy), WebScarab, or the Burp Suite.
Or you could use Harvestman (http://code.google.com/p/harvestman-crawler/) but it takes a bit to set up. Just my $.02 Ryan ----- Original Message ----- From: "Michael Chesmore [DAS]" <[email protected]> To: "PaulDotCom Security Weekly Mailing List" <[email protected]>, "PaulDotCom Security Weekly Mailing List" <[email protected]> Sent: Friday, April 8, 2011 9:39:21 AM GMT -05:00 US/Canada Eastern Subject: Re: [Pauldotcom] Web App Crawlers I think Hostmap is still available.. We used it in the past but I am not sure how far it will drill down...maybe there is a switch to tell it that ...check out the man page for it. Mike From: [email protected] [mailto:[email protected]] On Behalf Of Williams, Marn PENC:EX Sent: Thursday, April 07, 2011 3:52 PM To: 'PaulDotCom Security Weekly Mailing List' Subject: Re: [Pauldotcom] Web App Crawlers Try Nmap. Search for info on http://seclists.org ________________________________ From: [email protected] [mailto:[email protected]] On Behalf Of Michael Lubinski Sent: April 7, 2011 9:55 AM To: PaulDotCom Security Weekly Mailing List Subject: [Pauldotcom] Web App Crawlers I am trying to find all of the web apps currently hosted in an organization. What is a good crawler I could use? This is from the defensive side of things, its a network that I have full control over. _______________________________________________ Pauldotcom mailing list [email protected] http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom Main Web Site: http://pauldotcom.com _______________________________________________ Pauldotcom mailing list [email protected] http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom Main Web Site: http://pauldotcom.com
