Tim, On Wed, Aug 12, 2009 at 9:16 PM, Tim Medin<timme...@gmail.com> wrote: > I wanted to create a "portable spider format" that would allow spider > results from one tool to be imported into another so each tool wouldn't have > to re-spider the site.
I like the idea! > I was going to develop the first plugin for w3af and > then begin work on others. Cool, maybe it should be added to the discovery.importResults plugin, which already knows how to read CSV , Burp Suite and WebScarab logs. > While working on it I couldn't decide on a > fundamental architecture, xml or database? I discussed this with Kevin > Johnson at SANSFire 2009 and he felt that the database is the better choice > while I was slightly leaning in the other direction. I think that XML is the way to go with it. Why? If you tell a project leader that he'll need to add a new dependency to be able to open WXYZ database format files, they won't accept it and your spider format will die. If you use XML, all languages have an "embeded" XML library and that will make it possible to read your files easily. > Database > + Easier to query > + Easier for the average person to develop additional features > - More difficult to write - requires more checking for libraries > - Implementation for each RDMS Ahh, you meant database as DATABASE? When you wrote database before I automatically thought about sqlite. Following my previous point: if you tell project leaders that they have to install "mysql" together with their softwares, they'll laugh in your face and continue with whatever they are doing ;) > XML's Pros and Cons are the opposite of the database. > > Anyone have any good thoughts on the subject? Something that you failed to write, but I'm sure that you already know is... what do you want to store in the file? URLs is the initial answer, but what about: - URLs that link to this URL - Content-Type - Content-Length - HTTP response code - Are you going to store broken links? - Are you going to store parameters for query string and forms? > Also, anyone have any other good name ideas? > - Shelob - Spider from Lord of the Rings > - Loth - Spider God (from D&D) > - STFU - Spider Transcript Format - Universal > - ??? I'm the worst when it comes to naming something! > (Sorry if this is off topic) No, not off topic at all, lets keep talking about this here. Something that you need to know is that the guys from NIST were collecting XML files from different web application scanners in order to analyze what parts of the WAS findings were saved there by all of them. Maybe they are also doing the same with crawlers? I think you should contact them also, Cheers, > -Tim Medin > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus > on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > _______________________________________________ > W3af-develop mailing list > W3af-develop@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/w3af-develop > > -- Andrés Riancho Founder, Bonsai - Information Security http://www.bonsai-sec.com/ http://w3af.sf.net/ ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
