Thanks for the input, I was thinking like you and I think I will go for XML.
Oh, and I meant database, not RDMS, but I was was curious about thoughts on
using an RDMS anyhow.
Here is what I envisioned storing:
Request
- Full Url
- Parameters & Values
Response
- Content-Type
- Content-Length
- HTTP response code
- Full Response (text/html only)
- Parse out the following (The DTD would contain these but would probably
come in later implementations)
- javascript
- links
- comments
- forms
>Are you going to store broken links?
I would suggest storing the 404, since there may be some info in the
response that could be interesting
>Are you going to store parameters for query string and forms?
Yes
>URLs that link to this URL
The links in each response are stored so this data would be in the file,
just not in this exact form.
-Tim Medin
On Wed, Aug 12, 2009 at 8:19 PM, Andres Riancho <andres.rian...@gmail.com>wrote:
> Tim,
>
> On Wed, Aug 12, 2009 at 9:16 PM, Tim Medin<timme...@gmail.com> wrote:
> > I wanted to create a "portable spider format" that would allow spider
> > results from one tool to be imported into another so each tool wouldn't
> have
> > to re-spider the site.
>
> I like the idea!
>
> > I was going to develop the first plugin for w3af and
> > then begin work on others.
>
> Cool, maybe it should be added to the discovery.importResults plugin,
> which already knows how to read CSV , Burp Suite and WebScarab logs.
>
> > While working on it I couldn't decide on a
> > fundamental architecture, xml or database? I discussed this with Kevin
> > Johnson at SANSFire 2009 and he felt that the database is the better
> choice
> > while I was slightly leaning in the other direction.
>
> I think that XML is the way to go with it. Why? If you tell a project
> leader that he'll need to add a new dependency to be able to open WXYZ
> database format files, they won't accept it and your spider format
> will die. If you use XML, all languages have an "embeded" XML library
> and that will make it possible to read your files easily.
>
> > Database
> > + Easier to query
> > + Easier for the average person to develop additional features
> > - More difficult to write - requires more checking for libraries
> > - Implementation for each RDMS
>
> Ahh, you meant database as DATABASE? When you wrote database before I
> automatically thought about sqlite. Following my previous point: if
> you tell project leaders that they have to install "mysql" together
> with their softwares, they'll laugh in your face and continue with
> whatever they are doing ;)
>
> > XML's Pros and Cons are the opposite of the database.
> >
> > Anyone have any good thoughts on the subject?
>
> Something that you failed to write, but I'm sure that you already know
> is... what do you want to store in the file? URLs is the initial
> answer, but what about:
>
> - URLs that link to this URL
> - Content-Type
> - Content-Length
> - HTTP response code
> - Are you going to store broken links?
> - Are you going to store parameters for query string and forms?
>
> > Also, anyone have any other good name ideas?
> > - Shelob - Spider from Lord of the Rings
> > - Loth - Spider God (from D&D)
> > - STFU - Spider Transcript Format - Universal
> > - ???
>
> I'm the worst when it comes to naming something!
>
> > (Sorry if this is off topic)
>
> No, not off topic at all, lets keep talking about this here. Something
> that you need to know is that the guys from NIST were collecting XML
> files from different web application scanners in order to analyze what
> parts of the WAS findings were saved there by all of them. Maybe they
> are also doing the same with crawlers? I think you should contact them
> also,
>
> Cheers,
>
> > -Tim Medin
> >
> >
> ------------------------------------------------------------------------------
> > Let Crystal Reports handle the reporting - Free Crystal Reports 2008
> 30-Day
> > trial. Simplify your report design, integration and deployment - and
> focus
> > on
> > what you do best, core application coding. Discover what's new with
> > Crystal Reports now. http://p.sf.net/sfu/bobj-july
> > _______________________________________________
> > W3af-develop mailing list
> > W3af-develop@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/w3af-develop
> >
> >
>
>
>
> --
> Andrés Riancho
> Founder, Bonsai - Information Security
> http://www.bonsai-sec.com/
> http://w3af.sf.net/
>
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
W3af-develop mailing list
W3af-develop@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/w3af-develop
