Taras, On Mon, Apr 9, 2012 at 9:33 AM, Taras <ox...@oxdef.info> wrote: > Andres, >> >> ... >> >> >> * Discovery plugins x and y are enabled >> * URL 1 is set as target >> * x is fed with URL 1 as starting point >> * x is run and finds URLs 2, 3. For some reason we don't care about it >> performs HTTP GET requests to 2 and 3. >> * y is fed with URL 1 as starting point >> - If using the technique implemented in "y", it finds the URL "2" and >> "4" >> - For some reason we don't care about, it also performs HTTP GET >> requests to "2" and "4". >> - The HTTP requests are performed using the xUrllib >> - xUrllib has a cache, and because a request to "2" was already >> performed, it should take the response out of the cache; no network >> traffic is generated >> - The request for URL "4" is sent to the network >> - Plugin "y" returns the "2" and "4" knowledge to the w3afCore >> - The w3afCore should at that point say: "I already know about 2, >> thanks but I won't add duplicates", "I'll add URL 4 to my list" >> >> If the framework IS working like this, I think that the shared >> fuzzable request list wouldn't do much good. If it is not working like >> this (and I would love to get an output log to show it), it seems that >> we have a lot of work ahead of us. > > And w3afCore need to filter requests from discovery plugins on every loop in > _discover_and_bruteforce(), am I right?
It should filter things as they come out of the plugin and before adding them to the fuzzable request list, I'm unsure where in the code that is and don't have the time right now to verify it. > What I also think about is that filtering of variants like currently in > webSpider is not job of custom discovery plugin. That's a very good point. We shouldn't be filtering variants in webSpider, BUT we had to do it because we found that in cases where the spider found ONE page with 1k links that were all variants, we didn't want the spider to HTTP GET request each of those to verify if they were 404s or not. In most cases variant identification shouldn't be done at the plugin level, the webSpider was an exception. Please let me know if the discovery process is NOT working as we expect and if we have to filter stuff somewhere > >> PS: fuzzableRequestList can't be a child of Python's list, we >> shouldn't have all those items in memory >> >> Regards, >>> >>> -- >>> Taras >>> http://oxdef.info >>> >>> >>> ------------------------------------------------------------------------------ >>> For Developers, A Lot Can Happen In A Second. >>> Boundary is the first to Know...and Tell You. >>> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! >>> http://p.sf.net/sfu/Boundary-d2dvs2 >>> _______________________________________________ >>> W3af-develop mailing list >>> W3af-develop@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/w3af-develop >> >> >> >> > > > -- > Taras > http://oxdef.info -- Andrés Riancho Project Leader at w3af - http://w3af.org/ Web Application Attack and Audit Framework ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2 _______________________________________________ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
