Taras, On Fri, Jul 27, 2012 at 12:36 PM, Taras <ox...@oxdef.info> wrote: > Andres, > > >>>> Hmmm, but do we actually need to do something in the error handling? >> >> >> After some thinking, I came up with a solution that should solve the >> initial issue you pointed out: "the lack of /". For now and because >> I'm not sure if this fixes all issues (there might be more than one >> source of BadStatusLine) I commited it to my branch [0] , please >> review so I can merge into trunk. To test, please move this change to >> a local trunk, the threading2 branch isn't done yet. >> >> [0] http://sourceforge.net/apps/trac/w3af/changeset/5409 > > > Great! Can confirm that it fixes issue with absent "/" :)
Perfect, could you please merge it to trunk for me? > >>> Another result is...raising BadStatusLine with incorrect path in URL is >>> only >>> one example. As I think, when we talk about web app fuzzer there is also >>> possible a huge number of another cases with strange responses from web >>> server. Currently when w3af gets such strange response (with some >>> urllib2.URLError exception) it tries to send request again N times >>> (maxRetrys). If on N attempt w3af also gets strange response it **raises >>> w3afMustStopOnUrlError** and **stops** the whole scan. Imho, it is too >>> severely because one request with e.g. 2 retries can stop the whole scan. >>> Here we need to do same thing as for unexpected errors - increment global >>> error count. We also need to make errtotal value configurable. >> >> >> Yes, that's an issue we have in the error handling that should be >> fixed, but I'm not sure if elevating the maxRetrys is the way. > > > Why? What is difference between BadStatusLine exception and others? BadStatusLine we were generating a large part of those because of a bug in our code that we fixed. Others are generated (we think) because of server errors, network errors, etc. If we can identify the exact reasons for others, then maybe we can fix them and completely avoid them :) > Also I have gotten BadStatusLine exception in Python 2.7 with **correct** > URL but with HTTP response like: > > $ curl -v 'http://example.com/goto/?nh=1&path=<ScRIPT>a=/U7fZ/%0A1' > * About to connect() to example.com port 80 (#0) > * Trying 127.0.0.1... > * connected > * Connected to example.com (127.0.0.1) port 80 (#0) >> GET /goto/?nh=1&path=<ScRIPT>a=/U7fZ/%0A1 HTTP/1.1 >> User-Agent: curl/7.26.0 >> Host: example.com >> Accept: */* Hmmm, that changes stuff, so the reason for BadStatusLine is not only the missing / . Maybe the problem with that request you're sending now is that curl is not url-encoding the < and > ? Could you confirm with a pcap?. The bad thing is that in some cases we want to send those payloads without url-encoding. > * additional stuff not fine transfer.c:1037: 0 0 > * Empty reply from server > * Connection #0 to host example.com left intact > curl: (52) Empty reply from server > * Closing connection #0 > > And we can't fix such cases with correcting URL which are generated with > w3af. If the problem is around encoding those <>, we can fix it, but we don't want to because its a feature. > So what do you think about such patch? > > > $ diff -u xUrllib.py.old xUrllib.py > --- xUrllib.py.old 2012-07-26 16:13:10.000000000 +0400 > +++ xUrllib.py 2012-07-26 16:13:25.000000000 +0400 > @@ -558,7 +558,11 @@ > traceback.format_exc()) > req._Request__original = original_url > # Then retry! > - return self._retry(req, e, cache) > + try: > + return self._retry(req, e, cache) > + except w3afMustStopOnUrlError, e: > + self._incrementGlobalErrorCount(e) > + return self._new_no_content_resp(original_url_inst, > log_it=True) So your fix simply "eliminates the w3afMustStopOnUrlError" and w3af will keep trying to scan a host even if it is down? Regards, > except KeyboardInterrupt: > # Correct control+c handling... > raise > > >> For now, I would try to continue looking (and fixing) for the requests >> that w3af performs and force those badstatusline errors. Maybe we find >> some more, we fix them and the issue dissapears. > > > -- > Taras > http://oxdef.info > GPG: C8D1F510 -- Andrés Riancho Project Leader at w3af - http://w3af.org/ Web Application Attack and Audit Framework Twitter: @w3af GPG: 0x93C344F3 ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
