-------- Original-Nachricht --------
> Datum: Thu, 6 May 2010 21:53:10 -0300
> Von: Andres Riancho <andres.rian...@gmail.com>
> An: Sheriff Texer <sheriff.te...@gmx.de>
> CC: w3af-develop@lists.sourceforge.net
> Betreff: Re: [W3af-develop] xUrllib found too much consecutive errors - BUG
> found?
> Sheriff,
>
> Sorry for the late reply, I've been really busy with work. Please
> read inline,
This is really no problem, I know you have much to do. Thanks for reply anyway
;)
> On Tue, Apr 27, 2010 at 10:43 AM, Sheriff Texer <sheriff.te...@gmx.de>
> wrote:
> > Hey,
> >
> > I'm using w3af for 2 weeks now, so I'm still at the beginning right now.
> There was one big problem I got nearly everytime when I wanted to scan a
> private app of mine. Even if I wanted to do only spidering (webSpider,
> spiderMan) this error occurs:
> >
> > ----------
> > **IMPORTANT** The following error was detected by w3af and couldn't be
> > resolved: The xUrllib found too much consecutive errors. The remote
> > webserver doesn't seem to be reachable anymore; please verify
> > manually
> > ----------
> >
> > After this message occured, w3af stopped scanning (or spidering) without
> any result. After so many reading and searching for the answer, that led
> to nowhere, I decided to search a little bit in the code of
> /core/data/url/xUrllib.py.
> >
> > Here I found something intresting. The source of the error-message was
> >
> > if self._consecutiveErrorCount >= 10:
> > msg = 'The xUrllib found too much consecutive errors. The remote
> webserver doesn\'t'
> > msg += ' seem to be reachable anymore; please verify manually.'
> > raise w3afMustStopException( msg )
> >
> > This code is found in the funtion _incrementGlobalErrorCount( self ),
> which is called in the urllib2.URLError-exception. The problem is, that every
> 400-response raise the urllib2.URLError-
> > exception and increment the _consecutiveErrorCount. After the 10th
> 400-response w3af will raise the w3afMustStopException.
>
> hmmm, I think you're mistaken. Not by far I must say... but
> mistaken at the end. Maybe the source code comments are misleading?
>
> # I get to this section of the code if a 400 error is returned
> # also possible when a proxy is configured and not available
> # also possible when auth credentials are wrong for the URI
> if hasattr(e, 'reason'):
> self._incrementGlobalErrorCount()
>
> The thing is the I get to the point where the comments are, when I
> find a 4xx error, the proxy is unreachable, or the server is
> unreachable. Once there, I ask if the exception "e" has a reason
> attribute. If the reason attribute is set, it means that I got there
> because of the proxy is unreachable, or the server is unreachable. The
> rest of the cases are handled here:
>
> elif hasattr(e, 'code'):
> # We usually get here when the response has codes 404,
> 403, 401, etc...
> msg = req.get_method() + ' ' + original_url +'
> returned HTTP code "'
> msg += str(e.code) + '" - id: ' + str(e.id)
Yeah, I understood that before. The mistery is, that the app is reachable
anytime: before scanning, while scanning and even after scanning. Thats why I
cant imagine why there are so many 400-responses. But I have to admit that my
app is running in a VM but like I have mentioned before, there are normally no
problems with reaching the app...
> Just to be sure... have you tested if you get to line 440 of
> xUrllib.py when a 404 occurs?
>
> > I think this happens because of the function ignore_errors( self, yes_no
> ) or better, because of the attribute _ignore_errors_conf. This attribute
> was set to 'False' by default.
>
> And it should be the default. This parameter is there for
> "advanced users" that don't care about errors, just want the framework
> to go on scanning, and take the responsability of ignoring errors.
> Common users should have those defaults.
>
> > self._ignore_errors_conf = False
> >
> > The problem is that neither this attribute is set to be 'True' nor the
> function ignore_errors( self, yes_no ) is called on any point in the code!
> But this will be necessary to ignore the
> > 400-response like its commented in the code:
>
> Here is where the "self._ignore_errors_conf" attribute is used:
>
> def _incrementGlobalErrorCount( self ):
> '''
> Increment the error count, and if we got a lot of failures...
> raise a "afMustStopException"
But I think that only 10 400-responses are maybe insufficient to raise the
"w3afMustStopException":
if self._consecutiveErrorCount >= 10:
msg = 'The xUrllib found too much consecutive errors. The remote
webserver doesn\'t'
msg += ' seem to be reachable anymore; please verify manually.'
raise w3afMustStopException( msg )
What about the case when a target is for some reasons only for a few seconds
unreachable while scanning and after that will continuing online...than the
tester (just like me) got a really ugly problem.
I understand that at the beginning of a scan w3af must test if the target is
reachable or not and of course it also can happen that the app is going offline
while scanning. But like I said, when this happens only a for a few seconds,
than only 10 400-responses are maybe not enough.
> '''
> if self._ignore_errors_conf:
> return
>
> And the function is a "setter" that's called from
> xUrllibSettings.py (if I don't recall wrongly).
>
> > def ignore_errors( self, yes_no ):
> > '''
> > Let the library know if errors should be ignored or not. Basically,
> > ignore all calls to "_incrementGlobalErrorCount" and don't raise
> the
> > w3afMustStopException.
> >
> > @parameter yes_no: True to ignore errors.
> > '''
> > self._ignore_errors_conf = yes_no
> >
> > Since I set _ignore_errors_conf manually to 'True' the error-message
> occur no more. Will this "fix" the bug or am I completly wrong?
>
> Actually, this hides the errors. They still occur and the root
> cause should be fixed.
Yeah, understand. In my situation this was enough because I noticed that the
app was reachable anytime. Or better: Has incrementGlobalErrorCount any other
advantage than counting the errors and raise the exception?
I think maybe it will be a good agreement to raise the limit of errors from 10
to 100 or something like this?
> > Many thanks for any help...
>
> Thank YOU very much for taking the time to read the source code,
> and try to find the error.
No, thank you for your energy spending in this project and the dependence about
your help ;)
>
> Regards,
> > Yeehaw
> >
> > --
> > GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
> > Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
> >
> >
> ------------------------------------------------------------------------------
> > _______________________________________________
> > W3af-develop mailing list
> > W3af-develop@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/w3af-develop
> >
>
>
>
> --
> Andrés Riancho
> Founder, Bonsai - Information Security
> http://www.bonsai-sec.com/
> http://w3af.sf.net/
--
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
------------------------------------------------------------------------------
_______________________________________________
W3af-develop mailing list
W3af-develop@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/w3af-develop
