OK thanks a lot for your response. I'll experiment with extended_urllib.py and also try to fix the duplicates in the output.
Thanks On Wed, Aug 6, 2014 at 12:15 AM, Andres Riancho <andres.rian...@gmail.com> wrote: > Sergio, > > On Tue, Aug 5, 2014 at 5:42 PM, Sergio A <foobarm...@gmail.com> wrote: >> Hi guys, >> >> Yesterday, while "playing" with w3af I saw something (detailed below) >> with the "allowed methods" plugin related to checking if the the http >> CONNECT method is available in a server or not and I'd like to know if >> you think it could be a bug or not. >> >> In the case you think it is a bug, I'd be very happy if I can fix it >> and, if possible, would like some advice or guidelines on what next >> steps you think I should do for implement the fix. >> The problem is that I'm not a python guru like you at all, currently I >> just know some python for my day to day stuff. >> And if it is not a bug, I'd like as well to contribute. So guidelines, >> advice, etc on how to do it is welcome :) >> >> Here is what I did: >> >> 1. git pull the last w3af version. >> >> 2. setup an apache server as a forward proxy >> >> Setup an apache web server that has enabled >> the CONNECT method for proxying clients (forward proxy not reverse >> proxy) so them can navigate setting the apache box as their proxy (the >> only client I´m testing now is on the same host hence Allow from >> 127.0.0.1). >> This is the relevant apache config I have (apart of the enabled >> mod_proxy modules, etc): >> >> <VirtualHost *:80> >> ProxyRequests On >> ProxyVia On >> AllowCONNECT 80 443 563 >> </VirtualHost> >> >> <Proxy *> >> Order deny,allow >> Deny from all >> Allow from 127.0.0.1 >> </Proxy> >> >> 3. Manually check CONNECT method is working >> >> (look at the format on the CONNECT line): >> >> bla@ubuntu:~$ telnet localhost 80 >> Trying 127.0.0.1... >> Connected to localhost. >> Escape character is '^]'. >> CONNECT google.com:80 HTTP/1.1 >> HOST: google.com >> >> HTTP/1.0 200 Connection Established <--- it worked >> Proxy-agent: Apache/2.4.7 (Ubuntu) >> >> blablablabla >> >> HTTP/1.0 400 Bad Request <---- this is normal because blablabla is a bad >> request >> Content-Type: text/html; charset=UTF-8 >> Content-Length: 1419 >> Date: Mon, 04 Aug 2014 18:13:32 GMT >> Server: GFE/2.0 >> <snip> > > Ok, so far so good :) I understood the setup. > >> 4. Setup a w3af profile for checking CONNECT method >> >> I setup a profile enabling only "allowed methods" plugin >> and this is what I see as the output of that plugin: >> >> """ The URL "http://localhost/"; has the following enabled HTTP methods: >> CONNECT, GET, GET, HEAD, HEAD, OPTIONS, OPTIONS, POST, POST. This >> information was found in the requests with ids 32, 39, 47, 52, 55 and >> 71. """ > > Strange to see that there are duplicated methods! GET/GET, HEAD/HEAD, > etc. ugly at least. > >> Note that, apart of all the methods but CONNECT been duplicated, what >> I see when going into request/response navigator for the CONNECT >> request (which is the one with id 55) is (look on the CONNECT line format): >> >> Request: >> CONNECT http://localhost/ HTTP/1.1 >> Host: localhost >> Accept-encoding: gzip, deflate >> Accept: */* >> User-agent: w3af.org >> >> Response: >> HTTP/1.1 400 Bad Request >> date: Mon, 04 Aug 2014 16:57:55 GMT >> content-length: 300 >> content-type: text/html; charset=iso-8859-1 >> connection: close >> server: Apache/2.4.7 (Ubuntu) >> >> But, if I run again the same w3af profile and look with wireshark what >> I see "on the wire" (look here as well in the CONNECT line) is: >> >> Request: >> CONNECT / HTTP/1.1 >> Host: localhost >> Accept-encoding: gzip, deflate >> Accept: */* >> User-agent: w3af.org > > Ah, yes, that's a "bug" in the representation of the HTTP requests. It > is "known" (and ugly) that w3af sends: > > CONNECT / HTTP/1.1 > Host: localhost > Accept-encoding: gzip, deflate > Accept: */* > User-agent: w3af.org > > And this is recorded in the log/shown in the GUI: > > CONNECT http://localhost/ HTTP/1.1 > Host: localhost > Accept-encoding: gzip, deflate > Accept: */* > User-agent: w3af.org > > This was done to avoid adding host/port/protocol fields in the GUI and > text files. Does that make sense? Not sure. But now we just show: > > CONNECT http://localhost/ HTTP/1.1 > > And all the information is there. In an enhanced version we could have > a different GUI design with more widgets showing: > > Protocol: http > Port: 80 > Host: localhost > Request: > CONNECT / HTTP/1.1 > > That would be the ideal case. > > For your tests with the CONNECT method this is specially confusing. > >> Response: >> HTTP/1.1 400 Bad Request >> Date: Mon, 04 Aug 2014 17:06:57 GMT >> Server: Apache/2.4.7 (Ubuntu) >> Content-Length: 300 >> Connection: close >> Content-Type: text/html; charset=iso-8859-1 >> >> So in summary: >> >> (A) Manual request (which works): >> CONNECT google.com:80 HTTP/1.1 >> >> (B) w3af Request/Response navigator reported request: >> CONNECT http://localhost/ HTTP/1.1 >> >> (C) w3af on the wire request: >> CONNECT / HTTP/1.1 >> >> I think that according to the RFC it looks like a valid request should >> have just a hostname, a colon and a port, like (A) and not like (B) >> neither (C): >> http://tools.ietf.org/html/rfc7231#section-4.3.6 >> >> Do you think this could be a bug or issue on how w3af generates a >> CONNECT request ? > > * There is room for improvement in the way we show the request in the > GUI and store it in the log files. > > * In the allowed_methods plugin [0] we don't treat CONNECT in any > special way (it has a different format from the rest of the methods > but the plugin doesn't "know" about that) > > [0] > https://github.com/andresriancho/w3af/blob/master/w3af/plugins/infrastructure/allowed_methods.py > >> Also, how about the duplicates in the plugin output (I mean method >> names appearing here twice on all but CONNECT): >> "The URL "http://localhost/"; has the following enabled HTTP methods: >> CONNECT, GET, GET, HEAD, HEAD, OPTIONS, OPTIONS, POST, POST. This >> information was found in the requests with ids 32, 39, 47, 52, 55 and >> 71." > > That's a bug > >> 5. Digging a bit >> >> I went through the code trying to read it, and I think the connect >> request is generated in: >> w3af.core.data.url.extended_urllib.AnyMethod > > The request is sent here [1] and then it passed through the AnyMethod > (nice find there!). > > [1] > https://github.com/andresriancho/w3af/blob/master/w3af/plugins/infrastructure/allowed_methods.py#L174 > >> But I'm not sure on where you think it could be a good place to fix it >> and how (of course in the case you thinking there´s a bug). > > The best place to fix this might be the allowed_methods.py plugin + > extended_urllib.py, but it might be *really* difficult to fix since we > don't have support for sending "raw" text, and: > > CONNECT google.com:80 HTTP/1.1 > HOST: google.com > > Might be "difficult" to send the "google.com:80". My advise is that > you should experiment with extended_urllib.py and wireshark to see if > you can make it send what you need, and then (if you got it) write a > new method in allowed_methods.py. This can help get you started with > your tests: > > pedro@laptop:~/pch/w3af$ python > Python 2.7.3 (default, Feb 27 2014, 19:58:35) > [GCC 4.6.3] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> from w3af.core.data.url.extended_urllib import ExtendedUrllib >>>> from w3af.core.data.parsers.url import URL >>>> from w3af.core.data.dc.headers import Headers >>>> xu = ExtendedUrllib() >>>> xu.CONNECT(URL('http://google.com:80/'), headers=Headers([('Host', >>>> 'www.google.com')])) > <HTTPResponse | 400 | http://google.com/ | id:1> >>>> > > Regards, > >> Regards >> >> ------------------------------------------------------------------------------ >> Infragistics Professional >> Build stunning WinForms apps today! >> Reboot your WinForms applications with our WinForms controls. >> Build a bridge from your legacy apps to the future. >> http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk >> _______________________________________________ >> W3af-develop mailing list >> W3af-develop@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/w3af-develop > > > > -- > Andrés Riancho > Project Leader at w3af - http://w3af.org/ > Web Application Attack and Audit Framework > Twitter: @w3af > GPG: 0x93C344F3 ------------------------------------------------------------------------------ Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk _______________________________________________ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
