Re: [Bug-wget] Bug Track
Am Samstag, 18. Oktober 2014, 22:40:14 schrieb Tushar: Hi, I am a student who would like to contribute to GNU Project. I'm very passionate about GNU organization and would like to dedicate some time everyday for GNU. It was mentioned that I have to send an email to this address before diving into Bugs. I would look into the bugs and hopefully, I would be able to help. I don't have much programming experience and I know things will be difficult for me but it is my passion for GNU and it's philosophy that drives me. Hi Tushar, welcome to Wget. We appreciate any help you can give ! You don't have to be a (experienced) programmer to help out. I guess, you are already registered at Savannah (https://savannah.gnu.org) !? We are going to clean the bug list and if you could help, that would be very cool. Just dig into the bug list and - as a first step - try to reproduce bugs with the latest development version of Wget (clone from the git repo and compile). If you can't reproduce it, either create a comment and request more info (whatever is needed, e.g. debug out or Wget version) and/or sent me an email (maybe PM), so I can view and close the bug. You can also write me an email whenever you have any questions. e.g. if you are going to write a new test and you are unsure - just ask me (and/or the list if the question is of egenral interest). Please send patches directly to the list for reviewing / discussion / integration. Thank you. We thank you ! Tim signature.asc Description: This is a digitally signed message part.
[Bug-wget] Send Content-Length with POST 0 length body
Hi I was looking through the list archives to find something that useful I could contribute to wget and get familiar with the code, I have attached a patch for the following. Darshit Shah darnir at gmail.com wrote on 2014-09-05 07:31:34 GMT The Content-Length Header is expected by the server in many requests, most prominently in a POST Request. However, currently if a POST request is sent without any body, Wget does not send any Content-Length headers. Some servers seem to dislike this behaviour and respond with a 411 Length Required message. Patch Wget to always send a Content-Length header when the Request Type is POST. Hopefully I can find something less trivial to add or fix in the future, any suggestions, direction or feedback, please let me know. Thanks, Matt diff --git a/src/http.c b/src/http.c index 4b99c17..e020682 100644 --- a/src/http.c +++ b/src/http.c @@ -1875,6 +1875,8 @@ gethttp (struct url *u, struct http_stat *hs, int *dt, struct url *proxy, xstrdup (number_to_static_string (body_data_size)), rel_value); } + else if (strcasecmp (opt.method, post) == 0) +request_set_header (req, Content-Length, 0, rel_none); } retry_with_auth:
Re: [Bug-wget] [PATCH] V2 removed 'auto' SSLv3 also from OpenSSL code
Tim Rühsen tim.rueh...@gmx.de writes: patch V2 - removed SSLv3 from --secure-protocol=auto|pfs (GnuTLS code) - removed SSLv3 from --secure-protocol=auto (OpenSSL code) - amended the docs I am not an OpenSSL expert... please feel free to suggest improvements. same here, but it looks like a good idea to disable SSLv3, so feel free to push it. Regards, Giuseppe
Re: [Bug-wget] [PATCH] Small fix for limited number of strings (and potential garbage value) in arguments to concat_strings
Pär Karlsson feino...@gmail.com writes:
Hi, I fould a potential gotcha when playing with clang's code analysis tool.
The concat_strings function silently stopped counting string lengths when
given more than 5 arguments. clang warned about potential garbage values in
the saved_lengths array, so I redid it with this approach.
All tests working ok with this patch.
thanks for your contribution. I've just few comments:
commit 2d855670e0e1fbe578506b376cdd40b0e465d3ef
Author: Pär Karlsson feino...@gmail.com
Date: Thu Oct 16 21:41:36 2014 +0200
Updated ChangeLog
we usually update the changelog in the same commit that made the change,
so please squash these two commits into one.
Also, it doesn't apply on current git master, as it seems to be based on
a old version of git from the ChangeLog file context, could you rebase
onto the master branch?
diff --git a/src/utils.c b/src/utils.c
index 78c282e..93c9ddc 100644
--- a/src/utils.c
+++ b/src/utils.c
@@ -356,7 +356,8 @@ char *
concat_strings (const char *str0, ...)
{
va_list args;
- int saved_lengths[5]; /* inspired by Apache's apr_pstrcat */
+ size_t psize = sizeof(int);
please leave a space between sizeof and '(' as mandated by the GNU
coding standards.
+ int *saved_lengths = xmalloc (psize);
char *ret, *p;
const char *next_str;
@@ -370,8 +371,8 @@ concat_strings (const char *str0, ...)
for (next_str = str0; next_str != NULL; next_str = va_arg (args, char *))
{
int len = strlen (next_str);
- if (argcount countof (saved_lengths))
-saved_lengths[argcount++] = len;
+ saved_lengths[argcount++] = len;
+ xrealloc(saved_lengths, psize * argcount);
same here.
total_length += len;
}
va_end (args);
@@ -393,7 +394,7 @@ concat_strings (const char *str0, ...)
}
va_end (args);
*p = '\0';
-
+ free(saved_lengths);
and here.
Regards,
Giuseppe
Re: [Bug-wget] Send Content-Length with POST 0 length body
Am Sonntag, 19. Oktober 2014, 11:27:51 schrieb Matthew Atkinson: Hi I was looking through the list archives to find something that useful I could contribute to wget and get familiar with the code, I have attached a patch for the following. Darshit Shah darnir at gmail.com wrote on 2014-09-05 07:31:34 GMT The Content-Length Header is expected by the server in many requests, most prominently in a POST Request. However, currently if a POST request is sent without any body, Wget does not send any Content-Length headers. Some servers seem to dislike this behaviour and respond with a 411 Length Required message. Patch Wget to always send a Content-Length header when the Request Type is POST. Hopefully I can find something less trivial to add or fix in the future, any suggestions, direction or feedback, please let me know. Very good ! Thanks for your work. There two little things - well, just kinda organizational work: 1. Please also extend src/ChangeLog and include it in your patch 2. The maintainers will ask you for a 'git format-patch' output If you never worked with that: - git commit your change(s) - attach the files generated with 'git format-patch -1' Thank you Tim signature.asc Description: This is a digitally signed message part.
Re: [Bug-wget] Transparent proxy URL ariation on -E -k options ?
Gabriel Somlo wrote: If I try to add --convert-links into the mix, the referencing link does get rewritten, but ends up looking like ../site.com/article.cgi?25.html which is designed for offline viewing via file://, and is unsuitable for actually hosting both the referencing and referenced sites as virtual servers in a web server within the sandbox. Are you using --span-hosts ? Otherwise wget won't be crawling pages to a different host and thus won't produce a relative url down to the hostname. I think that not using --span-hosts will suit your use case. If not, assuming I can come up with a patch, would there be any interest in upstreaming this type of additional functionality ? However, you may provide a patch for making links relative to the hostname in such case (you would need to add a parameter for --convert-links to enable that alternative conversion). Bug notice: just listing the domains to span on --domains doesn't seem to work Deciding whether to enqueue http://www.example.org/script.js;. This is not the same hostname as the parent's (www.example.org and www.example.com). Decided NOT to load it.
Re: [Bug-wget] please remove SSLv3 from being used until explicitly specified
Tim Rühsen wrote: Hi Ángel, thanks for your testing. I would like to reproduce it - can you tell me what you did exactly ? I used a simple server that printed the TLS Client Hello and closed the connection. Browsers automatically retried with lower SSL versions. wget aborted with an «Unable to establish SSL connection.» message The original paper talks about 'client renegotiation dance'. What about renegotiation at protocol level ? Isn't it possible that a TLS connection goes down to SSLv3 intransparent to the client/server code ? AFAIK no. That is protected by the HMAC. The problem is the version downgrading on a network error, which can be inserted by a MiTM (and without TLS_FALLBACK_SCSV the server won't be able to that the client downgraded its version thinking the server didn't support a greater one). I am not that deep into the TLS/SSL libraries to answer that question myself right now. The paper talks about 'proper protocol version negotiation' - that seems to need some clarification. That's the server replying with a lower protocol version in the same connection. The downgrade was a hack for broken servers not properly supporting SSL. And we are paying it now.
Re: [Bug-wget] wget
Micah Cowan wrote: Hi! It should be noted that standard error is rather misleadingly named. By long-standing convention, it is generally where all logging and informational messages are meant to go. Standard output is (supposed to be) reserved for actual program output, suitable for potential piping to the stdin of another program. HTH, -mjc Note that if you're not interested in that verbose output, you can use -q (--quiet) parameter.
Re: [Bug-wget] [PATCH] V2 removed 'auto' SSLv3 also from OpenSSL code
Am Sonntag, 19. Oktober 2014, 16:07:35 schrieb Giuseppe Scrivano: Tim Rühsen tim.rueh...@gmx.de writes: patch V2 - removed SSLv3 from --secure-protocol=auto|pfs (GnuTLS code) - removed SSLv3 from --secure-protocol=auto (OpenSSL code) - amended the docs I am not an OpenSSL expert... please feel free to suggest improvements. same here, but it looks like a good idea to disable SSLv3, so feel free to push it. Regards, Giuseppe It just has been pushed. Tim
Re: [Bug-wget] please remove SSLv3 from being used until explicitly specified
Am Sonntag, 19. Oktober 2014, 21:11:01 schrieb Ángel González: Tim Rühsen wrote: Hi Ángel, thanks for your testing. I would like to reproduce it - can you tell me what you did exactly ? I used a simple server that printed the TLS Client Hello and closed the connection. Browsers automatically retried with lower SSL versions. wget aborted with an «Unable to establish SSL connection.» message The original paper talks about 'client renegotiation dance'. What about renegotiation at protocol level ? Isn't it possible that a TLS connection goes down to SSLv3 intransparent to the client/server code ? AFAIK no. That is protected by the HMAC. The problem is the version downgrading on a network error, which can be inserted by a MiTM (and without TLS_FALLBACK_SCSV the server won't be able to that the client downgraded its version thinking the server didn't support a greater one). I am not that deep into the TLS/SSL libraries to answer that question myself right now. The paper talks about 'proper protocol version negotiation' - that seems to need some clarification. That's the server replying with a lower protocol version in the same connection. The downgrade was a hack for broken servers not properly supporting SSL. And we are paying it now. Thank you ! Tim
Re: [Bug-wget] [PATCH] Add valgrind testing support via ./configure
On 10/09, Tim Rühsen wrote: Hence, hard coding the command actually reduces the amount of work a user needs to do in order to run the tests under valgrind. My suggestion is that we allow the configure option, but hard code the valgrind command into the test suites themselves, and not leave them environment variables. What about removing the configure option and if VALGRIND_TESTS is undefined or empty or 0: normal testing if VALGRIND_TESTS is 1: valgrind testing with hard-coded options else: testing with command given in VALGRIND_TESTS The above described workflow should still work, right ? And we could do the complete test suite with VALGRIND_TESTS=1 make check What do you think ? --- end quoted text --- I actually like this idea. case $VALGRIND_TESTS: , 0) Normal tests;; 1) Hard coded valgrind string;; *) Execute the provided command;; Could you please make the relevant changes to atleast the Perl based tests and to configure.ac? I'm currently traveling, but I'll fix the Python tests ASAP and send in a patch which will work well with the aforementioned cases. -- Thanking You, Darshit Shah pgpNQKuiMcpVE.pgp Description: PGP signature
[Bug-wget] [Bug-Wget] Summary of Pending Patches
I was going through my local repository and noticed that the following patches are currently pending. I wanted a final round of reviews on them so that we may merge them into mainline. I've tried to explain the context behind each patch below and have also resent the version of each patch I had locally to the relevant threads. So, I'm sorry for spamming your mailboxes. 1. Fix make distcheck for Python tests Tim discovered that the Python based test suite wasn't passing make distcheck due to a variety of reasons. I've fixed most of all the issues with the code. Some hacks still need to be eliminated. But in general, everything works right now. 2. Minor optimizations of Python tests While working with the Python test suite to fix the above mentioned issues, I came across and fixed some inefficiencies / cruft code in the test suite. Those have been eliminated in this patch. 3. Fixed IRI misbehaviors Wget wasn't functioning correctly with non C locales. Tim has fixed the issue but the patch hasn't yet been pushed. 4. Add ./configure valgrind support to test suites Tim added support for finding and fixing memory leaks in Wget by running the entire Perl based test suite under Valgrind. This feature had already been implemented in the python tests earlier. There's still rough edges to this patch that need fixing, so it may not be ready for merging right now. Current Issues == Test-proxied-https-auth.px fails more than 9/10 times when executed through make check. However when executed directly as ./Test-proxied-https-auth.px, it works perfectly. It also works perfectly when executed through a make check -jn command. So, it seems to be some sort of a race condition that requires to be identified. Side Note = During the various discussions across multiple threads for these patches, we discovered various ways of executing make check, make distcheck, and each test which had to be checked manually to ensure that everything was working fine. For example, Tim made a very nice list during one of the discussions: - make check - make check -jn - cd tests; ./Test-... - make distcheck - make distcheck -jn - TESTS_ENVIRONMENT=LC_ALL=whatever make check [-jn] - cd tests; TESTS_ENVIRONMENT=LC_ALL=whatever ./Test-... (i used to test LC_ALL=tr_TR.utf8 for TESTS_ENVIRONMENT) Maybe we could automate all of these under a single make target? Or else, I intend to set up a Travis CI build every week to run these tests. What is everyone's opinion on this? -- Thanking You, Darshit Shah pgpcxSpyZbIqn.pgp Description: PGP signature
Re: [Bug-wget] Send Content-Length with POST 0 length body
On 19/10/14 19:34, Tim Rühsen wrote: There two little things - well, just kinda organizational work: 1. Please also extend src/ChangeLog and include it in your patch 2. The maintainers will ask you for a 'git format-patch' output If you never worked with that: - git commit your change(s) - attach the files generated with 'git format-patch -1' Attached Thanks, Matt From af9326b1665e0e1d3b28ed0b66c6c99d07a88172 Mon Sep 17 00:00:00 2001 From: Matthew Atkinson mutley...@ntlworld.com Date: Sun, 19 Oct 2014 21:31:36 +0100 Subject: [PATCH] Always send Content-Length if method is post --- src/ChangeLog | 5 + src/http.c| 2 ++ 2 files changed, 7 insertions(+) diff --git a/src/ChangeLog b/src/ChangeLog index 1c4e2d5..447179e 100644 --- a/src/ChangeLog +++ b/src/ChangeLog @@ -1,3 +1,8 @@ +2014-10-19 Matthew Atkinson mutley...@ntlworld.com + + * http.c (gethttp): Always send Content-Length header when method is post, + even with no post body, as some servers will reject the request otherwise + 2014-05-03 Tim Ruehsen tim.rueh...@gmx.de * retr.c (retrieve_url): fixed memory leak diff --git a/src/http.c b/src/http.c index 4b99c17..e020682 100644 --- a/src/http.c +++ b/src/http.c @@ -1875,6 +1875,8 @@ gethttp (struct url *u, struct http_stat *hs, int *dt, struct url *proxy, xstrdup (number_to_static_string (body_data_size)), rel_value); } + else if (strcasecmp (opt.method, post) == 0) +request_set_header (req, Content-Length, 0, rel_none); } retry_with_auth: -- 1.9.1
