Re: [Bug-wget] Shouldn't wget strip leading spaces from a URL?
Dale R. Worley wrote: L A Walsh writes: if wget gets leading spaces in a URL, it complains: " http://www.kernel.org/pub/linux/utils/util-linux/v2.30: Scheme missing." Isn't it required for a web client to strip leading spaces from URLs? Strictly speaking, no, --- You might want to read this web requirements doc: https://www.w3.org/TR/2014/REC-html5-20141028/infrastructure.html#strip-leading-and-trailing-whitespace Especially this sentence: When a user agent is to strip leading and trailing whitespace from a string, the user agent must remove all space characters that are at the start or end of the string. As part of URL parsing, a user-agent(like wget) is required to strip leading and trailing whitespace. Go ahead, make me cite references... Nyaa! :-)
Re: [Bug-wget] Shouldn't wget strip leading spaces from a URL?
L A Walsh writes: > if wget gets leading spaces in a URL, it complains: > " http://www.kernel.org/pub/linux/utils/util-linux/v2.30: Scheme > missing." > > Isn't it required for a web client to strip leading spaces from > URLs? Strictly speaking, no, because a URL cannot contain spaces, leading or otherwise. Generally, Web browsers strip leading and trailing spaces to make it easier to cut-and-paste links into their location bars, but that is outside the formality of URLs. In addition, some browsers will automatically %-escape any character in a purported URL that isn't allowed in URLs, which makes it easier for people to circulated purported URLs that contain invalid characters. But it seems to me that it is better-defined and safer -- and more in the Un*x tradition -- to have wget simply give an error if the argument string isn't a valid URL. Dale
Re: [Bug-wget] [GSoC Update] Week 1
On Tue, 06 Jun 2017 02:37:54 +0700 Tim Rühsen wrote: > I guess Christian meant to skip the appropriate tests if MHD is not > installed/ > available. We should definitely get rid of our server code in libwget and > focus on client functionality. Kick it out once MHD code works :-) Understood. > But of course we want to run as many tests as possible with MinGW as well... > maybe you provide a script (e.g. in contrib/) that downloads/builds/installs > MHD. Add that script to our CI runner YAML file(s) and we can test on MinGW > as > well. Let me know which packages you need for the docker images. What I've done so far with .gitlab-ci.yml is add the following lines: MinGW64: + before_script: + - dnf -y install wget + - wget http://ftp.gnu.org/gnu/libmicrohttpd/libmicrohttpd-0.9.55.tar.gz + - tar zxf libmicrohttpd-0.9.55.tar.gz && cd libmicrohttpd-0.9.55/ + - mingw64-configure --prefix=/usr/x86_64-w64-mingw32/sys-root/mingw/ + - mingw64-make -j$(nproc) + - mingw64-make -j$(nproc) LOG_COMPILER=wine install + - cd - Should I move it to contrib/, or anything else? Because there is no such thing like mingw-libmicrohttpd available on the docker images. Regards, Didik Setiawan
[Bug-wget] Shouldn't wget strip leading spaces from a URL?
if wget gets leading spaces in a URL, it complains: " http://www.kernel.org/pub/linux/utils/util-linux/v2.30: Scheme missing." Isn't it required for a web client to strip leading spaces from URLs?
Re: [Bug-wget] HTTPS Python tests fail if localhost resolves to ::1
Hi Tomas,
On Freitag, 2. Juni 2017 11:38:42 CEST Tomas Hozza wrote:
> Hi.
>
> In Fedora 26+ the /etc/hosts lists "localhost" as a domain for both
> 127.0.0.1 and ::1. This makes wget's testsuite to fail during build.
>
> Failing tests:
> Test--https.py
> Test-pinnedpubkey-der-https.py
> Test-pinnedpubkey-hash-https.py
> Test-pinnedpubkey-pem-https.py
>
>
> [root@b28dfb71db2e sources]# cat testenv/Test--https.log
> Python runtime initialized with LC_CTYPE=C (a locale with default ASCII
> encoding), which may cause Unicode compatibility problems. Using C.UTF-8,
> C.utf8, or UTF-8 (if available) as alternative Unicode-compatible locales
> is recommended. Setting --no-config (noconfig) to 1
> Setting --ca-certificate (cacertificate) to
> /sources/testenv/certs/ca-cert.pem DEBUG output created by Wget
> 1.19.1.68-5d4ad on linux-gnu.
>
> Reading HSTS entries from /sources/testenv/Test--https.py-test/.wget-hsts
> URI encoding = 'ANSI_X3.4-1968'
> converted 'https://127.0.0.1:4/File1' (ANSI_X3.4-1968) ->
> 'https://127.0.0.1:4/File1' (UTF-8) Converted file name 'File1' (UTF-8)
> -> 'File1' (ANSI_X3.4-1968)
> --2017-06-02 09:28:34-- https://127.0.0.1:4/File1
> Loaded CA certificate '/sources/testenv/certs/ca-cert.pem'
> Certificates loaded: 1
> Connecting to 127.0.0.1:4... connected.
> Created socket 3.
> Releasing 0x024ee050 (new refcount 0).
> Deleting unused 0x024ee050.
> The certificate's owner does not match hostname '127.0.0.1'
Are you near latest git ?
testenv/certs/server-cert.pem should be made for '127.0.0.1' and 'localhost'
(see README) . There is also a recipe for certtool to (re-)generate server-
cert.pem.
Check the cert with 'certtool -i --infile server-cert.pem' (see Subject
Alternative Name).
> URI encoding = 'ANSI_X3.4-1968'
> converted 'https://127.0.0.1:4/File2' (ANSI_X3.4-1968) ->
> 'https://127.0.0.1:4/File2' (UTF-8) Converted file name 'File2' (UTF-8)
> -> 'File2' (ANSI_X3.4-1968)
> --2017-06-02 09:28:34-- https://127.0.0.1:4/File2
> Connecting to 127.0.0.1:4... connected.
> Created socket 3.
> Releasing 0x0278f1d0 (new refcount 0).
> Deleting unused 0x0278f1d0.
> The certificate's owner does not match hostname '127.0.0.1'
> Running Test Test--https.py
> /sources/src/wget --debug --no-config
> --ca-certificate=/sources/testenv/certs/ca-cert.pem
> https://127.0.0.1:4/File1 https://127.0.0.1:4/File2
> ['/sources/src/wget', '--debug', '--no-config',
> '--ca-certificate=/sources/testenv/certs/ca-cert.pem',
> 'https://127.0.0.1:4/File1', 'https://127.0.0.1:4/File2'] Error:
> Expected file File1 not found..
> Traceback (most recent call last):
> File "./Test--https.py", line 53, in
> protocols=Servers
> File "/sources/testenv/test/http_test.py", line 41, in begin
> self.do_test()
> File "/sources/testenv/test/base_test.py", line 187, in do_test
> self.post_hook_call()
> File "/sources/testenv/test/base_test.py", line 206, in post_hook_call
> self.hook_call(self.post_configs, 'Post Test Function')
> File "/sources/testenv/test/base_test.py", line 196, in hook_call
> conf.find_conf(conf_name)(conf_arg)(self)
> File "/sources/testenv/conf/expected_files.py", line 54, in __call__
> raise TestFailed('Expected file %s not found.' % file.name)
> exc.test_failed.TestFailed: Expected file File1 not found.
> FAIL Test--https.py (exit status: 1)
>
> I didn't have time to investigate this thoroughly yet, but I thought I'll
> let you know in case the issue will be obvious to anyone. I suspect that
> there will be a mismatch between the address on which the HTTPS server runs
> and the data in the certificate it uses.
>
> Regards,
> Tomas
With Best Regards, Tim
signature.asc
Description: This is a digitally signed message part.
Re: [Bug-wget] [GSoC Update] Week 1
Hi Didik, thanks for the summary of week #1 ! On Dienstag, 6. Juni 2017 00:40:56 CEST Didik Setiawan wrote: > On Mon, 05 Jun 2017 23:51:00 +0700 Christian Grothoff wrote: > > Maybe I missed something, but why should the MHD dependency become > > _mandatory_? Why not conditionally-compile and run the tests only if MHD > > is present? > > Maybe I just misinterpreting this [1]? What Darshit said about removing old > server code from the codebase will make Wget2 more stable since we will not > be susceptible to bugs in the HTTP Server code. > So, if it is better options, I will amend my current change, make MHD > optional, give fallback to the old server code if MHD not available. I guess Christian meant to skip the appropriate tests if MHD is not installed/ available. We should definitely get rid of our server code in libwget and focus on client functionality. Kick it out once MHD code works :-) But of course we want to run as many tests as possible with MinGW as well... maybe you provide a script (e.g. in contrib/) that downloads/builds/installs MHD. Add that script to our CI runner YAML file(s) and we can test on MinGW as well. Let me know which packages you need for the docker images. > > MHD does the same for libcurl. Making the dependency optional also avoids > > the obvious possibility of circular dependencies if we ever were to add > > libwget2-based tests to MHD :-). > > That's our goal too. Make libwget as alternative for libcurl. > > [1] https://github.com/rockdaboot/wget2/issues/128 With Best Regards, Tim signature.asc Description: This is a digitally signed message part.
Re: [Bug-wget] [GSoC Update] Week 1
On Mon, 05 Jun 2017 23:51:00 +0700 Christian Grothoff wrote: > Maybe I missed something, but why should the MHD dependency become > _mandatory_? Why not conditionally-compile and run the tests only if MHD > is present? Maybe I just misinterpreting this [1]? What Darshit said about removing old server code from the codebase will make Wget2 more stable since we will not be susceptible to bugs in the HTTP Server code. So, if it is better options, I will amend my current change, make MHD optional, give fallback to the old server code if MHD not available. > MHD does the same for libcurl. Making the dependency optional also avoids the > obvious possibility of circular dependencies if we ever were to add > libwget2-based tests to MHD :-). That's our goal too. Make libwget as alternative for libcurl. [1] https://github.com/rockdaboot/wget2/issues/128
Re: [Bug-wget] [GSoC Update] Week 1
Maybe I missed something, but why should the MHD dependency become _mandatory_? Why not conditionally-compile and run the tests only if MHD is present? MHD does the same for libcurl. Making the dependency optional also avoids the obvious possibility of circular dependencies if we ever were to add libwget2-based tests to MHD :-). On 06/05/2017 06:46 PM, Didik Setiawan wrote: > * With Libmicrohttpd becomes mandatory package to install before building > Wget2 >binary, there must be proper warning about this requirement, otherwise the >building process will fail. I have add oneliner information into README.md. >Please notice me if there are some other places where I can add warning > like >this. signature.asc Description: OpenPGP digital signature
[Bug-wget] [GSoC Update] Week 1
= SUMMARY == My public fork of Wget2 project is available here [1]. I will continuously keep pushing my work so anyone interested can track me there. Feel free to participate in the discussions going on merge request with my mentors. Your feedback are highly appreciated. === INTRODUCTION === The purpose of this project is to use Libmicrohttpd as test suite for Wget2. I plan to do this by do some changes on function wget_test_start_server() also wget_test_stop_server() on src/libtest.c of Wget2. With this approach, I don't need to change existing test suite which call the internal server code through functions mentioned above. I've count there are 36 test file which use wget_test_start_server(). I must ensure all the test passed. And for installation prerequisite, I must ensure that Libmicrohttpd are included when building Wget2 binary. Then I need to modify configure.ac. I will give proper warning about this requirement. There is a section in README.md where I must explain to user to provide Libmicrohttpd to make all test running correctly. With Libmicrohttpd I can add new test using feature that not yet implemented in old server code, but ready on Libmicrohttpd, such as HTTP authentication [2] and concurrent request checking. Mentors: Darshit Shah Ander Juaristi == UPDATES = Things which were done in this week: * I have finished modify configure.ac to include Libmicrohttpd into Wget2. I keep my work in this branch [3] of my repository. * I have ensured that all make check passed on several testing machine including: Debian/GCC, Fedora/Clang, MingW64 [4] and OSX [5]. Especially for MingW64 build, because I haven't found the correct package for Libmicrohttpd, I include Libmicrohttpd by downloading the source and compile manually. See [6]. Any better idea will be reconsidered and appreciated. * With Libmicrohttpd becomes mandatory package to install before building Wget2 binary, there must be proper warning about this requirement, otherwise the building process will fail. I have add oneliner information into README.md. Please notice me if there are some other places where I can add warning like this. = NEXT STEPS === Things which would be done in the coming week: * Finished on wget_test_start_server() in order to call Libmicrohttpd as service for wget_test(). [1]: https://gitlab.com/dstw/wget2 [2]: https://www.gnu.org/software/libmicrohttpd/manual/libmicrohttpd.html#microhttpd_002ddauth [3]: https://gitlab.com/dstw/wget2/tree/use-mhd [4]: https://gitlab.com/dstw/wget2/builds/ [5]: https://travis-ci.org/dstw/wget2/jobs/239096327 [6]: https://gitlab.com/dstw/wget2/builds/17826461 Regards, Didik Setiawan
