[Bug-wget] [GSoC Update] Week 11

[Didik Setiawan]

= SUMMARY ==
My public fork of Wget2 project is available here [0]. I will continuously keep
pushing my work so anyone interested can track me there. Feel free to
participate in the discussions going on merge request with my mentors [1]. Your
feedback are highly appreciated.


=== INTRODUCTION ===
The purpose of this project is to use Libmicrohttpd as test suite for Wget2.  I
plan to do this by do some changes on function wget_test_start_server() also
wget_test_stop_server() on src/libtest.c of Wget2. With this approach, I don't
need to change existing test suite which call the internal server code through
functions mentioned above.
I've count there are 33 test file which use wget_test_start_server(). I must
ensure all the test passed when I've done the transition.
For installation prerequisite, I make that Libmicrohttpd support will needed
when we need to run `make check` or run test suite. Then I need to modify
configure.ac. I will give proper warning about this requirement. There is a
section in README.md where I must explain to user to provide Libmicrohttpd to
make all test running correctly.
With Libmicrohttpd I can add new test using feature that not yet implemented in
old server code, but ready on Libmicrohttpd, such as HTTP digest 
authentication. 

Mentors:
Darshit Shah <dar...@gmail.com>
Ander Juaristi <ajuari...@gmx.es>


== UPDATES =
Things which were done until this week:

 * Started working on wget_test_start_server() to use Libmicrohttpd as HTTP
   server. Workflow to resolve this:
   - Remove initial process for HTTP server socket
   - Create _http_server_start() function, wrapper for Libmicrohttpd. There is
 also function _answer_to_connection() which use to create proper HTTP
 response
   - Use select method (MHD_USE_SELECT_INTERNALLY) for threading model in
 Libmicrohttpd to get better compatibility
   - http_server_port seized automatically using Libmicrohttpd function by
 passing MHD_DAEMON_INFO_BIND_PORT or MHD_DAEMON_INFO_LISTEN_FD parameter to
 MHD_get_daemon_info() 
   - Using iteration to parse urls data in _answer_to_connection(). This
 guarantee we can pass any variadic data to Libmicrohttpd and prevent
 segmentation fault 
   - Fix _answer_to_connection() function to create proper HTTP response:
 - Handle arguments (query string) on URL
 - Handle redirection
 - Handle directory creation
 - Handle URL with IRI object
 - Handle URL with IDN hostname
 - Handle URL with query string which contains space
 - Handle If-Modified-Since header
 - Handle Byte Range request
 - Handle HTTP basic authentication 
 - Handle HTTPS
 - Handle chunked transfer encoding Notes: I use
   MHD_create_response_from_callback() from Libmicrohttpd to generate
   chunked transfer method. For simulation with bad chunked data, I use
   MHD_create_response_from_buffer() with proper header and use connection
   close. I think the result of this method is most closely approached with
   the result from old HTTP server
 * Fix compiler warning -Wunused-param
 * Use persistent connection (Connection: keep-alive) instead multiple
   connection (Connection: close). This prevent hiding several problems of the
   connection between HTTP server and client
 * Fix CI testing on Travis OSX: must install Libmicrohttpd from source to get
   support for HTTPS. Maybe the Libmicrohttpd package provided by brew not
   compiled with HTTPS support
 * Fix CI testing on Gitlab MinGW64. That problem caused by variable clash
   between Wget2 and MinGW64 header have been reported and fixed
 * Add CI testing to simulate Wget2 build without Libmicrohttpd installed. This
   is just a copy of main build process but without 'make check' part, so the
   Wget2 binary can be built with skipped testing [2][3]


= NEXT STEPS ===
Things which would be done in the coming week:

 * Add HTTP digest authentication test
 * Add other tests which seems to possible to add in this time frame
   (compression test, IPv6 test, etc). I need more suggestion about this


[0]: https://gitlab.com/gnuwget/wget2/tree/tmp-use-mhd
[1]: https://gitlab.com/gnuwget/wget2/merge_requests/265 
[2]: https://gitlab.com/dstw/wget2/tree/mhd-not-install
[3]: https://github.com/dstw/wget2/tree/mhd-not-install

Regards,
Didik Setiawan

[Bug-wget] [GSoC Update] Week 10

[Didik Setiawan]

= SUMMARY ==
My public fork of Wget2 project is available here [0]. I will continuously keep
pushing my work so anyone interested can track me there. Feel free to
participate in the discussions going on merge request with my mentors. Your
feedback are highly appreciated.


=== INTRODUCTION ===
The purpose of this project is to use Libmicrohttpd as test suite for Wget2.  I
plan to do this by do some changes on function wget_test_start_server() also
wget_test_stop_server() on src/libtest.c of Wget2. With this approach, I don't
need to change existing test suite which call the internal server code through
functions mentioned above.
I've count there are 38 test file which use wget_test_start_server(). I must
ensure all the test passed when I've done the transition.
For installation prerequisite, I make that Libmicrohttpd support will needed
when we need to run `make check` or run test suite. Then I need to modify
configure.ac. I will give proper warning about this requirement. There is a
section in README.md where I must explain to user to provide Libmicrohttpd to
make all test running correctly.
With Libmicrohttpd I can add new test using feature that not yet implemented in
old server code, but ready on Libmicrohttpd, such as HTTP digest authentication
and concurrent request checking. 

Mentors:
Darshit Shah <dar...@gmail.com>
Ander Juaristi <ajuari...@gmx.es>


== UPDATES =
Things which were done until this week:

 * Started working on wget_test_start_server() to use Libmicrohttpd as HTTP
   server. Workflow to resolve this:
   - Remove initial process for HTTP server socket
   - Create _http_server_start() function, wrapper for Libmicrohttpd. There is
 also function _answer_to_connection() which use to create proper HTTP
 response
   - Use select method (MHD_USE_SELECT_INTERNALLY) for threading model in
 Libmicrohttpd to get better compatibility
   - http_server_port seized automatically using Libmicrohttpd function by
 passing MHD_DAEMON_INFO_BIND_PORT or MHD_DAEMON_INFO_LISTEN_FD parameter to
 MHD_get_daemon_info() 
   - Using iteration to parse urls data in _answer_to_connection(). This
 guarantee we can pass any variadic data to Libmicrohttpd and prevent
 segmentation fault 
   - Fix _answer_to_connection() function to create proper HTTP response:
 - Handle arguments (query string) on URL
 - Handle redirection
 - Handle directory creation
 - Handle URL with IRI object
 - Handle URL with IDN hostname
 - Handle URL with query string which contains space
 - Handle If-Modified-Since header
 - Handle Byte Range request
 - Handle HTTP basic authentication 
 - Handle HTTPS
 - Handle chunked transfer encoding Notes: I use
   MHD_create_response_from_callback() from Libmicrohttpd to generate
   chunked transfer method. For simulation with bad chunked data, I use
   MHD_create_response_from_buffer() with proper header and use connection
   close. I think the result of this method is most closely approached with
   the result from old HTTP server.
 * Fix compiler warning -Wunused-param
 * Use persistent connection (Connection: keep-alive) instead multiple
   connection (Connection: close). This prevent hiding several problems of the
   connection between HTTP server and client
 * Fix CI testing on Travis OSX: must install Libmicrohttpd from source to get
   support for HTTPS. Maybe the Libmicrohttpd package provided by brew not
   compiled with HTTPS support.


= NEXT STEPS ===
Things which would be done in the coming week:

 * Fix CI testing on Gitlab MingW64
   - MingW64: Need a script for complete setup Libmicrohttpd
 * Add HTTP digest authentication test


[0]: https://gitlab.com/gnuwget/wget2/tree/tmp-use-mhd

Regards,
Didik Setiawan

[Bug-wget] [GSoC Update] Week 9

[Didik Setiawan]

= SUMMARY ==
My public fork of Wget2 project is available here [0]. I will continuously keep
pushing my work so anyone interested can track me there. Feel free to
participate in the discussions going on merge request with my mentors. Your
feedback are highly appreciated.


=== INTRODUCTION ===
The purpose of this project is to use Libmicrohttpd as test suite for Wget2.  I
plan to do this by do some changes on function wget_test_start_server() also
wget_test_stop_server() on src/libtest.c of Wget2. With this approach, I don't
need to change existing test suite which call the internal server code through
functions mentioned above.
I've count there are 38 test file which use wget_test_start_server(). I must
ensure all the test passed when I've done the transition.
For installation prerequisite, I make that Libmicrohttpd support will needed
when we need to run `make check` or run test suite. Then I need to modify
configure.ac. I will give proper warning about this requirement. There is a
section in README.md where I must explain to user to provide Libmicrohttpd to
make all test running correctly.
With Libmicrohttpd I can add new test using feature that not yet implemented in
old server code, but ready on Libmicrohttpd, such as HTTP digest authentication
and concurrent request checking. 

Mentors:
Darshit Shah <dar...@gmail.com>
Ander Juaristi <ajuari...@gmx.es>


== UPDATES =
Things which were done until this week:

 * Started working on wget_test_start_server() to use Libmicrohttpd as HTTP
   server. Workflow to resolve this:
   - Remove initial process for HTTP server socket
   - Create _http_server_start() function, wrapper for Libmicrohttpd. There is
 also function _answer_to_connection() which use to create proper HTTP
 response
   - Use select method (MHD_USE_SELECT_INTERNALLY) for threading model in
 Libmicrohttpd to get better compatibility
   - http_server_port seized automatically using Libmicrohttpd function by
 passing MHD_DAEMON_INFO_BIND_PORT or MHD_DAEMON_INFO_LISTEN_FD parameter to
 MHD_get_daemon_info() 
   - Using iteration to parse urls data in _answer_to_connection(). This
 guarantee we can pass any variadic data to Libmicrohttpd and prevent
 segmentation fault 
   - Fix _answer_to_connection() function to create proper HTTP response:
 - Handle arguments (query string) on URL
 - Handle redirection
 - Handle directory creation
 - Handle URL with IRI object
 - Handle URL with IDN hostname
 - Handle URL with query string which contains space
 - Handle If-Modified-Since header
 - Handle Byte Range request
 - Handle HTTP basic authentication 
 - Handle HTTPS
 - Handle chunked tranfer encoding (though still need more discussion)
 * Fix compiler warning -Wunused-param caused by MHD API
 * Use persistent connection (Connection: keep-alive) instead multiple
   connection (Connection: close). This prevent hiding several problems of the
   connection between HTTP server and client


= NEXT STEPS ===
Things which would be done in the coming week:

 * Fix CI testing on Gitlab MingW64 and Travis OSX
   - MingW64: Need a script for complete setup Libmicrohttpd with HTTPS support
   - OSX: Need more verbose debug since it fails to run HTTPS server
 * Add HTTP digest authentication test


[0]: https://gitlab.com/gnuwget/wget2/tree/tmp-use-mhd

Regards,
Didik Setiawan

[Bug-wget] [GSoC Update] Week 3

[Didik Setiawan]

= SUMMARY ==
My public fork of Wget2 project is available here [1]. I will continuously keep
pushing my work so anyone interested can track me there. Feel free to
participate in the discussions going on merge request with my mentors. Your
feedback are highly appreciated.


=== INTRODUCTION ===
The purpose of this project is to use Libmicrohttpd as test suite for Wget2.  I
plan to do this by do some changes on function wget_test_start_server() also
wget_test_stop_server() on src/libtest.c of Wget2. With this approach, I don't
need to change existing test suite which call the internal server code through
functions mentioned above.
I've count there are 38 test file which use wget_test_start_server(). I must
ensure all the test passed when I've done the transition.
For installation prerequisite, I must ensure that Libmicrohttpd are included
when building Wget2 binary. Then I need to modify configure.ac. I will give
proper warning about this requirement. There is a section in README.md where I
must explain to user to provide Libmicrohttpd to make all test running
correctly.
With Libmicrohttpd I can add new test using feature that not yet implemented in
old server code, but ready on Libmicrohttpd, such as HTTP authentication [2] and
concurrent request checking. 

Mentors:
Darshit Shah <dar...@gmail.com>
Ander Juaristi <ajuari...@gmx.es>


== UPDATES =
Things which were done in this week:

 * I have finished modify configure.ac to include Libmicrohttpd into Wget2. I
   keep my work in this branch [3] of my repository.
 * I have ensured that all make check passed on several testing machine
   including: Debian/GCC, Fedora/Clang, MingW64 and OSX.
 * Started working on wget_test_start_server(). Workflow to resolve this:
   - Remove initial process for HTTP server socket.
   - Create _http_server_start() function, wrapper for Libmicrohttpd. There is
 also function answer_to_connection() which use to create proper HTTP
 response.
   - Use select method (MHD_USE_SELECT_INTERNALLY) for threading model in
 Libmicrohttpd to get better compatibility.
   - http_server_port seized automatically using Libmicrohttpd function by
 passing MHD_DAEMON_INFO_BIND_PORT or MHD_DAEMON_INFO_LISTEN_FD parameter to
 MHD_get_daemon_info(). 
   - Using iteration to parse urls data in answer_to_connection(). This
 guarantee we can pass any variadic data to Libmicrohttpd and prevent
 segmentation fault. 
   - Fix answer_to_connection() function to create proper HTTP response (to deal
 with parameters and arguments on url, to add proper HTTP headers).


= NEXT STEPS ===
Things which would be done in the coming week:

 * Make sure all test suite running correctly. I've created spreadsheet about
   summary of test file which use wget_test_start_server() [4]. Currently, it
   reaches 38% (14 of 22). I targeted to reaches 100% by start of first
   evaluation date.


[1]: https://gitlab.com/dstw/wget2
[2]: 
https://www.gnu.org/software/libmicrohttpd/manual/libmicrohttpd.html#microhttpd_002ddauth
[3]: https://gitlab.com/dstw/wget2/tree/use-mhd
[4]: 
https://docs.google.com/spreadsheets/d/13KzAXiB2mfud-c9LSdvKnnDHmAr5kUaDulkCA-p5q7M/edit?usp=sharing

Regards,
Didik Setiawan

Re: [Bug-wget] [GSoC Update] Week 2

[Didik Setiawan]

On Sat, Jun 17, 2017 at 01:30:57PM +0300, Evgeny Grin wrote:
> 
> On 17.06.2017 11:36, Tim Rühsen wrote:
>
> > All you need is the socket descriptor. How to call getsockname() + 
> > getnameinfo() to retrieve the port number you see in libwget/net.c/
> > wget_tcp_get_local_port().
> > 
> > If MHD doesn't have such a function, either try to get the socket 
> > descriptor 
> > or extend MHD with a small function (similar code as in  
> > wget_tcp_get_local_port).
> 
> There are several ways to implement it.
> First one:
> Initialise socket externally and start listen() on it. You can use any
> bind and port detection function.
> Pass socket FD to MHD_start_daemon() by MHD_OPTION_LISTEN_SOCKET.
> 
> Second one:
> Use MHD_start_daemon() with "0" as port number then use
> MHD_get_daemon_info() with MHD_DAEMON_INFO_LISTEN_FD to get listen
> socket FD. Use any port detection.
> 
> Third one:
> Implemented two day ago, works with MHD_VERSION >= 0x00095501 and on
> platforms that support getsockname().
> Use MHD_start_daemon() with "0" as port number then use
> MHD_get_daemon_info() with MHD_DAEMON_INFO_BIND_PORT to get port number.

Thanks for the detailed explanations. That's what I'm looking for, to fill the
missing piece.
 
> You can combine second and third methods:

I use this combination.

> ---
>   int port_num;
> 
>   if(0) {}
> #if MHD_VERSION >= 0x00095501
>   else if (MHD_NO !=
>  MHD_is_feature_supported (MHD_FEATURE_AUTODETECT_BIND_PORT))
>   {
> const union MHD_DaemonInfo *dinfo;
> dinfo = MHD_get_daemon_info (d, MHD_DAEMON_INFO_BIND_PORT);
> if (NULL == dinfo || 0 == dinfo->port)
> {
>   /* Insert code to handle error. */
>   return -1;
> }
> port_num = (int)dinfo->port;
>   }
> #endif /* MHD_VERSION >= 0x00095501 */
>   else
>   {
> const union MHD_DaemonInfo *dinfo;
> MHD_socket sock_fd;
> dinfo = MHD_get_daemon_info (d, MHD_OPTION_LISTEN_SOCKET);

Don't you mean:

dinfo = MHD_get_daemon_info(d, MHD_DAEMON_INFO_LISTEN_FD);

Because when I tried using MHD version < 0.9.55 the result is not as
expected.

> if (NULL == dinfo)
> {
>   /* Insert code to handle error. */
>   return -1;
> }
> sock_fd = dinfo->listen_fd;
> /* Insert code to use port detection on provided socket. */
>   }
> ---
> 
> Note that socket type used by MHD functions is 'MHD_socket'.
> MHD_socket is 'int' on POSIX platforms (Linux, *BSD, Unix, Darwin) and
> 'SOCKET' on Windows platform (excluding Cygwin, where MHD_socket is 'int').
> 
> -- 
> Best Wishes,
> Evgeny Grin
> 

Regards,
Didik Setiawan

Re: [Bug-wget] [GSoC Update] Week 2

[Didik Setiawan]

On Tue, Jun 13, 2017 at 06:37:27PM +0200, Darshit Shah wrote:
> >   - http_server_port still hardcoded.
> This is important. The port should be a random number. Usually, passing 0 in
> the port number makes the kernel choose an open one for you.
> 
> Having a randomised port is important to ensure that multiple runs don't
> step on each other.

I still don't know how to accomplish this. Maybe, it just my understanding that
when I pass 0 in MHD port number, the result is still 0.
Another approach, when I look into the old code, it generate port number by
calling wget_tcp_get_local_port(). But, I need to call wget_tcp_init() and
wget_tcp_listen() respectively in order to get proper result.
Conclusion, do I need to use existing wget_tcp_get_local_port() to get the port,
or maybe there is a function in MHD to do that?

Thank you.

Regards,
Didik Setiawan

Re: [Bug-wget] [GSoC Update] Week 1

[Didik Setiawan]

On Mon, 05 Jun 2017 23:51:00 +0700 Christian Grothoff  wrote:
> Maybe I missed something, but why should the MHD dependency become 
> _mandatory_? Why not conditionally-compile and run the tests only if MHD 
> is present?

Maybe I just misinterpreting this [1]? What Darshit said about removing old
server code from the codebase will make Wget2 more stable since we will not be
susceptible to bugs in the HTTP Server code. 
So, if it is better options, I will amend my current change, make MHD optional,
give fallback to the old server code if MHD not available.

> MHD does the same for libcurl.  Making the dependency optional also avoids the
> obvious possibility of circular dependencies if we ever were to add
> libwget2-based tests to MHD :-). 

That's our goal too. Make libwget as alternative for libcurl.

[1] https://github.com/rockdaboot/wget2/issues/128

Re: [Bug-wget] [GSoC] Proposal | Design and Implementation of Test Suite Using Libmicrohttpd

[Didik Setiawan]

On Sat, Apr 01, 2017 at 11:03:47AM +0200, Ander Juaristi wrote:
> Hi there Didik,
> 
> I added some comments to the doc. Hope you can have some time to look at
> them.

Hi Ander. Thanks for your review. It seems like I still miss a lot of point.
Your guidance is very valuable to me. I will update soon.

Regards,
Didik Setiawan

> Best regards,
> - AJ

[Bug-wget] [GSoC] Proposal | Design and Implementation of Test Suite Using Libmicrohttpd

[Didik Setiawan]

Hi!

I have prepared a proposal for Google Summer of Code 2017. I make decision to 
change my project from my initial subject intention, considering that currently 
I working on small patch covering the area of test suite which make me more
familiar with Wget2 codebase. 
I know this is a bit late, but please try to give your comments and suggestions.

Here is a Google doc for my proposal.
https://docs.google.com/document/d/1CHy-8_yKVYrBmkC9i4Sb7shpd1feec3Dqtp2WI2KY9M/edit?usp=sharing

Regards,
Didik Setiawan

Re: [Bug-wget] GSoC Project | Design and Implementation of a Framework for Plugins

[Didik Setiawan]

On Tue, Mar 21, 2017 at 10:58:57AM +0100, Tim Ruehsen wrote:
> Hi Didik,
> 
> please, don't start coding yet. The application period is not over (deadline 
> 3rd April). During 3-24 April we (the mentors) will review all applications 
> and then decide who will be accepted.
> 
> Work on a proposal about the plugin feature - feel free to discuss details or 
> ask questions.

My proposal plan:

- Implement libltdl (part of libtool) to main program. For now, I still need to
  learn the code base of Wget2, in order to find which files I need to change to
  implement this feature.
- Create data structure and an API to provide an access to all the details of a
  connection so it could be used by the plugins.
- Write some plugins or use external program so it could be loaded by Wget2 to
  do custom request.

> Give us a good impression by starting small works on the code base / testing 
> / 
> create issues / whatever (see also the answers to Avinash's questions about 
> what to do). Prove your skills.

You gave some valuable clue about how the unit test works and how to write it.
I'll try to write the new one.

Re: [Bug-wget] GSoC Project | Design and Implementation of a Framework for Plugins

[Didik Setiawan]

On Mon, Mar 20, 2017 at 06:44:18PM +0100, Ander Juaristi wrote:
> Maybe this is not 100% related to the plugin framework, but we need a
> consistent API to manipulate all stages of an HTTP request (connection
> establishment, SSL/TLS handshake, SSL/TLS cert verification, etc.) in a
> single place, as I once told Tim off-list.
> 
> Have a look at #142 [0], for instance. Although it mentions HPKP, there
> are other parts that would benefit from this as well.
> 
> I was thinking of an event-based framework (e.g. plugins subscribe to
> 'events' and get called when they happen). Tim, on the other hand,
> prefers a data structure where you could access all the details of a
> connection (e.g. something like 'wget_connection_t', with references to
> 'wget_tls_t', etc.), and this data structure gets passed to plugins and
> they decide what to do.

Thanks for the explanation. I would to ask, what is benefit of this event-based
framework approach compared to the data structure implementation?

Re: [Bug-wget] GSoC Project | Design and Implementation of a Framework for Plugins

[Didik Setiawan]

On Mon, Mar 20, 2017 at 04:15:44PM +0100, Tim Ruehsen wrote:
> 
> Welcome, Didik !

Thanks, Tim.

> One way to implement plugins is via libdl (dlopen(), ...), and that is what I 
> have in mind. That is not perfectly portable, but our first goal will be 
> systems that support dlopen().

So, should I continue to use dlopen() or there is another better method? In 
case 
we need more portability.

> Since we can't (or want) to make everything pluggable, we should first 
> address 
> the already requested functionality. There should be (some ?) requests in the 
> wget1.x issue tracker that ask for plugin / external program for some good 
> reasons. You should review these, pick the parts that seems reasonable to you 
> and implement these first.
>
> E.g. I remember someone asking for some customisable 'filter' for URLs to 
> follow. In terms of a loadable module, that would be a function that takes a 
> URL and returns 1|0 (yes|no). Or a bit more general: takes a URL plus extra 
> information, e.g. referer, host, mimetype, ... and maybe returns a list of 
> URLs to fetch. Whatever makes sense.

Do you talking about this [1]? Then I refer to one of the three idea that says
call an external program which returns 0 or 1 to signal acceptance or 
rejectance 
of each proposed URI. Wget shall wait for the program to return. Any other exit
status shall cause wget to terminate.
I will try to implement this. 

> One goal would be to make up data structures and an API that can be extended 
> without breaking compatibility between  wget2 and the plugin in the future. 
> E.g. a newer wget2 should still be able to work with an older plugin and vice 
> versa. 

I will note this.

[1] https://savannah.gnu.org/bugs/?45803