[Bug-wget] [GSoC] Proposal: “Design and implementation of a framework for plugins.”
Hello, First of all, thanks for the initial review of my proposal! I have made some changes in response to the comments as well as last mail on the mailing list by Darshit Shah. Kindly consider reviewing it. For reference here is the link. https://docs.google.com/document/d/1SBM-jNG5dJL85C7iK9aid2eDlByrxOsYUNDrk4049vQ/edit?usp=sharing The proposal is still not final, in the sense that I am analysing the code for further ideas of what can be customized using plugins. I was asked to elaborate on rewriting of command line option processing code. So here are my thoughts on it. The current code in src/options.c has following drawbacks: - Built around order-insensitive and idempotent model (Supplying an option modifies a variable). This might be adequete for wget, but might not be the case for plugins. Repeated options or order of options can matter for plugins. - The need for keeping the list of options sorted. I don't think we can expect plugin writers to do the same. - And most importantly, options cannot be added at runtime. So far I think rewriting the option processing code would be easier than trying to patch existing code to support custom options. Some parts of the original code could be reused but structure would undergo a drastic change anyway. The idea I have is to expose an interface for plugins to describe command line options they want to receive. I have attached a prototype header. From this interface, wget will have a list of command line options to support, along with plugin function to be called for handling plugin specific options. In addition to using this to parse command line, wget can Use the descriptions and category information provided to generate --help output. Regarding the need to support plugin side addition of command line options at the first place, here are my thoughts. Plugins might need additional data from the user. I can think of 3 ways in which it can be done: 1. Environment variables. 2. Files with well-known names. 3. Command line. Environment variables have a disadvantage that they are automatically passed to the subprocesses, creating a chance for interference. Also, name clashes cannot be discovered. Using files with well-known names has a chance of interference with parallel running instances of wget. None of these disadvantages are shared with using command line options. Opinions are welcome. Regards, Akash Rawal. /* This is a prototype. Final product can be different. */ /** * \addtogroup argp * \{ * * This section documents functions for extending command line * option processing. * * The procedure is to first create a new option list using * _wget_argp_optlist_init_. Options and categories are then added by * _wget_argp_optlist_add_option_ and _wget_argp_optlist_add_category_. * Finally, options are added for consideration by _wget_argp_add_optlist_. * After this, option list should be free'd using _wget_argp_optlist_free_. */ ///Data structure for command line options typedef struct _wget_argp_optlist_t wget_argp_optlist_t; /**Creates a new option list * \return A newly constructed option list. Free with _wget_argp_optlist_free_ */ wget_argp_optlist_t *wget_argp_optlist_init(); /**Frees all memory associated with _optlist_. * \param optlist An option list created with _wget_argp_optlist_init_ */ void wget_argp_optlist_free(wget_argp_optlist_t *optlist); /**Adds a new category to command line options. * \param optlist An option list * \param catid Category identifier. This must be unique. * \param desc Description which will appear in the _--help_ output. */ void wget_argp_optlist_add_category (wget_argp_optlist_t *optlist, const char *catid, const char *desc); ///Type of command line options supported typedef enum { ///Option accepts no arguments WGET_ARGP_ARG_NONE, //Option requires argument. WGET_ARGP_ARG_REQUIRED, //Option accepts argument, but it is optional. WGET_ARGP_ARG_OPTIONAL } wget_argp_option_arg_type; /**Adds a command line option to option list. * \param catid Category identifier to specify which category to * add option to in _--help_ output. * \param optlist An option list * \param id A numeric ID to associate with the option. * \param short_option Single letter short option, or 0 for no short option. * \param option Long option. * \param arg_desc A short description of the argument it takes. * \param argtype Argument accepting policy. See _wget_argp_option_arg_type_ * \param negatable Nonzero if _option_ can be supplied as --no-option. * \param desc Description which will appear in _--help_ output. */ void wget_argp_optlist_add_option (wget_argp_optlist_t *optlist, const char *catid, int id, char short_option, const char *option, const char *arg_desc, wget_argp_option_arg_type argtype, int negatable, const char *desc); /**Type of function to be registered as a callback * \param id Identifier of the opti
Re: [Bug-wget] [GSoC] Proposal: “Design and implementation of a framework for plugins.”
On 04/02/2017 10:41 PM, Darshit Shah wrote: * Akash Rawal [170331 23:34]: Hello, First of all, thanks for the initial review of my proposal! I have made some changes in response to the comments as well as last mail on the mailing list by Darshit Shah. Kindly consider reviewing it. For reference here is the link. https://docs.google.com/document/d/1SBM-jNG5dJL85C7iK9aid2eDlByrxOsYUNDrk4049vQ/edit?usp=sharing The proposal is still not final, in the sense that I am analysing the code for further ideas of what can be customized using plugins. I was asked to elaborate on rewriting of command line option processing code. So here are my thoughts on it. The current code in src/options.c has following drawbacks: - Built around order-insensitive and idempotent model (Supplying an option modifies a variable). This might be adequete for wget, but might not be the case for plugins. Repeated options or order of options can matter for plugins. Breaking the order-insensitive and idempotent model of command line processing is an absolute no-go. It will definitely break many scripts out there. We do wish to make it easy to transition from Wget to Wget2 and this is a huge change in how options are processed. Also, I see no clear advantages to why this should be required. Backward compatibility will only be broken if options are changed, which I didn't plan to do. wget2 will support all options it supports now. Existing options shall remain order-insensitive and idempotent, only new options and plugin-side options may take advantage of greater flexibility. Many programs have order-sensitive (e.g. ffmpeg, options apply to input or output file based on placement) and non-idempotent (e.g. -v (verbose) option in many program, -vv for higher verbosity) - The need for keeping the list of options sorted. I don't think we can expect plugin writers to do the same. No. Plugins should not be allowed to tinker with certain core functionalities and that includes how options are processed. - And most importantly, options cannot be added at runtime. Which is a very good thing. Keeping everything at compile time means we can have most of that code on the stack without the need for costly dynamic memory allocations on each invokation. Options processing code should definitely not be dynamic. Okay if that is your requirement I'll remove it from my proposal. So far I think rewriting the option processing code would be easier than trying to patch existing code to support custom options. Some parts of the original code could be reused but structure would undergo a drastic change anyway. Easier? Have you every tried writing one? The number of use cases and corner cases you'd have to handle is massive. That alone could be an entire GSoC project. You can't expect to write it as a byproduct of your project. I have written simpler ones. But even in wget's code option processor is not so large (~400 LOC, starting from src/option.c:853) Cases for long options: --option --option=value --option value --no-option Cases for short options: -x -x value -xvalue -xyzw -xyzvalue -xyz value The idea I have is to expose an interface for plugins to describe command line options they want to receive. I have attached a prototype header. From this interface, wget will have a list of command line options to support, along with plugin function to be called for handling plugin specific options. In addition to using this to parse command line, wget can Use the descriptions and category information provided to generate --help output. First let me ask you, how exactly do you envision a user using plugins with Wget2? How are plugins selected? How does the user even know if a plugin exists? How about: $ wget_plugins="foo" wget2 --foo-option http://example.com/file.html Or: $ wget2 --plugin=foo --foo-option http://example.com/file.html Hypothetical example: music site parser that filters by rating $ wget2 --plugin=music_rating_filter --music_min_rating=4 \ --recursive --level=20 \ http://example_music_site.com/music/1 Multiple plugins: $ wget_plugins="foo:bar" wget2 --plugin=foo --plugin=bar \ --foo-option --bar-option=value http://example.com/file.html Or: $ wget2 --plugin=foo --plugin=bar --foo-option --bar-option=value \ http://example.com/file.html $ #The above is also an example of non-idempotent option --plugin Loading plugins which are not installed: $ wget2 --local-plugin=/path/to/libfoo.so --foo-option \ http://example.com/file.html And regarding user knowing which plugins are installed: $ wget2 --list-plugins foo bar music_rating_filter Plugins could be installed in a directory like ${PREFIX}/lib/wget/plugins/. Many software do like this. Regarding the need to support plugin side addition of command line options at the fir
Re: [Bug-wget] [GSoC] Proposal: “Design and implementation of a framework for plugins.”
On 04/03/2017 04:20 PM, Tim Rühsen wrote: I like this. We can have this without changing much of Wget2's option processing code. Add --plugin to Wget2 (we will have a list/array of plugin names than). Define a option name scheme to identify options for a certain plugin, e.g. --plugin-foo-=value. Make as as simple as possible, e.g. a value is mandatory (bool is 1|0 or y|n or whatever). When the current option processing doesn't find an option in it's list, we special check for --plugin-- and give the key/value pair / to a function that is provided by the plugin (e.g. wget_plugin_set_option(const char *option_name, const char *value). With this you can provide CLI options to plugins without changing much in Wget2's code. Regards, Tim This could be done with the following: - Add condition to set_long_option() in option.c checking if _option_ begins with --plugin- (4-5 lines), if yes strip the leading "plugin-" and call another function, say set_plugin_option(option, value) - set_plugin_option() will look up maximum length matching "-" using a variant of binary search. - If the plugin discovered has opted in to receive options, strip "-" _option_ and pass the remaining option and value to plugin-registered function. To implement this all we need to maintain is a sorted list of plugins. No extra memory allocations will be needed for option processing. I could still change proposal before 16:00 GMT. So what do you people think? (Also a friendly reminder to everyone that final proposals can be reuploaded till 16:00 GMT, so don't wait till last moment. ) Regards, Akash Rawal
Re: [Bug-wget] Problems with (not) building wget against libiconv
On 04/18/2017 02:32 PM, Tim Rühsen wrote: I have some problems with bootstrapping. On Solaris it's: ./bootstrap ./bootstrap: syntax error at line 91: `me_=$' unexpected I experience this as well and use 'bash ./bootstrap'. Autoconf manual says $(...) style command substitution is less portable than using backticks `...`. https://www.gnu.org/software/autoconf/manual/autoconf.html#Shell-Substitutions Hope it helps. I am unable to do any testing at the moment, so sorry. Regards, Akash Rawal
Re: [Bug-wget] Installing wget2 error
On 05/20/2017 04:44 AM, Naoki Eto wrote: Hi, I am having trouble installing wget2 . I am getting the following error when trying to use wget2: wget2: error while loading shared libraries: libwget.so.0: cannot open shared object file: No such file or directory Consider adding $prefix/lib to /etc/ld.so.conf. Replace $prefix with the prefix where you installed wget2. Or pass --prefix=/usr to configure script. When I do make check, it passes all 33 tests. When I do make check-valgrind, it only passes 1 test. I have attached the error log. You need to install valgrind for valgrind tests. I was wondering if you have any suggestions on how to fix this. I am using ubuntu 16.04 currently. -Naoki Regards, Akash Rawal
Re: [Bug-wget] wget2 fails to build if spaces in directory path
On 06/26/2017 10:28 PM, Avinash Sonawane wrote: On Mon, Jun 26, 2017 at 3:49 PM, chris wrote: Sure: https://gist.github.com/anfractuosity/0a9c12f81ebbb1033883bbec28b62dcd This is happening because absolute paths have been used in Makefiles. Please find the attached patch which fixes the issue. Sorry, haven't sent the merge request to upstream as `make check` still fails. but you can do 'make' and 'sudo make install' the wget2. We use absolute paths because relative paths break tests when build tree is separate from source tree. mkdir build cd build ../configure && make && make check The reason the make check fails is that test program cd's to a subdirectory to run tests. After changing directory, the relative path is no longer correct. To further complicate the issue, srcdir and top_srcdir are not guaranteed to be relative paths. Apply the patch to your local git wget2 repo (git apply) if you are keen on building the wget2 at a location with spaces in it's absolute path. Regards, Avinash Sonawane (rootKea) PICT, Pune https://rootkea.wordpress.com Regards, Akash Rawal