-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 So, I've been spending some time thinking about how I want to break things down in the next several versions of Wget. Here's a summary of how I think it'll break down; following it is a more in-depth discussion.
1.12 - ---- Support for parsing links from CSS. Polishing and bug-fixing of Content-Disposition support. RFC-compliant HTTP authentication. Support for IRIs, or at least basic support for IDNs. GNUTLS. 1.13 - ---- Many, many small bugfixes and improvements. HTTP/1.1 support, especially Transfer-Encoding. A streaming HTML parser. Support for basic external program filters. Filtering HTML before processing Filtering Content-Encoding (e.g., gzip) Filtering names of locally-saved files. Registered filter handlers for non-HTML content types, to allow traversal of links in (e.g.) PDFs or XML, etc. The "metadata database" 2.0 - ---- Support for dynamically-loaded modules: Modular protocol handling. Modular transport-layer handling (SOCKS, SCTP, connection via the HTTP CONNECT method, TLS, IP over Carrier Pigeon, ...). Modular filestore support (Download to tarball, .mht, ...). Revamped commandline and .wgetrc: Support for host- and path-specific configuration settings. Regex support. 2.1 - ---- Support for multiple simultaneous connections? 1.12 - ---- The really big deal here, to me, is CSS. I want to have CSS support for Wget ASAP. It's an essential part of the Web, and users definitely suffer for the lack of support for it. I'd love to just get that done, and get it out; but there are a number of other things that for me are too important to hold off. For one, the Content-Disposition thing needs some cleaning up, polishing, and a bit of bugfixing. And, the HTTP authentication code gained some improvement in 1.11, but still isn't RFC-compliant, and I'd really like it brought up to speed. Internationalization is quickly becoming more and more of an issue. I had been planning on putting that off for a couple revisions, but I don't think I can do that any longer. At the very least, we need to handle IDNs, as it looks like ICANN will be introducing IDN ccTLDs in the latter half of 2008: http://ccnso.icann.org/workinggroups/idn-time-table-19dec07.htm While implementing IDN support wouldn't be terribly difficult, it would be much more effective as part of a more complete support for IRIs in general; especially, transcoding between arguments entered in the user's locale to an IRI, and back to the user's character encoding for filenames. I don't know if all this can be accomplished for 1.12, but getting _something_ accomplished toward this end would be a very good idea. Support for GNUTLS would also be great to have in 1.12, if it's not too much effort. As I understand it, there are just a few kinks to be worked out for that, in which case it'd be worth trying to get in for then. 1.13 - ---- In my haste to get these all-important features out the door, a lot of other, lesser but worthy improvements will have gone by the wayside. 1.13 will be when I finally have a little breathing room to tackle some of these. I've gone through all the bug reports very carefully, and everything that I could see myself postponing has been shifted from 1.12 to 1.13 in Savannah. This includes several "pet" issues I'd really like to see go in for 1.12--and many of them really wouldn't take too long--but together they add up quite a bit, and considering that even without them, 1.12 has got a pile of work to be done, it's best to postpone them. Among some of the more interesting and significant things I'd like to do in 1.13, however, is the ability to have Wget call out to external programs for further processing. A good example is the recent - --transform-html proposal, for post-processing downloaded HTML files (though for that and some other use cases, it might make more sense to feed the already-parsed links to the filter, rather than the entire HTML file); this idea could also be adapted for processing Content-Encoded resources (say, compressed via gzip). Registering handlers for non-HTML content types, to spit out more URLs, could also be useful; as would be filters to translate URLs into paths suitable for placing downloaded files. Reworking the HTML parsing code to be able to handle streamed content instead of slurping everything in memory would then be advantageous, as it would avoid the use of temporary files, and allow Wget to process data as it comes, rather than after everything's been received. This would make particular sense in feeding URLs to filter programs. HTTP/1.1 support, especially support for the "chunked" Transfer-Encoding, would augment Wget's existing support for persistent connections. Web sites are relying more and more on dynamic content generation, and when a dynamically generated page means that the remote server can't know in advance what the Content-Length should be, its only option to indicate to Wget that a file has completed transfer is to close the connection. Note that, in addition to being inefficient, a connection that was closed in order to indicate the end of a resource is indistinguishable from a connection that was lost, and did not complete. The "chunked" transfer encoding enables servers to indicate the end of dynamically-generated content, so that the connection may be kept open. With filtered filenames--and already for features like Content-Disposition, it will be extremely useful to have a file that maps between local filenames and the URLs from which they came; instead of having to perform a GET on a URL, discover that it has a Content-Disposition header, look for the local file corresponding to that header, and check the timestamps and/or accept/reject lists and terminate the connection (a nasty habit to get into with persistent connections), Wget would be able to just look up the URI in the local database, find out what local file it mapped to, and perform the appropriate checks based on that information. Also, with IRI support, it will be important for Wget to obtain information about a resource's character encoding, so that it knows how to interpret the IRIs found within it. This kind of information, and the URI <-> filename mapping, will be stored in what I've been calling, the "metadata database." 2.0 and beyond - ---- I'm not going to talk much about the rest here, except to say that multiple-connection support is not likely to get any of my attention until a long time from now. There are simply too many other important matters to address. As far as Wget's current functionality goes, multiple connections won't drastially improve most use cases, since we'd limit the number of connections to a single host anyway: the biggest gains would be in multi-host recursive fetches, and extensions like the MetaLink functionality. For these areas, of course, it would bring great benefit, but the amount of work involved in achieving it simply isn't feasible for the time being. All the other features I've mentioned can be adapted on top of the current code base without an inordinate amount of trouble. However, supporting multiple connections would require a complete reworking of pretty much every aspect of Wget's code. It's better to hold off on that so we can focus on more pressing needs. - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer... http://micah.cowan.name/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHnGjA7M8hyUobTrERAvXvAJ43cIMZQvnoG/0gOG+SFqMV9FpGvwCff8T4 WcIkduwSWb6ZQDBVkxpKfDg= =BDMJ -----END PGP SIGNATURE-----