Hi, I'm using wget for years now on various OSes, it's a wonderful tool! Many thanks for it!
BTW1: It took me some time to find get-rid-of-robots feature as '-e robots=off'. It probably is an intention. Funny.:) BTW2: The source is really nice and it's commentary is great! I have some notes and "feature requests". I needed some minor new functionality quickly so I downloaded 1.9, changed it and I'm using it locally for myself. I think someone else may like my changes and ideas, there are more of them than I realized to the source so far. Tell me please, what is better way: 1) Just to post my ideas here to discuss and let the core developers to code them the way they know. I'm not experienced in writing nor changing .texi files (as requested in http://wget.sunsite.dk/wgetdev.html if you're sending patches). So this way is convenient for me as I'm only writing my ideas and someone else is doing the job for me (probably better, but he's not as interested in new features as I am...) 2) Or should I learn http://wget.sunsite.dk/wgetdev.html well, write all the stuff myself and send it to wget-patches and hope I didn't throw away hours of my work and the patches will be accepted? I have five ideas to enhance the code. The approver will like two of them and disapprove the other two. Is it better for me then to send all my ideas in separate patchfiles? --- In short, I like my ideas and would be honoured to see them permanently in the future wget.:) But I don't want to spend some useless hours coding it my way and be rejected. If the maintainer will like such features and noone else will volunteer, I'll gladly do it. And now for something completely different: For the patient ones I have my list. I propose these features: <<>> --skip-requisites Sometimes I wish to ignore all the inline images and sounds and just want to get the linked content. Example: Gallery page with many <A..."contentXX.jpg"> and <IMG SRC..."thumbXX.jpg">. I want to get the content and ignore the thumbnails. My solution: --- wget-1.9/src/recur.c 2003-10-11 14:57:12.000000000 +0100 +++ wget-1.9a/src/recur.c 2003-12-28 21:42:58.000000000 +0100 @@ -485,6 +485,9 @@ goto out; } + if (opt.skip_requisites && upos->link_inline_p) + goto out; + /* 4. Check for parent directory. <<>> --purge-partial-downloads I know there is an advanced feature of continuing the interrupted downloads, but I hate to have the files on my disk garbled when the wget gets terminated. Therefore I think such option would be useful. My solution: Incorporates SIGQUIT, SIGTERM, SIGINT and SIGABRT interception during get_contents(), special return code -3 from this function. In this case, if (opt.dfp == NULL), the local file is deleted, such event logged and wget is exit()ed. The patch is not short and I can post it on request. <<>> --host-level=n It would be useful for me to differentiate the maximum recursion level in the case wget is spidering on the original host and when it is spaning other hosts. I may want the files in 5 levels on the original host but just to the level of 2 on the other hosts. To let -l5 and -H together can quickly try to fill my disk.:) Example: Galleries again.:) I'm downloading images which may have separate html page on the original host (gallery page with thumbs <A...> -> many HTML pages each with inline big image). I want to download the big images even when the original host links them from another host. But to let the same recursion level is dangerous (or expensive). <<>> Finish the job on SIGUSR2 "Time to die, wget!" "I'm not quite dead!" In short: First SIGUSR2 - finish current URL (cmd line and -i) and terminate Second SIGUSR2 - finish current file and terminate In both cases: log the reason at the end. This feature is meant for cases when I have not enough time to continue entire download job but there still is some time to cleanly finish part of it. IMHO, documentation does not mention signal other than SIGHUP. I think some note about interception of SIGUSR1 and SIGWINCH would be useful. <<>> Job progress info in log Sometimes I have a list of URLs to be recursively downloaded and have not enough time to finish all the job. It would be great to have the information about successully finished URLs in the log. This feature may be turned on by a switch and the output should be easily parsed - so some simple script is able to produce the NEW list of still-not-successfully-downloaded URLs from the original list and the log. The -nv log mode is better for parsing so I'd like this "DONE URL: xyz" message in -nv log too... Generally, I hope the developers are taking care of the fact, that the log output of wget is frequently being parsed by smart or even dumb scripts.:) Thank you for your time, Vlada Macek