Re: -nv option; printing out infos via stderr[http://bugs.debian.org/141323]
Ian Abbott [EMAIL PROTECTED] writes: On 5 Apr 2002 at 18:17, Noel Koethe wrote: Will this be changed so the user could use -nv with /dev/null and get only errors or warnings displayed? So what I think you want is for any log message tagged as LOG_VERBOSE (verbose information) or LOG_NONVERBOSE (basic information) in the source to go to stdout when no log file has been specified and the `-O -' option has not been used and for everything else to go to stderr? That change sounds dangerous. Current Wget output doesn't really have a concept of errors that would be really separate from other output; it only operates on the level of verbosity. This was, of course, a bad design decision, and I agree that steps need to be taken to change it. I'm just not sure that this is the right step. For one, I don't know of any utility that splits its output this way. It is true that many programs print their output on stdout and errors to stderr, but Wget's log output is hardly the actual, programmatic, output of the program. That can only be the result of `-O -'. Suddenly `wget -o X' is no longer equivalent to `wget 2x', which violates the Principle of Least Surprise.
Re: -nv option; printing out infos via stderr [http://bugs.debian.org/141323]
On 9 Apr 2002 at 10:34, Hrvoje Niksic wrote: Ian Abbott [EMAIL PROTECTED] writes: On 5 Apr 2002 at 18:17, Noel Koethe wrote: Will this be changed so the user could use -nv with /dev/null and get only errors or warnings displayed? So what I think you want is for any log message tagged as LOG_VERBOSE (verbose information) or LOG_NONVERBOSE (basic information) in the source to go to stdout when no log file has been specified and the `-O -' option has not been used and for everything else to go to stderr? That change sounds dangerous. Current Wget output doesn't really have a concept of errors that would be really separate from other output; it only operates on the level of verbosity. This was, of course, a bad design decision, and I agree that steps need to be taken to change it. I'm just not sure that this is the right step. Neither am I, but I knocked up the patch on a whim. Suddenly `wget -o X' is no longer equivalent to `wget 2x', which violates the Principle of Least Surprise. Perhaps we just need a --log-level=N option: Level 0: output just the LOG_ALWAYS messages. Level 1: output the above and LOG_NOTQUIET messages. Level 2: output the above and LOG_NONVERBOSE messages. Level 3: output the above and LOG_VERBOSE messages. The --verbose option would be equivalent to --log-level=3 (the default). The --non-verbose option would be equivalent to --log-level=2. The --quiet option would be equivalent to --log-level=1. Noel would specify --log-level=1 to get the output he wants. How does that sound?
Re: -nv option; printing out infos via stderr [http://bugs.debian.org/141323]
Guys
For what its worth:-
Most of my code logs to a level using something like
command --verbose 10 .|| -v10 etc.
In my code, I call a trace routine that simply checks the verbose level
with the call level and log if it is -ge.
EG:-
// in simple terms.
tr ( level, msg ) {
if ( level -ge g_level ) {
print_whatever ( $msg )
}
}
tr 12 MSG:some relevant message
.
tr 2 ERROR:some error message
..
etc
All level 0 are printed regardless, IE g_level defaults to 0.
The print_whatever is a complex function that automatically opens a log
file etc
and prints ERROR to stderr and all others to stdout.
I am sure there must be thousands of variations on this theme.
My contribution as a long time wget user. Thanks Guys.
Regards
Ian
Re: Satellite [NetGain 2000] [Corruption]
Justin Piszcz [EMAIL PROTECTED] writes: --12:12:21-- ftp://war:*password*@0.0.0.0:21//iso/file.iso = `iso/file.iso' == CWD not required. == PASV ... done.== RETR file.iso ... done. Length: 737,402,880 24% [] 180,231,952 37.40K/s ETA 4:02:27 13:31:51 (37.40 KB/s) - Data connection: Connection timed out; Data transfer aborted. Retrying. This causes corruption in the file. I need to try a client which supports rollback I guess. It seems so. But I really find it strange that there would be an FTP server that fails in this scenario. I've only heard of such proxy failures, and that's not applicable to your case.
Re: Malformed status line error
Torsten Fellhauer -iXpoint- #429 [EMAIL PROTECTED] writes: when connecting to a FTP-Server using a TrendMicro Viruswall Proxy, we get the error Malformed status line, Unfortunately, Wget is right; that status line is quite different from what HTTP mandates. The status line should be something like: HTTP/1.0 200 Ok Or, more generally: HTTP/1.x status message Instead, the TrendMicro Viruswall Proxy returns: 220 InterScan Version 3.6-Build_1166 $Date: 04/24/2001 22:13:0052$ (mucint01, dynamic, get: N, put: N): Ready That is so far from HTTP that even if Wget's parser were lenient it still wouldn't make sense out of it. Is 220 an HTTP status code? If so, which one? What version of HTTP is the proxy speaking? Someone should write to the makers of TrendMicro Viruswall Proxy and ask them to fix this bug.
Re: GNU Wget 1.5.3
Matthias Jim Knopf [EMAIL PROTECTED] writes: there is a bug (or a feature...) in the version 1.5.3 Note that the latest version of Wget is 1.8.1. I suggest you to upgrade because the new version handles URLs much better. I discovered that every doubled slash (//) is converted to a single slash (/) which might make sense for real file-paths, but which does not allow me to retrieve an url like This bug still exists in the latest version, but I plan to fix it before the next release. http://my.server/some/file?get_url=http://foo.bar/ ^^ will be converted to http:/foo.bar which is not a valid url this also does not work if the value for 'get_url' is url-encoded as it should be. 1.8.1 handles this correctly when quoted. For example: $ wget -d 'http://fly.srk.fer.hr/?foo=http%2f%2fbar/baz' DEBUG output created by Wget 1.8.1 on linux-gnu. [...] ---request begin--- GET /?foo=http%2f%2fbar/baz HTTP/1.0
Re: Malformed status line error
On Tue, 9 Apr 2002, Hrvoje Niksic wrote: Instead, the TrendMicro Viruswall Proxy returns: 220 InterScan Version 3.6-Build_1166 $Date: 04/24/2001 22:13:0052$ (mucint01, dynamic, get: N, put: N): Ready That is so far from HTTP that even if Wget's parser were lenient it still wouldn't make sense out of it. Is 220 an HTTP status code? If so, which one? What version of HTTP is the proxy speaking? That looks *so* much like a FTP server response... -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Satellite [NetGain 2000] [Corruption]
Netgain specifics are closed source and proprietrary, however the BST protocol is open. Bug in BST/TCP/NetGain, no matter which it is, the rollback option [when used] appears to fix the problem quite nicely. Alexander V. Lukyanov wrote: On Tue, Apr 09, 2002 at 02:55:25PM +0200, Hrvoje Niksic wrote: It seems so. But I really find it strange that there would be an FTP server that fails in this scenario. I've only heard of such proxy failures, and that's not applicable to your case. I think it is tcp layer bug. Does the NetGain network translation or something like that? -- Alexander.
[] / . -
¾È³çÇϽʴϱî? ÇÁ¸°ÅÍ À×Å©/Åä³Ê Àü¹®¼îÇθô À×Å©·Î´åÄÄ(www.inkro.com)ÀÔ´Ï´Ù. º» ¸ÞÀÏÀº À¥¼ÇÎÁß ¾Ë°ÔµÇ¾úÀ¸¸ç °í°´´ÔÀÇ À̸ÞÀÏ ÁÖ¼ÒÀÌ¿Ü¿¡ ¾î¶°ÇÑ Á¤º¸µµ °®°íÀÖÁö ¾Ê½À´Ï´Ù. Çã¶ô¾øÀÌ ¸ÞÀÏÀ» ¹ß¼ÛÇÏ¿© Á˼ÛÇÕ´Ï´Ù. (1ȸ¼º ¸ÞÀÏÀÔ´Ï´Ù.) »ç¹«½Ç¿¡¼ ¶Ç´Â °¡Á¤¿¡¼ ¾²½Ã´Â ÇÁ¸°ÅÍ À×Å©/Åä³Ê °¡°ÝÀÌ ºÎ´ãµÇ½ÃÁö ¾ÊÀ¸½Ê´Ï±î? À×Å©·Î´åÄÄ(www.inkro.com) ¿¡¼´Â Á¤Ç°, Àç»ý¿ÏÁ¦Ç°, ±¹»êÁ¦Ç° »Ó¸¸¾Æ´Ï¶ó °í°´´Ô²²¼ Á÷Á¢ ¸®ÇÊÇÏ¿© ¾²½Ç ¼ö ÀÖµµ·Ï ¸®ÇÊÁ¦Ç°À» ÆǸÅÇÏ°í ÀÖ½À´Ï´Ù. Á¤Ç°°ú Â÷À̾ø´Â Ç°Áú°ú Àú·ÅÇÑ °¡°ÝÀ¸·Î ÇÁ¸°ÅͼҸðÇ° ºñ¿ëÀ» ³·Ãâ ¼ö ÀÖ½À´Ï´Ù. £Ü 9,000¿ø £Ü 18,000¿ø £Ü 9,000¿ø £Ü 11,000¿ø À×Å©·Î´åÄÄ¿¡¼´Â º¸´Ù Àú·ÅÇÑ °¡°ÝÀ¸·Î ÆǸÅÇÕ´Ï´Ù. ( Á¤Ç° ÃÖÀú°¡, ¸®ÇÊÀ×Å© °¡°ÝÀÎÇÏ!! ) ȸ¿ø°¡ÀÔÈÄ ±¸¸Å½Ã¸¶´Ù ±¸¸Å¾×ÀÇ 1%¸¦ Àû¸³ÇÏ¿© µå¸³´Ï´Ù. ¸®ÇÊÁ¦Ç°Àº ¼³¸í¼¿Í ÇÔ²² ¹ß¼ÛµÇ¸ç ȨÆäÀÌÁö¿¡ ÀÚ¼¼ÇÑ ¸®Çʹæ¹ýÀ» ´ãÀº µ¿¿µ»óÀÌ ÀÖ½À´Ï´Ù. °í°´´Ô²² º¸´Ù ³ªÀº ¼ºñ½º¸¦ À§ÇØ ³ë·ÂÇÏ´Â À×Å©·Î´åÄÄ(www.inkro.com)ÀÌ µÇ°Ú½À´Ï´Ù. [ ¼ö½Å°ÅºÎ ]
Re: getting time stamp via FTP
On 8 Apr 2002 at 11:43, Urs Thuermann wrote: Please CC: any answers to my email address, since I'm not on this list. I'd like wget to get the time stamp of a file that is downloaded via FTP and to set the mtime after writing the file to the local disk. When using HTTP, this already happens, i.e. when doing a wget http://host/file the file has the same time stamp in the local file system as on the remote server, but not with FTP. FTP supports the MODTIME command to get the time stamp of a file from the server. Could wget be changed to use this? the modtime command supported by some clients uses an FTP extension (MDTM). How widely is this supported by FTP servers? Wget recently adopted use of another extension (SIZE) and has long supported another extension (REST), so it could potentially adopt other extensions if commonly used. Currently, Wget extracts the timestamp from a directory listing of the file, but that doesn't always work, as the format for the directory listing is not standardized. Ideally, I think Wget should only have to fall back on old-style directory listings as a last resort, but that will have to wait a few years for newer mechanisms to be standardized and commonly adopted (i.e. the MLST/MLSD extensions). These links may be useful: http://www.ietf.org/html.charters/ftpext-charter.html http://www.ietf.org/internet-drafts/draft-ietf-ftpext-mlst-15.txt
Re: getting time stamp via FTP
On 9 Apr 2002 at 16:52, Ian Abbott wrote: Wget recently adopted use of another extension (SIZE) and has long supported another extension (REST), so it could potentially adopt other extensions if commonly used. Correction: 'REST' is a standard FTP protocol command, not an extension.
Current download speed in progress bar
Since I implemented the progress bar, I've progressively become more
and more annoyed by the fact that the download speed it reports is the
average download speed. What I'm usually much more interested in is
the current download speed.
This patch implements this change; the current download speed is
calculated as the speed of the most recent 30 network reads. I think
this makes sense -- for very downloads, you'll get the average
spanning several seconds; for the fast ones, you'll get the average in
this fraction of a second. This is what I want -- I think.
The one remaining problem is the ETA. Based on the current speed, it
changes value wildly. Of course, over time it is generally
decreasing, but one can hardly follow it. I removed the flushing by
making sure that it's not shown more than once per second, but this
didn't fix the problem of unreliable values.
Should we revert to the average speed for ETA, or is there a smarter
way to handle it? What are other downloaders doing?
2002-04-09 Hrvoje Niksic [EMAIL PROTECTED]
* progress.c (bar_update): Maintain an array of the time it took
to perform previous 30 network reads.
(create_image): Calculate the download speed and ETA based on the
last 30 reads, not the entire download.
(create_image): Make sure that the ETA is not changed more than
once per second.
Index: src/progress.c
===
RCS file: /pack/anoncvs/wget/src/progress.c,v
retrieving revision 1.23
diff -u -r1.23 progress.c
--- src/progress.c 2001/12/10 05:31:45 1.23
+++ src/progress.c 2002/04/09 18:49:45
@@ -401,6 +401,9 @@
create_image will overflow the buffer. */
#define MINIMUM_SCREEN_WIDTH 45
+/* Number of recent packets we keep the stats for. */
+#define RECENT_ARRAY_SIZE 30
+
static int screen_width = DEFAULT_SCREEN_WIDTH;
struct bar_progress {
@@ -410,7 +413,7 @@
download finishes */
long count; /* bytes downloaded so far */
- long last_update;/* time of the last screen update. */
+ long last_screen_update; /* time of the last screen update. */
int width; /* screen width we're using at the
time the progress gauge was
@@ -420,7 +423,27 @@
signal. */
char *buffer;/* buffer where the bar image is
stored. */
- int tick;
+ int tick;/* counter used for drawing the
+ progress bar where the total size
+ is not known. */
+
+ /* The following variables (kept in a struct for namespace reasons)
+ keep track of how long it took to read recent packets. See
+ bar_update() for explanation. */
+ struct {
+long previous_time;
+long times[RECENT_ARRAY_SIZE];
+long bytes[RECENT_ARRAY_SIZE];
+int count;
+long summed_times;
+long summed_bytes;
+ } recent;
+
+ /* create_image() uses these to make sure that ETA information
+ doesn't flash. */
+ long last_eta_time; /* time of the last update to download
+ speed and ETA. */
+ long last_eta_value;
};
static void create_image PARAMS ((struct bar_progress *, long));
@@ -453,7 +476,8 @@
bar_update (void *progress, long howmuch, long dltime)
{
struct bar_progress *bp = progress;
- int force_update = 0;
+ int force_screen_update = 0;
+ int rec_index;
bp-count += howmuch;
if (bp-total_length 0
@@ -465,21 +489,75 @@
equal to the expected size doesn't abort. */
bp-total_length = bp-count + bp-initial_length;
+ /* The progress bar is supposed to display the current download
+ speed. The first version of the progress bar calculated it by
+ dividing the total amount of data with the total time needed to
+ download it. The problem with this was that stalled or suspended
+ download could unduly influence the current time. Taking just
+ the time needed to download the current packet would not work
+ either because packets arrive too fast and the varitions would be
+ too jerky.
+
+ It would be preferrable to show the speed that pertains to a
+ recent period, say over the past several seconds. But to do this
+ accurately, we would have to record all the packets received
+ during the last five seconds.
+
+ What we do instead is maintain a history of a fixed number of
+ packets. It actually makes sense if you think about it -- faster
+ downloads will have a faster response to speed changes. */
+
+ rec_index = bp-recent.count % RECENT_ARRAY_SIZE;
+ ++bp-recent.count;
+
+ /* Instead of calculating the sum of times[] and bytes[], we
+ maintain the summed quantities. To maintain each sum, we must
+ make sure that it gets increased by
Re: Current download speed in progress bar
On Tue, 9 Apr 2002, Hrvoje Niksic wrote: Should we revert to the average speed for ETA, or is there a smarter way to handle it? What are other downloaders doing? I'll grab the other part and explain what curl does. It shows a current speed based on the past five seconds, it shows an average speed during the entire transfer and it shows an ETA that is based on the average speed... -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Current download speed in progress bar
Hrvoje Niksic wrote: The one remaining problem is the ETA. Based on the current speed, it changes value wildly. Of course, over time it is generally decreasing, but one can hardly follow it. I removed the flushing by making sure that it's not shown more than once per second, but this didn't fix the problem of unreliable values. I'm often annoyed by ETA estimates that make no sense. How about showing two values -- something like: ETA at average speed: 1:05:17 ETA at current speed: 15:05 Then the user can decide which value is more meaningful. In addition, it gives feedback about the current speed versus the average. Tony
Re: Current download speed in progress bar
Maurice Cinquini [EMAIL PROTECTED] writes: I don't think using only a fraction of a second is a reliable method for estimating current bandwidth. Here are some factors that can make for a wildly varing ETAs when just looking at the last fraction of a second. - TCP slow start. - Kernel level buffering - Other network traffic That's beside the point; this was never intended to be a scientific method of determining bandwidth. All I aimed for was something more useful than dividing total bytes with total time. And for bandwidth, I'm confident that my current method is better than what was previously in place. I'm not so sure about ETA, though. I don't like apt's method of calculating the CPS only every N seconds, because -- if I'm reading it right -- it means that you see the same value for 6 seconds, and then have to wait another 6 seconds for refresh. That sucks. `links', for example, offers both average and current speed, and the latter seems to be updated pretty swiftly. Still, thanks for the suggestions. Unless I find a really cool different suggestion, I'll fall back to the previous method for ETA.
Re: Current download speed in progress bar
Daniel Stenberg [EMAIL PROTECTED] writes: On Tue, 9 Apr 2002, Hrvoje Niksic wrote: Should we revert to the average speed for ETA, or is there a smarter way to handle it? What are other downloaders doing? I'll grab the other part and explain what curl does. It shows a current speed based on the past five seconds, Does it mean that the speed doesn't change for five seconds, or that you always show the *current* speed, but relative to the last five seconds? I may be missing something, but I don't see how to efficiently implement the latter.
Re: Current download speed in progress bar
Tony Lewis [EMAIL PROTECTED] writes: I'm often annoyed by ETA estimates that make no sense. How about showing two values -- something like: ETA at average speed: 1:05:17 ETA at current speed: 15:05 The problem is that Wget is limited by what fits in one line. I'd like to keep enough space for the progress bar, so I can add no additional information.
Re: Current download speed in progress bar
Hrvoje Niksic wrote: I'll grab the other part and explain what curl does. It shows a current speed based on the past five seconds, Does it mean that the speed doesn't change for five seconds, or that you always show the *current* speed, but relative to the last five seconds? I may be missing something, but I don't see how to efficiently implement the latter. Could you keep an array of speeds that is updated once a second such that the value from six seconds ago is discarded and when the value for the second that just ended is recorded? Tony
Re: Current download speed in progress bar
Tony Lewis [EMAIL PROTECTED] writes: Could you keep an array of speeds that is updated once a second such that the value from six seconds ago is discarded and when the value for the second that just ended is recorded? Right now I'm doing that kind of trick, but for the last N reads from the network. This translates to a larger interval for slower downloads, and the other way around, which is, I think, what one would want.
Re: Current download speed in progress bar
Andre Majorel [EMAIL PROTECTED] writes: If find it very annoying when a downloader plays yoyo with the remaining time. IMHO, remaining time is by nature a long term thing and short term jitter should not cause it to go up and down. Agreed wholeheartedly, but how would you *implement* a non-jittering ETA? Do you think it makes sense the way 1.8.1 does it, i.e. to calculate the ETA from the average speed?
LAN with Proxy, no Router
Hi! I recently managed to get my big machine online using a two PC (Windows boxes) LAN. A PI is the server, running both Zonealaram and Jana under Win98. The first one a firewall, the second one a proxy programme. On my client, an Athlon 1800+ with Windows 2000 I want to work with wget and download files over http from the www. For Netscape, I need to specify the LAN IP of the server as Proxy address. Setting up LeechFTP works similarly, IE is also set up (all three work). But wget does not work the way I tried. I just basically started it, it failed (of course) and I searched the wget help and the www with google. However, the only thing that looks remotely like what I need is '' -Y on/off --proxy=on/off Turn proxy support on or off. The proxy is on by default if the appropriate environmental variable is defined. '' Could someone please tell me, what the appropriate environmental variable is and how do I change it in Windows or what else I need to do? I'd expect something like --proxy=on/off --proxy-address --proxy-user --proxy-passwd as a collection of proxy-related commands. All except --proxy-address=IP exist, so it is apparently not necessary. Kind regards Jens
