Hi all,
I have come across an interesting situation. I am trying to debug some speed
issues with a HTTP server (apt-proxy) and was trying to use wget to just grab
the file after each iteration to see what changes there were in speed.
Unfortunately, wget would get most of the file, and then just block, at the end
of the file, for 15 minutes. Figuring that this was a timeout problem of some
form I had a look, and sure enough, there it was, sitting on a select to read
more information. Why would it be doing that I thought?
Digging through the code I came across the following:
In retr.c (get_contents) there is a test
while (!use_expected || (*len < expected))
// read and process
This "read and process" then calls select with the timeout of 15minutes
(configurable, but defaulted to 900 seconds), and then reads from the socket if
there is anything to read. OK, so we only go into this loop when use_expected is
false or when *len is less than expected. <tickety-tick> Turns out that for my
server in this test mode, use_expected is false, and so wget just tries to read
the whole stream... which it already has.
OK, so why is use_expected false? Turns out that there is an assumption that
when the server sends through the header:
Connection: close
wget assumes that we cannot use use_expected, and that the server will close the
connection immediately. Looking at section 14.10 of RFC 2616 I don't see where
it says that the server needs to close the connection, or when. It just says
that the connection will be closed, and that you cannot use it again. In any
case, I have a proposed fix which you see attached to this mail.
Basically, instead of using keep_alive as the use_length parameter, I calculate
it based on whether or not the length has been passed to us.
Hope this helps,
Paul
Only in wget-1.9.1.fix: .libs
Only in wget-1.9.1.fix: Makefile
Only in wget-1.9.1.fix: config.log
Only in wget-1.9.1.fix: config.status
Only in wget-1.9.1.fix/doc: Makefile
Only in wget-1.9.1.fix/doc: texi2pod.pl
Only in wget-1.9.1.fix/doc: wget.1
Only in wget-1.9.1.fix/doc: wget.pod
Only in wget-1.9.1.fix: libtool
Only in wget-1.9.1.fix/po: Makefile
Only in wget-1.9.1.fix/po: Makefile.in
Only in wget-1.9.1.fix/po: POTFILES
Only in wget-1.9.1.fix/src: .libs
Only in wget-1.9.1.fix/src: Makefile
Only in wget-1.9.1.fix/src: cmpt.o
Only in wget-1.9.1.fix/src: config.h
Only in wget-1.9.1.fix/src: connect.o
Only in wget-1.9.1.fix/src: convert.o
Only in wget-1.9.1.fix/src: cookies.o
Only in wget-1.9.1.fix/src: ftp-basic.o
Only in wget-1.9.1.fix/src: ftp-ls.o
Only in wget-1.9.1.fix/src: ftp-opie.o
Only in wget-1.9.1.fix/src: ftp.o
Only in wget-1.9.1.fix/src: gen-md5.o
Only in wget-1.9.1.fix/src: gen_sslfunc.o
Only in wget-1.9.1.fix/src: hash.o
Only in wget-1.9.1.fix/src: headers.o
Only in wget-1.9.1.fix/src: host.o
Only in wget-1.9.1.fix/src: html-parse.o
Only in wget-1.9.1.fix/src: html-url.o
diff --recursive --unified wget-1.9.1/src/http.c wget-1.9.1.fix/src/http.c
--- wget-1.9.1/src/http.c 2003-10-15 01:32:15.000000000 +0200
+++ wget-1.9.1.fix/src/http.c 2004-08-31 23:30:09.000000000 +0200
@@ -1533,7 +1533,7 @@
/* Get the contents of the document. */
hs->res = get_contents (sock, fp, &hs->len, hs->restval,
(contlen != -1 ? contlen : 0),
- &rbuf, keep_alive, &hs->dltime);
+ &rbuf, (contlen != -1), &hs->dltime);
if (hs->res >= 0)
CLOSE_FINISH (sock);
Only in wget-1.9.1.fix/src: http.o
Only in wget-1.9.1.fix/src: init.o
Only in wget-1.9.1.fix/src: log.o
Only in wget-1.9.1.fix/src: main.o
Only in wget-1.9.1.fix/src: netrc.o
Only in wget-1.9.1.fix/src: progress.o
Only in wget-1.9.1.fix/src: rbuf.o
Only in wget-1.9.1.fix/src: recur.o
Only in wget-1.9.1.fix/src: res.o
Only in wget-1.9.1.fix/src: retr.o
Only in wget-1.9.1.fix/src: safe-ctype.o
Only in wget-1.9.1.fix/src: snprintf.o
Only in wget-1.9.1.fix/src: url.o
Only in wget-1.9.1.fix/src: utils.o
Only in wget-1.9.1.fix/src: version.o
Only in wget-1.9.1.fix/src: wget
Only in wget-1.9.1.fix: stamp-h
Only in wget-1.9.1.fix/util: Makefile
Only in wget-1.9.1.fix/windows: Makefile
