According to Katherine Porter:
> Gilles - 3.1.5 has no problems with it at all. The web site is
> www.dol.net, if you'd like to test it yourself.
OK, I tried it, and sure enough I was able to reproduce the problem.
I'm at a bit of a loss to explain why the server is behaving this way,
though. As far as I can tell, it's not because of a bug in the HTTP
code in htdig.
The problem seems to be that the server doesn't consistently return
headers in response to a GET request. With the 3.1.6 development code,
it doesn't return headers after the first request (for the robots.txt
file), which doesn't matter, but it does for the subsequent requests.
With the 3.2.0b4 development code, it doesn't seem to return headers
for either the GET request for robots.txt nor for "GET /", unless both
persistent_connections and head_before_get are set to true. It turns out
that head_before_get has no effect if persistent_connections is false.
With both of these attributes set to true, htdig only makes the HEAD
request for the first file (robots.txt), and then sees that the server is
HTTP/1.0 so it falls back to that mode (turns off persistent connections)
but that first HEAD request seems to be enough to make the server give
headers for subsequent GET requests.
I don't know what's different about the 3.1.x code that makes the server
decide to give headers after the 2nd and subsequent GET requests, but
that is the observed behaviour.
In any case, this sure looks like a server bug to me. I'm no HTTP
expert, but a server that doesn't consistently give HTTP headers before
the documents it serves can't be behaving according to the specs.
At least you have a workaround, as long as you know for sure it will
work consistently after that first HEAD request.
--
Gilles R. Detillieux E-mail: <[EMAIL PROTECTED]>
Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba Phone: (204)789-3766
Winnipeg, MB R3E 3J7 (Canada) Fax: (204)789-3930
_______________________________________________
htdig-dev mailing list
[EMAIL PROTECTED]
http://lists.sourceforge.net/lists/listinfo/htdig-dev
