On Wed, 6 Oct 2004, Christophe Duval wrote:
pick: www.colloc.minefi.gouv.fr, # servers = 1www.colloc.minefi.gouv.fr supports HTTP persistent connections (infinite)0:2:0:http://www.colloc.minefi.gouv.fr/: Making HTTP request on http://www.colloc.minefi.gouv.fr/ Header line: HTTP/1.1 500 Server Error Header line: Server: Netscape-Enterprise/3.6 Header line: Date: Wed, 06 Oct 2004 11:49:20 GMT Header line: Content-length: 305 Header line: Content-type: text/html Request time: 0 secs not found pick: www.colloc.minefi.gouv.fr, # servers = 1www.colloc.minefi.gouv.fr supports HTTP persistent connections (infinite)ht://dig End Time: Wed Oct 6 13:50:20 2004 Deleted, not found: ID: 2 URL: http://www.colloc.minefi.gouv.fr/
There is something a bit strange about the configuration of this web server. While HEAD requests appear to work for most pages, the server errors out and returns a 500 status if a HEAD request is made on /
This is not an htdig specific issue and can be duplicated by using something like telnet or nc to connect to the server and manually feed
in the request.
You should be able to get around this problem by setting the head_before_get attribute to false.
http://www.htdig.org/dev/htdig-3.2/attrs.html#head_before_get
This may degrade performance a bit depending on the nature of the sites that you are indexing.
Jim
------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ ht://Dig general mailing list: <[EMAIL PROTECTED]> ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-general
