On 1 Oct 2001 at 17:23, Geoff Hutchison wrote: > On Mon, 1 Oct 2001, Dan Langille wrote: > > > etc. I suspect the lack of a trailing / is causing the redirect > > as mentioned above and then htdig never retries the > > redirected URL. Does that sound plausible? > > Sure, this is probably why those URLs are being added incorrectly. But the > fundamental question remains--redirects should be followed, but you're not > seeing this. So what happened after the redirect?
Does this extract help (the output log is about 1MB): Rejected: URL not in the limits! url rejected: (level 1)http://www.hypermail.org/ size = 139953 pick: unixathome.org, # servers = 1 5:5:1:http://unixathome.org/adsl/archives/2001_05: Retrieval command for http://unixathome.org/adsl/archives/2001_05: GE T /adsl/archives/2001_05 HTTP/1.0 User-Agent: htdig/3.1.5 ([EMAIL PROTECTED]) Referer: http://unixathome.org/adsl/ Host: unixathome.org Header line: HTTP/1.1 301 Moved Permanently Header line: Date: Mon, 01 Oct 2001 16:41:45 GMT Header line: Server: Apache/1.3.20 (Unix) mod_gzip/1.3.19.1a PHP/4.0.6 mod_ssl/2.8.4 OpenSSL/0.9.6 Header line: Location: http://www.unixathome.org/adsl/archives/2001_05/ Header line: Connection: close Header line: Content-Type: text/html; charset=iso-8859-1 Header line: returnStatus = 3 redirect redirect: http://www.unixathome.org/adsl/archives/2001_05/ Rejected: URL not in the limits!pick: unixathome.org, # servers = 1 6:6:1:http://unixathome.org/adsl/archives/2001_06: Retrieval command for http://unixathome.org/adsl/archives/2001_06: GE T /adsl/archives/2001_06 HTTP/1.0 User-Agent: htdig/3.1.5 ([EMAIL PROTECTED]) Referer: http://unixathome.org/adsl/ Host: unixathome.org Header line: HTTP/1.1 301 Moved Permanently Header line: Date: Mon, 01 Oct 2001 16:41:45 GMT Header line: Server: Apache/1.3.20 (Unix) mod_gzip/1.3.19.1a PHP/4.0.6 mod_ssl/2.8.4 OpenSSL/0.9.6 Header line: Location: http://www.unixathome.org/adsl/archives/2001_06/ Header line: Connection: close Header line: Content-Type: text/html; charset=iso-8859-1 Header line: returnStatus = 3 redirect redirect: http://www.unixathome.org/adsl/archives/2001_06/ Rejected: URL not in the limits!pick: unixathome.org, # servers = 1 7:7:1:http://unixathome.org/adsl/archives/2001_07: Retrieval command for http://unixathome.org/adsl/archives/2001_07: GE T /adsl/archives/2001_07 HTTP/1.0 User-Agent: htdig/3.1.5 ([EMAIL PROTECTED]) Referer: http://unixathome.org/adsl/ Host: unixathome.org -- Dan Langille The FreeBSD Diary - http://freebsddiary.org/ - practical examples _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
