I'm using my own build of htdig 3.2.0b6 with SSL, and crawling a site running IIS and SSL. htdig gets a lot of the data, but misses quite a few pages. I get a lot of the following error for various pages referenced by .asp pages:

Not found: https://intranet.affymetrix.com/cis/FAQ/remote_access_faqs.html Ref: https://intranet.affymetrix.com/faq.asp

Details below. As you can see, the IIS server is giving me a 403 error. I can grab that page with no problem from the same host using wget.

Hugh

Here is that URL using htdig -vvvvv.

228:42:2:https://intranet.affymetrix.com/cis/FAQ/remote_access_faqs.html: Making HTTPS request on https://intranet.affymetrix.com/cis/FAQ/remote_access_faqs.html
Making a HEAD call before the GET
Try to get through to host intranet.affymetrix.com (port 443)
97 - Open of the connection ok
Assigned the remote host intranet.affymetrix.com
Assigned the port 443
Header line: HTTP/1.1 403 Access Forbidden
Header line: Server: Microsoft-IIS/5.0
Header line: Date: Tue, 09 Nov 2004 23:21:27 GMT
Header line: Connection: close
Header line: Content-Length: 4126
Header line: Content-Type: text/html
No modification time returned: assuming now
Retrieving document /cis/FAQ/remote_access_faqs.html on host: intranet.affymetrix.com:443
Http version : HTTP/1.1
Server : HTTP/1.1
Status Code : 403
Reason : Access Forbidden
Access Time : Tue, 09 Nov 2004 23:21:27 GMT
Modification Time : Tue, 09 Nov 2004 23:21:55 GMT
Content-type : text/html
Connection : close
Persistent connection: not accepted
Body not retrieved
97 - Connection closed (No persistent connection)
Request time: 0 secs


--
Hugh Caley | Unix Systems Administrator | CIS
AFFYMETRIX, INC. | 6550 Vallejo St. Ste 100 | Emeryville, CA 94608
Tel: 510-428-8537 | [EMAIL PROTECTED]



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
ht://Dig general mailing list: <[EMAIL PROTECTED]>
ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html
List information (subscribe/unsubscribe, etc.)
https://lists.sourceforge.net/lists/listinfo/htdig-general

Reply via email to