Okay, did only one page and a robots.txt. See for
yourself about modification dates etc.
robots.txt:
The request is with 1 jan 1970 and you get back:
21 sep 2001. But the status-report says 1 jan 1970.
startpage:
The request is with 1 jan 1970 and you get back:
01 Mar 2000. But the status-report says 1 jan 1970.
So where do I start to look?
---------------------begin-------------------------
ht://dig Start Time: Fri Sep 21 11:44:37 2001
1:1:http://crytonii.st.hhs.nl/
New server: crytonii.st.hhs.nl, 80
- Persistent connections: enabled
- HEAD before GET: disabled
- Timeout: 30
- Connection space: 1
- Max Documents: -1
- TCP retries: 1
- TCP wait time: 5
Trying to retrieve robots.txt file
Creating an HtHTTPBasic object
Making HTTP request on http://crytonii.st.hhs.nl/robots.txt
Try to get through to host crytonii.st.hhs.nl (port 80)
1 - Open of the connection ok
Assigning the server (crytonii.st.hhs.nl) to the TCP connection
Assigned the remote host crytonii.st.hhs.nl
Assigning the port (80) to the TCP connection
Assigned the port 80
Connecting via TCP to (crytonii.st.hhs.nl:80)
New connection open successfully
Request
GET /robots.txt HTTP/1.1
Host: crytonii.st.hhs.nl
User-Agent: htdig
If-Modified-Since: Thu, 01 Jan 1970 01:00:00 MET
Header line: HTTP/1.1 200 OK
Header line: Date: Fri, 21 Sep 2001 08:46:20 GMT
Header line: Server: Apache/1.3.12 (Unix) (Red Hat/Linux)
Header line: Last-Modified: Fri, 21 Sep 2001 08:46:05 GMT
Header line: ETag: "1f00e-22-3baafe4d"
Discarded header line: ETag: "1f00e-22-3baafe4d"
Header line: Accept-Ranges: bytes
Discarded header line: Accept-Ranges: bytes
Header line: Content-Length: 34
Header line: Content-Type: text/plain
Retrieving document /robots.txt on host: crytonii.st.hhs.nl:80
Http version : HTTP/1.1
Server : HTTP/1.1
Status Code : 200
Reason : OK
Access Time : Thu, 01 Jan 1970 01:00:00 MET
Modification Time : Thu, 01 Jan 1970 01:00:00 MET
Content-type : text/plain
Persistent connection: would be accepted
Reading the body of the response
Contents:
User-agent: *
Disallow: /foo.html
Connection stays up ... (Persistent connection)
Request time: 0 secs
Contents:
User-agent: *
Disallow: /foo.html
Content Type: text/plain
Content Length: 34
Modification Time: 1970-01-01 01:00:00 MET
Parsing robots.txt file using myname = htdig
Robots.txt line: User-agent: *
Found 'user-agent' line: *
Robots.txt line: Disallow: /foo.html
Found 'disallow' line: /foo.html
Pattern: /foo.html
pushed
pick: crytonii.st.hhs.nl, # servers = 1
> crytonii.st.hhs.nl supports HTTP persistent connections (infinite)
0:2:0:http://crytonii.st.hhs.nl/: Creating an HtHTTPBasic object
Making HTTP request on http://crytonii.st.hhs.nl/
Try to get through to host crytonii.st.hhs.nl (port 80)
2 - Open of the connection ok
Assigning the server (crytonii.st.hhs.nl) to the TCP connection
Assigned the remote host crytonii.st.hhs.nl
Assigning the port (80) to the TCP connection
Assigned the port 80
Connecting via TCP to (crytonii.st.hhs.nl:80)
New connection open successfully
Request
GET / HTTP/1.1
Host: crytonii.st.hhs.nl
User-Agent: htdig
If-Modified-Since: Thu, 01 Jan 1970 01:00:00 MET
Header line: HTTP/1.1 200 OK
Header line: Date: Fri, 21 Sep 2001 08:46:20 GMT
Header line: Server: Apache/1.3.12 (Unix) (Red Hat/Linux)
Header line: Last-Modified: Wed, 01 Mar 2000 18:37:44 GMT
Header line: ETag: "1f008-9cf-38bd6378"
Discarded header line: ETag: "1f008-9cf-38bd6378"
Header line: Accept-Ranges: bytes
Discarded header line: Accept-Ranges: bytes
Header line: Content-Length: 2511
Header line: Content-Type: text/html
Retrieving document / on host: crytonii.st.hhs.nl:80
Http version : HTTP/1.1
Server : HTTP/1.1
Status Code : 200
Reason : OK
Access Time : Thu, 01 Jan 1970 01:00:00 MET
Modification Time : Thu, 01 Jan 1970 01:00:00 MET
Content-type : text/html
Persistent connection: would be accepted
Reading the body of the response
------------------------snip----------------------
--jesse
--------------------------------------------------------------------
J. op den Brouw Johanna Westerdijkplein 75
Haagse Hogeschool 2521 EN DEN HAAG
Faculty of Engeneering Netherlands
Electrical Engeneering +31 70 4458936
-------------------- [EMAIL PROTECTED] --------------------
Linux - because reboots are for hardware changes
_______________________________________________
htdig-dev mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/htdig-dev
