Hi all. I see there is a problem spidering forums like phpBB and electrifiedpenguin . Because these forums return a Session ID, it occurr that htdig spidering the forum pages will get more than one SID. The result is that 1. the same forum page is indexed more than one time 2. the amount of CPU and time used for indexing is very large.
Take a look at the log above.... Thank you. Paolo 217.168.237.106 - - [14/Oct/2002:01:06:54 +0200] "GET /forum/index.php HTTP/1.0" 200 35398 "http://www.ir3ip.net/forum/"; "htdig/3.1.5 217.168.237.106 - - [14/Oct/2002:01:06:55 +0200] "GET /forum/search.php HTTP/1.0" 200 19754 "http://www.ir3ip.net/forum/"; "htdig/3.1.5 217.168.237.106 - - [14/Oct/2002:01:06:57 +0200] "GET /forum/faq.php HTTP/1.0" 200 51949 "http://www.ir3ip.net/forum/"; "htdig/3.1.5 (ro 217.168.237.106 - - [14/Oct/2002:01:07:00 +0200] "GET /forum/memberlist.php HTTP/1.0" 200 22715 "http://www.ir3ip.net/forum/"; "htdig/3. 217.168.237.106 - - [14/Oct/2002:01:07:02 +0200] "GET /forum/index.php?sid=6923db608dd988b9167c2464278dcffb HTTP/1.0" 200 35398 "http:/ 217.168.237.106 - - [14/Oct/2002:01:07:04 +0200] "GET /forum/faq.php?sid=6923db608dd988b9167c2464278dcffb HTTP/1.0" 200 51949 "http://w ..... another dozen of lines with the same sid was removed.... after few minutes, several dozens of access with another sid: 217.168.237.106 - - [14/Oct/2002:01:09:19 +0200] "GET /forum/index.php?sid=27619eca3a821c36bbfe3222b99f62aa HTTP/1.0" 200 35398 "http://www.ir3ip.net/forum/viewforum.php?f=12"; "htdig/3.1.5 217.168.237.106 - - [14/Oct/2002:01:09:21 +0200] "GET /forum/faq.php?sid=27619eca3a821c36bbfe3222b99f62aa HTTP/1.0" 200 51949 "http://www.ir3ip.net/forum/viewforum.php?f=12"; "htdig/3.1.5 (r 217.168.237.106 - - [14/Oct/2002:01:09:23 +0200] "GET /forum/search.php?sid=27619eca3a821c36bbfe3222b99f62aa HTTP/1.0" 200 19754 "http://www.ir3ip.net/forum/viewforum.php?f=12"; "htdig/3.1.5 217.168.237.106 - - [14/Oct/2002:01:09:26 +0200] "GET /forum/memberlist.php?sid=27619eca3a821c36bbfe3222b99f62aa HTTP/1.0" 200 22715 "http://www.ir3ip.net/forum/viewforum.php?f=12"; "htdig/3 ------------------------------------------------------- This sf.net email is sponsored by: viaVerio will pay you up to $1,000 for every account that you consolidate with us. http://ad.doubleclick.net/clk;4749864;7604308;v? http://www.viaverio.com/consolidator/osdn.cfm _______________________________________________ htdig-dev mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/htdig-dev
