On Mon, 1 Apr 2002, Ronnie Koch wrote: > > Problem resolved. I removed the "Disallow: /index" from robots.txt. > > FWIW > > ht://Dig 3.2.0b3
The problem with zillion refreshes is/was caused by this version of htdig calling itself "htdig" in the headers and not "htdig/.....". The test was for the first 6 digits "htdig/" - local mod to wwwoffle fixed problem for now. > wwwoffled version 2.7a > > Thanks - once I realised that */search/index had hidden links the rest was > easy :-) > > > > Ronnie > > On 31 Mar 2002, Andrew M. Bishop wrote: > > > Ronnie Koch <[EMAIL PROTECTED]> writes: > > > > > I have been using wwwoffle for a couple of years and only recently decided > > > to get the htdig search operational - with minimal success. I upgraded to > > > 2.7a, from source, this week from 2.6d on RedHat 7.2 (Intel). The original > > > cache was xfered from another machine that ran 2.5x and was converted to > > > work with 2.6d (months ago). > > > > > > 0. I can only search the WWWOFFLE FAQ & Welcome pages with htdig. > > > > I use htdig with WWWOFFLE (I don't use the other search options > > although I have tested them). I don't have any problems, so I believe > > that it works in the default installation (my installation is not very > > different from the default). > > > > What are the error messages that you get from htdig? > > No errors - just nothing indexed. > > > > 1. I changed htdig-full.conf to point to a existing page in the cache and > > > got a lot of good stuff pluss a zillion refresh requests that I did not > > > need. It kind of confirmed that the basic search is working. > > > > The htdig configuration file is set up to search to a depth of 4. This > > is why the first search index page is start4.html so that the depth 4 > > pages are the cached ones and no further links are followed. > > > > There should not be any refresh requests since WWWOFFLE should detect > > that it is htdig that is requesting the page and not store the > > request. > > Noticed in the source - to rusty to realistically follow the logic though. > > > > 2. The localhost:8080/search/index/ page is empty (link from > > > localhost:8080/search/start4.html) > > > > It may look empty, but if you examine the source code you should find > > that it is not really empty. > > Lynx cleared that one up. > > > -------------------------------------------------------------------- > Ronnie Koch | Snailmail: POBox 60124 > Bsc Eng (Elec)(Pret) MSAIEE | Pierre van Ryneveld 0045 > Tel: +27-(0)83-2270548 | South Africa > Email: [EMAIL PROTECTED] > http://www.infinityio.co.za > -------------------------------------------------------------------- > > > > Regards Ronnie -------------------------------------------------------------------- Ronnie Koch | Snailmail: POBox 60124 Bsc Eng (Elec)(Pret) MSAIEE | Pierre van Ryneveld 0045 Tel: +27-(0)83-2270548 | South Africa Email: [EMAIL PROTECTED] http://www.infinityio.co.za --------------------------------------------------------------------
