Hi
I have a problem with htdig indexing a virtual host on one of my web
servers - the other virtual hosts work fine. The server is a Sparc
running Debian Slink and HTDig 3.1.5. Under the virtual domain
http://www.ltbp.org, only the main page gets indexed. We have tried
everything we can think of, but without any joy.
I'd be grateful if you could shed any ideas on this.
I've included the htdig.conf file for the ltbp domain and the index output.
Thanks, Paul.
=========================> start of htdig.conf <======================
To: [EMAIL PROTECTED]
Subject: htdig.conf
From: root <[EMAIL PROTECTED]>
Date: Sun, 17 Dec 2000 19:06:39 +0000
#
# Example config file for ht://Dig.
#
# This configuration file is used by all the programs that make up ht://Dig.
# Please refer to the attribute reference manual for more details on what
# can be put into this file. (http://www.htdig.org/confindex.html)
# Note that most attributes have very reasonable default values so you
# really only have to add attributes here if you want to change the defaults.
#
# What follows are some of the common attributes you might want to change.
#
#
# Specify where the database files need to go. Make sure that there is
# plenty of free disk space available for the databases. They can get
# pretty big.
#
database_dir: /SERVER/HTDig/db
#
# This specifies the URL where the robot (htdig) will start. You can specify
# multiple URLs here. Just separate them by some whitespace.
# The example here will cause the ht://Dig homepage and related pages to be
# indexed.
# You could also index all the URLs in a file like so:
# start_url: `${common_dir}/start.url`
#
start_url: http://www.ltbp.org/
###start_url: /SERVER/Hosts/LTBP/
#
# This attribute limits the scope of the indexing process. The default is to
# set it to the same as the start_url above. This way only pages that are on
# the sites specified in the start_url attribute will be indexed and it will
# reject any URLs that go outside of those sites.
#
# Keep in mind that the value for this attribute is just a list of string
# patterns. As long as URLs contain at least one of the patterns it will be
# seen as part of the scope of the index.
#
###limit_urls_to: ${start_url} HTM
#limit_urls_to: ${start_url} HTM
limit_normalized: ${start_url} HTM
#
# If there are particular pages that you definately do NOT want to index, you
# can use the exclude_urls attribute. The value is a list of string patterns.
# If a URL matches any of the patterns, it will NOT be indexed. This is
# useful to exclude things like virtual web trees or database accesses. By
# default, all CGI URLs will be excluded. (Note that the /cgi-bin/ convention
# may not work on your web server. Check the path prefix used on your web
# server.)
#
exclude_urls: /cgi-bin/ /Boon .cgi
#
# Since ht://Dig does not (and cannot) parse every document type, this
# attribute is a list of strings (extensions) that will be ignored during
# indexing. These are *only* checked at the end of a URL, whereas
# exclude_url patterns are matched anywhere.
#
bad_extensions: .wav .gz .z .sit .au .zip .tar .hqx .exe .com .gif \
.jpg .jpeg .aiff .class .map .ram .tgz .bin .rpm .mpg .mov .avi
#
# The string htdig will send in every request to identify the robot. Change
# this to your email address.
#
maintainer: [EMAIL PROTECTED]
#
# The excerpts that are displayed in long results rely on stored information
# in the index databases. The compiled default only stores 512 characters of
# text from each document (this excludes any HTML markup...) If you plan on
# using the excerpts you probably want to make this larger. The only concern
# here is that more disk space is going to be needed to store the additional
# information. Since disk space is cheap (! :-)) you might want to set this
# to a value so that a large percentage of the documents that you are going
# to be indexing are stored completely in the database. At SDSU we found
# that by setting this value to about 50k the index would get 97% of all
# documents completely and only 3% was cut off at 50k. You probably want to
# experiment with this value.
# Note that if you want to set this value low, you probably want to set the
# excerpt_show_top attribute to false so that the top excerpt_length characters
# of the document are always shown.
#
max_head_length: 10000
#
# To limit network connections, ht://Dig will only pull up to a certain limit
# of bytes. This prevents the indexing from dying because the server keeps
# sending information. However, several FAQs happen because people have files
# bigger than the default limit of 100KB. This sets the default a bit higher.
# (see <http://www.htdig.org/FAQ.html> for more)
#
max_doc_size: 2000000
#
# Most people expect some sort of excerpt in results. By default, if the
# search words aren't found in context in the stored excerpt, htsearch shows
# the text defined in the no_excerpt_text attribute:
# (None of the search words were found in the top of this document.)
# This attribute instead will show the top of the excerpt.
#
no_excerpt_show_top: true
#
# Depending on your needs, you might want to enable some of the fuzzy search
# algorithms. There are several to choose from and you can use them in any
# combination you feel comfortable with. Each algorithm will get a weight
# assigned to it so that in combinations of algorithms, certain algorithms get
# preference over others. Note that the weights only affect the ranking of
# the results, not the actual searching.
# The available algorithms are:
# exact
# endings
# metaphone
# prefix
# soundex
# synonyms
# By default only the "exact" algorithm is used with weight 1.
# Note that if you are going to use the endings, metaphone, soundex,
# or synonyms algorithms, you will need to run htfuzzy to generate
# the databases they use.
#
search_algorithm: exact:1 synonyms:0.5 endings:0.1
#
# The following are the templates used in the builtin search results
# The default is to use compiled versions of these files, which produces
# slightly faster results. However, uncommenting these lines makes it
# very easy to change the format of search results.
# See <http://www.htdig.org/hts_templates.html for more details.
#
# template_map: Long long ${common_dir}/long.html \
# Short short ${common_dir}/short.html
# template_name: long
#
# The following are used to change the text for the page index.
# The defaults are just boring text numbers. These images spice
# up the result pages quite a bit. (Feel free to do whatever, though)
#
next_page_text: <img src="/numbers/buttonr.gif" border="0"
align="middle" width="30" height="30" alt="next">
no_next_page_text:
prev_page_text: <img src="/numbers/buttonl.gif" border="0"
align="middle" width="30" height="30" alt="prev">
no_prev_page_text:
page_number_text: '<img src="/numbers/button1.gif" border="0"
align="middle" width="30" height="30" alt="1">' \
'<img src="/numbers/button2.gif" border="0"
align="middle" width="30" height="30" alt="2">' \
'<img src="/numbers/button3.gif" border="0"
align="middle" width="30" height="30" alt="3">' \
'<img src="/numbers/button4.gif" border="0"
align="middle" width="30" height="30" alt="4">' \
'<img src="/numbers/button5.gif" border="0"
align="middle" width="30" height="30" alt="5">' \
'<img src="/numbers/button6.gif" border="0"
align="middle" width="30" height="30" alt="6">' \
'<img src="/numbers/button7.gif" border="0"
align="middle" width="30" height="30" alt="7">' \
'<img src="/numbers/button8.gif" border="0"
align="middle" width="30" height="30" alt="8">' \
'<img src="/numbers/button9.gif" border="0"
align="middle" width="30" height="30" alt="9">' \
'<img src="/numbers/button10.gif" border="0"
align="middle" width="30" height="30" alt="10">'
#
# To make the current page stand out, we will put a border arround the
# image for that page.
#
no_page_number_text: '<img src="/numbers/button1.gif" border="2"
align="middle" width="30" height="30" alt="1">' \
'<img src="/numbers/button2.gif" border="2"
align="middle" width="30" height="30" alt="2">' \
'<img src="/numbers/button3.gif" border="2"
align="middle" width="30" height="30" alt="3">' \
'<img src="/numbers/button4.gif" border="2"
align="middle" width="30" height="30" alt="4">' \
'<img src="/numbers/button5.gif" border="2"
align="middle" width="30" height="30" alt="5">' \
'<img src="/numbers/button6.gif" border="2"
align="middle" width="30" height="30" alt="6">' \
'<img src="/numbers/button7.gif" border="2"
align="middle" width="30" height="30" alt="7">' \
'<img src="/numbers/button8.gif" border="2"
align="middle" width="30" height="30" alt="8">' \
'<img src="/numbers/button9.gif" border="2"
align="middle" width="30" height="30" alt="9">' \
'<img src="/numbers/button10.gif" border="2"
align="middle" width="30" height="30" alt="10">'
star_blank /numbers/star.gif
star_image /numbers/star.gif
# local variables:
# mode: text
# eval: (if (eq window-system 'x) (progn (setq font-lock-keywords
(list '("^#.*" . font-lock-keyword-face) '("^[a-zA-Z][^ :]+" .
font-lock-function-name-face) '("[+$]*:" . font-lock-comment-face) ))
(font-lock-mode)))
# end:
######
###### Additions
######
local_default_doc: default.html default.htm index.html index.htm
INDEX.HTM INDEX.HTML
local_urls: http://www.ltbp.org/=/SERVER/Hosts/LTBP/
allow_virtual_hosts: true
case_sensitive: false
max_hop_count: 4
common_url_parts: HTM
local_urls_only: true
create_url_list: true
url_log: /tmp/htdig.progress
valid_extensions: .html .htm .shtml .HTM .HTML
logging: true
------------------> end of htdig.conf <-----------------------
----------------> start of index output<----------------------
To: [EMAIL PROTECTED]
Subject: output
From: root <[EMAIL PROTECTED]>
Date: Sun, 17 Dec 2000 19:07:01 +0000
1:0:http://www.ltbp.org/
New server: www.ltbp.org, 80
Trying local files
tried local file /SERVER/Hosts/LTBP/robots.txt
pushed
pick: www.ltbp.org, # servers = 1
0:0:0:http://www.ltbp.org/: Trying local files
tried local file /SERVER/Hosts/LTBP/default.html
tried local file /SERVER/Hosts/LTBP/default.htm
tried local file /SERVER/Hosts/LTBP/index.html
tried local file /SERVER/Hosts/LTBP/index.htm
found existing file /SERVER/Hosts/LTBP/INDEX.HTM
Read 8192 from document
Read 4498 from document
Read a total of 12690 bytes
title: ltbp.org - Lake Tanganyika Biodiversity Project
META Description: The Lake Tanganyika Biodiversity Project is a UNDP
-funded Global Environmental Facility project aimed at helping the
riparian states to produce an effective and sustainable system for
managing and conserving the biodiversity of Lake Tanganyika into the
foreseeable future.
A tag: pos = 2, position = ="SITEMAP.HTM">
A tag: pos = 2, position = ="HELP.HTM">
A tag: pos = 2, position = ="DLOAD.HTM">
A tag: pos = 2, position = ="COMMENT.HTM">
A tag: pos = 2, position = ="COMMENT.HTM">
A tag: pos = 2, position = ="FINDEX.HTM">
image: http://www.ltbp.org/gifs/frflag.gif
image: http://www.ltbp.org/gifs/nuv3.gif
image: http://www.ltbp.org/gifs/spacer4.gif
A tag: pos = 2, position = ="OVIEW.HTM">
A tag: pos = 2, position = ="OVIEW.HTM">
A tag: pos = 2, position = ="OVIEW.HTM">
A tag: pos = 2, position = ="OVIEW.HTM">
A tag: pos = 2, position = ="PART.HTM">
image: http://www.ltbp.org/gifs/spacer4.gif
A tag: pos = 2, position = ="REGOFF.HTM">
A tag: pos = 2, position = ="REGOFF.HTM">
A tag: pos = 2, position = ="BJM.HTM">
A tag: pos = 2, position = ="DSM.HTM">
A tag: pos = 2, position = ="DSM.HTM">
A tag: pos = 2, position = ="DSM.HTM">
A tag: pos = 2, position = ="KGM.HTM">
A tag: pos = 2, position = ="MPU.HTM">
A tag: pos = 2, position = ="UVI.HTM">
image: http://www.ltbp.org/gifs/spacer4.gif
A tag: pos = 2, position = ="PDD.HTM">
A tag: pos = 2, position = ="PDDGEN.HTM">
A tag: pos = 2, position = ="PDDGEN.HTM">
A tag: pos = 2, position = ="PDDPROG.HTM">
A tag: pos = 2, position = ="PDDPROG.HTM">
A tag: pos = 2, position = ="PDDPROG.HTM">
A tag: pos = 2, position = ="PDDMM.HTM">
A tag: pos = 2, position = ="PDDMM.HTM">
A tag: pos = 2, position = ="PDDMM.HTM">
A tag: pos = 2, position = ="NLETTER.HTM">
A tag: pos = 2, position = ="PDDBL.HTM">
A tag: pos = 2, position = ="PDDBL.HTM">
image: http://www.ltbp.org/gifs/spacer4.gif
A tag: pos = 2, position = ="CAL.HTM">
A tag: pos = 2, position = ="CAL98.HTM">
A tag: pos = 2, position = ="CAL99.HTM">
A tag: pos = 2, position = ="CAL00.HTM">
image: http://www.ltbp.org/gifs/spacer4.gif
A tag: pos = 2, position = ="GALLERY.HTM">
A tag: pos = 2, position = ="GALLERY.HTM">
A tag: pos = 2, position = ="PHTUVI.HTM">
A tag: pos = 2, position = ="PHTUVI.HTM">
A tag: pos = 2, position = ="PHTWED.HTM">
A tag: pos = 2, position = ="PHTWED.HTM">
A tag: pos = 2, position = ="PHTWWED.HTM">
A tag: pos = 2, position = ="PHTETC.HTM">
A tag: pos = 2, position = ="http://www.gg.rhbnc.ac.uk/cedar/">
image: http://www.ltbp.org/gifs/rhbnc.gif
image: http://www.ltbp.org/gifs/s3.gif
image: http://www.ltbp.org/gifs/spacer5.gif
image: http://www.ltbp.org/gifs/boat1.jpg
image: http://www.ltbp.org/gifs/text2.gif
image: http://www.ltbp.org/gifs/s4.gif
image: http://www.ltbp.org/gifs/spacer4.gif
A tag: pos = 2, position = ="PROG.HTM">
A tag: pos = 2, position = ="BIOSS.HTM">
A tag: pos = 2, position = ="EESS.HTM">
A tag: pos = 2, position = ="EESS.HTM">
A tag: pos = 2, position = ="FPSS.HTM">
A tag: pos = 2, position = ="FPSS.HTM">
A tag: pos = 2, position = ="TANGIS.HTM">
A tag: pos = 2, position = ="TANGIS.HTM">
A tag: pos = 2, position = ="TANGIS.HTM">
A tag: pos = 2, position = ="PSS.HTM">
A tag: pos = 2, position = ="SSS.HTM">
A tag: pos = 2, position = ="SESS.HTM">
A tag: pos = 2, position = ="TRAIN.HTM">
image: http://www.ltbp.org/gifs/spacer4.gif
A tag: pos = 2, position = ="PROC.HTM">
A tag: pos = 2, position = ="LGLCON.HTM">
A tag: pos = 2, position = ="LGLCON.HTM">
A tag: pos = 2, position = ="LATSAP.HTM">
A tag: pos = 2, position = ="LATSAP.HTM">
A tag: pos = 2, position = ="LATSAP.HTM">
image: http://www.ltbp.org/gifs/spacer4.gif
A tag: pos = 2, position = ="MTG.HTM">
A tag: pos = 2, position = ="MTG.HTM">
A tag: pos = 2, position = ="MTG.HTM">
image: http://www.ltbp.org/gifs/spacer4.gif
A tag: pos = 2, position = ="PROJ.HTM">
A tag: pos = 2, position = ="NYANZA.HTM">
A tag: pos = 2, position = ="NYANZA.HTM">
A tag: pos = 2, position = ="UVIREN.HTM">
A tag: pos = 2, position = ="UVIREN.HTM">
A tag: pos = 2, position = ="TANESCO.HTM">
A tag: pos = 2, position = ="TANESCO.HTM">
A tag: pos = 2, position = ="TANESCO.HTM">
A tag: pos = 2, position = ="WED.HTM">
A tag: pos = 2, position = ="WED.HTM">
A tag: pos = 2, position = ="WED.HTM">
image: http://www.ltbp.org/gifs/spacer4.gif
A tag: pos = 2, position = ="./priv/ADMIN.HTM">
A tag: pos = 2, position = ="ADMIN.HTM">
A tag: pos = 2, position = ="ADMIN.HTM">
image: http://www.ltbp.org/gifs/nuv1.gif
A tag: pos = 2, position = ="AUTHOR.HTM">
A tag: pos = 2, position = ="COMMENT.HTM">
A tag: pos = 2, position = ="http://www.worldlakes.org">
image: http://www.ltbp.org/gifs/lake.gif
A tag: pos = 2, position = ="http://hitometer.netscape.com">
image: http://i50.netscape.com/c.cgi?b4677650$3704694
size = 12690
pick: www.ltbp.org, # servers = 1
htdig: Run complete
htdig: 1 server seen:
htdig: www.ltbp.org:80 1 document
htmerge: Sorting...
htmerge: Merging...
htmerge: 100:ordination
htmerge: Total word count: 168
0/http://www.ltbp.org/
htmerge: Total documents: 1
htmerge: Total doc db size (in K): 12
----------------> end of index output <----------------------
--
----------------------------------
"The Macintosh isn't a computer...
it's a way of life." Don Rittner.
o~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~o
Paul Broome
Centre for Developing Areas Research
Department of Geography
Royal Holloway, University of London
Egham, Surrey, TW20 OEX, UK
Tel (Work):+44 (0)178 444 3574
Fax: :+44 (0)178 447 2386
Voice Mail:+44 (0)207 681 2867
http://www.appleonline.net/pbroome
o~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~o
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
List archives: <http://www.htdig.org/mail/menu.html>
FAQ: <http://www.htdig.org/FAQ.html>
