On Fri, 12 Jan 2001, Evelio Martinez wrote:Correct.> htdig is ignoring the files with pdf and doc extension.
By this, I assume you mean they're not indexed.
I have run bin/htdig -i -vvv -s | tee /tmp/ht and the 3 .doc and 2 .pdf files that areTry running htdig -vvv and take a look at what happens when it encounters
a link to a PDF file. Does it reject the link? Or does it get to the link
and try to index it later?
under /home/httpd/html does not have any reference in the debug file /tmp/ht.
Is this normal?
I have not seen anything apparently wrong. Do you?If it's the former, then one of your limits is set incorrectly. (e.g.
bad_extensions, valid_extensions, exclude_urls, limit_urls_to ...)
I attached the htdig.conf
If I execute /usr/local/bin/catdoc /home/httpd/html/*.doc I can see a reasonable-looking output.If it's the latter, then make sure you can run a .doc or a .pdf through
the external converter itself and get reasonable-looking output.
Any idea?
Thanks
-- Evelio Martínez Testanet. Dept. desarrollo software. Av. Reino de Valencia, 15 - 5 46005 Valencia (Spain) Tel: +34 96 395 90 00 Fax: +34 96 316 23 19
# # Example config file for ht://Dig. # # This configuration file is used by all the programs that make up ht://Dig. # Please refer to the attribute reference manual for more details on what # can be put into this file. (http://www.htdig.org/confindex.html) # Note that most attributes have very reasonable default values so you # really only have to add attributes here if you want to change the defaults. # # What follows are some of the common attributes you might want to change. #
#
# Specify where the database files need to go. Make sure that there is
# plenty of free disk space available for the databases. They can get
# pretty big.
#
database_dir: /opt/www/htdig/db
#
# This specifies the URL where the robot (htdig) will start. You can specify
# multiple URLs here. Just separate them by some whitespace.
# The example here will cause the ht://Dig homepage and related pages to be
# indexed.
# You could also index all the URLs in a file like so:
# start_url: `${common_dir}/start.url`
#
start_url: http://correo.testanet.com/ \
http://correo.testanet.com/akopia/
# http://correo.testanet.com/manual \
# http://correo.testanet.com/tareas \
# http://correo.testanet.com/tienda \
# http://correo.testanet.com/phpshop \
# http://correo.testanet.com/icons \
# http://correo.testanet.com/pruebas \
# http://correo.testanet.com/ps_image \
# http://correo.testanet.com/freetrade \
# http://correo.testanet.com/phpfwgen \
# http://correo.testanet.com/akopia \
# http://correo.testanet.com/construct
# http://www.testanet.com/
#
# This attribute limits the scope of the indexing process. The default is to
# set it to the same as the start_url above. This way only pages that are on
# the sites specified in the start_url attribute will be indexed and it will
# reject any URLs that go outside of those sites.
#
# Keep in mind that the value for this attribute is just a list of string
# patterns. As long as URLs contain at least one of the patterns it will be
# seen as part of the scope of the index.
#
limit_urls_to: ${start_url}
#
# If there are particular pages that you definately do NOT want to index, you
# can use the exclude_urls attribute. The value is a list of string patterns.
# If a URL matches any of the patterns, it will NOT be indexed. This is
# useful to exclude things like virtual web trees or database accesses. By
# default, all CGI URLs will be excluded. (Note that the /cgi-bin/ convention
# may not work on your web server. Check the path prefix used on your web
# server.)
#
exclude_urls: /cgi-bin/ .cgi
#
# Since ht://Dig does not (and cannot) parse every document type, this
# attribute is a list of strings (extensions) that will be ignored during
# indexing. These are *only* checked at the end of a URL, whereas
# exclude_url patterns are matched anywhere.
#
bad_extensions: .wav .gz .z .sit .au .zip .tar .hqx .exe .com .gif \
.jpg .jpeg .aiff .class .map .ram .tgz .bin .rpm .mpg .mov .avi
#
# The string htdig will send in every request to identify the robot. Change
# this to your email address.
#
maintainer: [EMAIL PROTECTED]
#
# The excerpts that are displayed in long results rely on stored information
# in the index databases. The compiled default only stores 512 characters of
# text from each document (this excludes any HTML markup...) If you plan on
# using the excerpts you probably want to make this larger. The only concern
# here is that more disk space is going to be needed to store the additional
# information. Since disk space is cheap (! :-)) you might want to set this
# to a value so that a large percentage of the documents that you are going
# to be indexing are stored completely in the database. At SDSU we found
# that by setting this value to about 50k the index would get 97% of all
# documents completely and only 3% was cut off at 50k. You probably want to
# experiment with this value.
# Note that if you want to set this value low, you probably want to set the
# excerpt_show_top attribute to false so that the top excerpt_length characters
# of the document are always shown.
#
max_head_length: 10000
#
# To limit network connections, ht://Dig will only pull up to a certain limit
# of bytes. This prevents the indexing from dying because the server keeps
# sending information. However, several FAQs happen because people have files
# bigger than the default limit of 100KB. This sets the default a bit higher.
# (see <http://www.htdig.org/FAQ.html> for more)
#
max_doc_size: 200000
#
# Most people expect some sort of excerpt in results. By default, if the
# search words aren't found in context in the stored excerpt, htsearch shows
# the text defined in the no_excerpt_text attribute:
# (None of the search words were found in the top of this document.)
# This attribute instead will show the top of the excerpt.
#
no_excerpt_show_top: true
#
# Depending on your needs, you might want to enable some of the fuzzy search
# algorithms. There are several to choose from and you can use them in any
# combination you feel comfortable with. Each algorithm will get a weight
# assigned to it so that in combinations of algorithms, certain algorithms get
# preference over others. Note that the weights only affect the ranking of
# the results, not the actual searching.
# The available algorithms are:
# exact
# endings
# metaphone
# prefix
# soundex
# synonyms
# By default only the "exact" algorithm is used with weight 1.
# Note that if you are going to use the endings, metaphone, soundex,
# or synonyms algorithms, you will need to run htfuzzy to generate
# the databases they use.
#
search_algorithm: exact:1 synonyms:0.5 endings:0.1
#
# The following are the templates used in the builtin search results
# The default is to use compiled versions of these files, which produces
# slightly faster results. However, uncommenting these lines makes it
# very easy to change the format of search results.
# See <http://www.htdig.org/hts_templates.html for more details.
#
# template_map: Long long ${common_dir}/long.html \
# Short short ${common_dir}/short.html
# template_name: long
#
# The following are used to change the text for the page index.
# The defaults are just boring text numbers. These images spice
# up the result pages quite a bit. (Feel free to do whatever, though)
#
next_page_text: <img src="/htdig/buttonr.gif" border="0" align="middle"
width="30" height="30" alt="next">
no_next_page_text:
prev_page_text: <img src="/htdig/buttonl.gif" border="0" align="middle"
width="30" height="30" alt="prev">
no_prev_page_text:
page_number_text: '<img src="/htdig/button1.gif" border="0" align="middle"
width="30" height="30" alt="1">' \
'<img src="/htdig/button2.gif" border="0" align="middle"
width="30" height="30" alt="2">' \
'<img src="/htdig/button3.gif" border="0" align="middle"
width="30" height="30" alt="3">' \
'<img src="/htdig/button4.gif" border="0" align="middle"
width="30" height="30" alt="4">' \
'<img src="/htdig/button5.gif" border="0" align="middle"
width="30" height="30" alt="5">' \
'<img src="/htdig/button6.gif" border="0" align="middle"
width="30" height="30" alt="6">' \
'<img src="/htdig/button7.gif" border="0" align="middle"
width="30" height="30" alt="7">' \
'<img src="/htdig/button8.gif" border="0" align="middle"
width="30" height="30" alt="8">' \
'<img src="/htdig/button9.gif" border="0" align="middle"
width="30" height="30" alt="9">' \
'<img src="/htdig/button10.gif" border="0" align="middle"
width="30" height="30" alt="10">'
#
# To make the current page stand out, we will put a border arround the
# image for that page.
#
no_page_number_text: '<img src="/htdig/button1.gif" border="2" align="middle"
width="30" height="30" alt="1">' \
'<img src="/htdig/button2.gif" border="2" align="middle"
width="30" height="30" alt="2">' \
'<img src="/htdig/button3.gif" border="2" align="middle"
width="30" height="30" alt="3">' \
'<img src="/htdig/button4.gif" border="2" align="middle"
width="30" height="30" alt="4">' \
'<img src="/htdig/button5.gif" border="2" align="middle"
width="30" height="30" alt="5">' \
'<img src="/htdig/button6.gif" border="2" align="middle"
width="30" height="30" alt="6">' \
'<img src="/htdig/button7.gif" border="2" align="middle"
width="30" height="30" alt="7">' \
'<img src="/htdig/button8.gif" border="2" align="middle"
width="30" height="30" alt="8">' \
'<img src="/htdig/button9.gif" border="2" align="middle"
width="30" height="30" alt="9">' \
'<img src="/htdig/button10.gif" border="2" align="middle"
width="30" height="30" alt="10">'
# local variables:
# mode: text
# eval: (if (eq window-system 'x) (progn (setq font-lock-keywords (list '("^#.*" .
font-lock-keyword-face) '("^[a-zA-Z][^ :]+" . font-lock-function-name-face) '("[+$]*:"
. font-lock-comment-face) )) (font-lock-mode)))
# end:
external_parsers: application/msword->text/html
/opt/www/htdig/scripts/doc2html.pl \
application/pdf->text/html /opt/www/htdig/scripts/doc2html.pl
\
application/rtf->text/html /usr/local/scripts/doc2html.pl \
text/rtf->text/html /usr/local/scripts/doc2html.pl \
application/postscript->text/html
/usr/local/scripts/doc2html.pl \
application/Wordperfect5.1->text/html
/usr/local/scripts/doc2html.pl \
application/msexcel->text/html /usr/local/scripts/doc2html.pl \
application/vnd.ms-excel->text/html
/usr/local/scripts/doc2html.pl \
application/vnd.ms-powerpoint->text/html
/usr/local/scripts/doc2html.pl
------------------------------------ To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this. List archives: <http://www.htdig.org/mail/menu.html> FAQ: <http://www.htdig.org/FAQ.html>
