Hello, Thanks for your response, this is the http.conf file:
# $Id: htdig.conf,v 1.6 2007/06/27 13:12:14 migonzalez Exp $ # # Example config file for ht://Dig. # # This configuration file is used by all the programs that make up ht://Dig. # Please refer to the attribute reference manual for more details on what # can be put into this file. (See http://www.htdig.org/confindex.html or # locally at file:/usr/share/doc/htdig-doc/htmlconfindex.html) # Note that most attributes have very reasonable default values so you # really only have to add attributes here if you want to change the defaults. # # What follows are some of the common attributes you might want to change. # # # Specify where the database files need to go. Make sure that there is # plenty of free disk space available for the databases. They can get # pretty big. # database_dir: /var/lib/htdig # # This specifies the URL where the robot (htdig) will start. You can specify # multiple URLs here. Just separate them by some whitespace. # The example here will cause the ht://Dig homepage and related pages to be # indexed. # You could also index all the URLs in a file like so: # start_url: `${common_dir}/start.url` # start_url: http://localhost/manuales2/manuales2.html http://localhost/proyectos/proyectos.html http://localhost/indices # # This attribute limits the scope of the indexing process. The default is to # set it to the same as the start_url above. This way only pages that are on # the sites specified in the start_url attribute will be indexed and it will # reject any URLs that go outside of those sites. # # Keep in mind that the value for this attribute is just a list of string # patterns. As long as URLs contain at least one of the patterns it will be # seen as part of the scope of the index. # limit_urls_to: ${start_url} http://petete/ # # If there are particular pages that you definitely do NOT want to index, you # can use the exclude_urls attribute. The value is a list of string patterns. # If a URL matches any of the patterns, it will NOT be indexed. This is # useful to exclude things like virtual web trees or database accesses. By # default, all CGI URLs will be excluded. (Note that the /cgi-bin/ convention # may not work on your web server. Check the path prefix used on your web # server.) # exclude_urls: /cgi-bin/ .cgi /images # # Since ht://Dig does not (and cannot) parse every document type, this # attribute is a list of strings (extensions) that will be ignored during # indexing. These are *only* checked at the end of a URL, whereas # exclude_url patterns are matched anywhere. # bad_extensions: .png .wav .gz .z .bz2 .sit .au .zip .tar .hqx .exe .com \ .gif .jpg .jpeg .aiff .class .map .ram .tgz .bin .rpm .mpg .mov .avi .css # # The string htdig will send in every request to identify the robot. Change # this to your email address. # maintainer: [EMAIL PROTECTED] # # The excerpts that are displayed in long results rely on stored information # in the index databases. The compiled default only stores 512 characters of # text from each document (this excludes any HTML markup...) If you plan on # using the excerpts you probably want to make this larger. The only concern # here is that more disk space is going to be needed to store the additional # information. Since disk space is cheap (! :-)) you might want to set this # to a value so that a large percentage of the documents that you are going # to be indexing are stored completely in the database. At SDSU we found # that by setting this value to about 50k the index would get 97% of all # documents completely and only 3% was cut off at 50k. You probably want to # experiment with this value. # Note that if you want to set this value low, you probably want to set the # excerpt_show_top attribute to false so that the top excerpt_length characters # of the document are always shown. # max_head_length: 10000 # # To limit network connections, ht://Dig will only pull up to a certain limit # of bytes. This prevents the indexing from dying because the server keeps # sending information. However, several FAQs happen because people have files # bigger than the default limit of 100KB. This sets the default a bit higher. # (see <http://www.htdig.org/FAQ.html> for more) # max_doc_size: 200000 # # Most people expect some sort of excerpt in results. By default, if the # search words aren't found in context in the stored excerpt, htsearch shows # the text defined in the no_excerpt_text attribute: # (None of the search words were found in the top of this document.) # This attribute instead will show the top of the excerpt. # no_excerpt_show_top: true # Miqui locale: es_ES case_sensitive: false # Factores # Por defecto: #title_factor: 100 #keywords_factor: 100 #meta_description_factor: 50 #heading_factor_1: 5 #heading_factor_2: 4 #heading_factor_3: 3 #heading_factor_4: 1 #heading_factor_5: 1 #heading_factor_6: 0 #text_factor: 1 title_factor: 100 keywords_factor: 500 meta_description_factor: 50 heading_factor: 10 text_factor: 1 # # Depending on your needs, you might want to enable some of the fuzzy search # algorithms. There are several to choose from and you can use them in any # combination you feel comfortable with. Each algorithm will get a weight # assigned to it so that in combinations of algorithms, certain algorithms get # preference over others. Note that the weights only affect the ranking of # the results, not the actual searching. # The available algorithms are: # accents # exact # endings # metaphone # prefix # soundex # substring # synonyms # By default only the "exact" algorithm is used with weight 1. # Note that if you are going to use the endings, metaphone, soundex, accents, # or synonyms algorithms, you will need to run htfuzzy to generate # the databases they use. # search_algorithm: exact:1 prefix:0.5 accents:0.2 # # The following are the templates used in the builtin search results # The default is to use compiled versions of these files, which produces # slightly faster results. However, uncommenting these lines makes it # very easy to change the format of search results. # See <http://www.htdig.org/hts_templates.html> for more details. # # template_map: Long long ${common_dir}/long.html \ # Short short ${common_dir}/short.html # template_name: long # # The following are used to change the text for the page index. # The defaults are just boring text numbers. These images spice # up the result pages quite a bit. (Feel free to do whatever, though) # next_page_text: <img src="/htdig/buttonr.gif" border="0" align="middle" width="30" height="30" alt="next"> no_next_page_text: prev_page_text: <img src="/htdig/buttonl.gif" border="0" align="middle" width="30" height="30" alt="prev"> no_prev_page_text: page_number_text: '<img src="/htdig/button1.gif" border="0" align="middle" width="30" height="30" alt="1">' \ '<img src="/htdig/button2.gif" border="0" align="middle" width="30" height="30" alt="2">' \ '<img src="/htdig/button3.gif" border="0" align="middle" width="30" height="30" alt="3">' \ '<img src="/htdig/button4.gif" border="0" align="middle" width="30" height="30" alt="4">' \ '<img src="/htdig/button5.gif" border="0" align="middle" width="30" height="30" alt="5">' \ '<img src="/htdig/button6.gif" border="0" align="middle" width="30" height="30" alt="6">' \ '<img src="/htdig/button7.gif" border="0" align="middle" width="30" height="30" alt="7">' \ '<img src="/htdig/button8.gif" border="0" align="middle" width="30" height="30" alt="8">' \ '<img src="/htdig/button9.gif" border="0" align="middle" width="30" height="30" alt="9">' \ '<img src="/htdig/button10.gif" border="0" align="middle" width="30" height="30" alt="10">' # # To make the current page stand out, we will put a border around the # image for that page. # no_page_number_text: '<img src="/htdig/button1.gif" border="2" align="middle" width="30" height="30" alt="1">' \ '<img src="/htdig/button2.gif" border="2" align="middle" width="30" height="30" alt="2">' \ '<img src="/htdig/button3.gif" border="2" align="middle" width="30" height="30" alt="3">' \ '<img src="/htdig/button4.gif" border="2" align="middle" width="30" height="30" alt="4">' \ '<img src="/htdig/button5.gif" border="2" align="middle" width="30" height="30" alt="5">' \ '<img src="/htdig/button6.gif" border="2" align="middle" width="30" height="30" alt="6">' \ '<img src="/htdig/button7.gif" border="2" align="middle" width="30" height="30" alt="7">' \ '<img src="/htdig/button8.gif" border="2" align="middle" width="30" height="30" alt="8">' \ '<img src="/htdig/button9.gif" border="2" align="middle" width="30" height="30" alt="9">' \ '<img src="/htdig/button10.gif" border="2" align="middle" width="30" height="30" alt="10">' minimum_word_length: 2 extra_words_characters: _:- allow_numbers: true bad_word_list: /etc/htdig/bad_words # # These files don't belong to /etc/htdig # synonym_db: /usr/lib/htdig/synonyms.db endings_root2word_db: /usr/lib/htdig/root2word.db endings_word2root_db: /usr/lib/htdig/word2root.db # # image_url_prefix is the relative url for the images htdig uses. # This defaults to "/doc/htdig/images". To get the images from another dir # if you've turned off the /doc alias, you'll need to modify the .html files # in /etc/htdig as well as this setting. # image_url_prefix: /htdig # # external parser for msword, postscript and pdf files # # If you really want to parse pdf, ps and doc files, make sure that the # max_doc_size value above is set to a larger value than your largest file # of this type. If it is smaller, the indexing might hang # #external_parsers: application/msword /usr/share/htdig/parse_doc.pl \ # application/postscript /usr/share/htdig/parse_doc.pl \ # application/pdf /usr/share/htdig/parse_doc.pl # # special Debian variable for PDF documents # # If you want to parse PDF documents, set the (htdig variable) pdf_parser # to the wrapper script /usr/bin/htdig-pdfparser (which is default). Using # the debian_pdf_parser variable the actual pdf parser can be controlled. # Make sure you set the external_parsers option too. # Recognized options are: acrobat, xpdf # #debian_pdf_parser: acrobat search_rewrite_rules: http://localhost/(.*) http://petete/\\1 # local variables: # mode: text # eval: (if (eq window-system 'x) (progn (setq font-lock-keywords (list '("^#.*" . font-lock-keyword-face) '("^[a-zA-Z][^ :]+" . font-lock-function-name-face) '("[+$]*:" . font-lock-comment-face) )) (font-lock-mode))) # end: ############################################################# Regards, Marcelo 2008/3/4, [EMAIL PROTECTED] <[EMAIL PROTECTED]>: > Marcelo, > The limit_urls_to: > line looks a bit iffy - is this exactly how it looks in your config file? > (It might be easier if you attached the whole file) > > Given that the number of documents is increasing, it rather looks as though > there is some limit on how far the indexing goes each time, which is then > being built upon. > > Regards, > Mike > > > >-----Original Message----- > >From: [EMAIL PROTECTED] > >[mailto:[EMAIL PROTECTED] On Behalf > >Of Marcelo García > >Sent: Tuesday, March 04, 2008 3:29 PM > >To: [email protected] > >Subject: [htdig] Problem with htdig indexation > > > >Hello, > > > >I am a new user of htdig and I have a problem with indexation, > >My server is runing with Ubuntu 7, with htdig 3.2.0b6, the > >documents to index are in one server only. The server name is petete. > > > >The number of documents found by htdig change every time I run > >rundig (rundig -a -s) and because of this some documents are > >not found when we search them. > > > >Example: > >First execution of rundig -a -s: > >htdig: localhost:80 24 documents > > > >Second: > >htdig: 2 servers seen: > >htdig: localhost:80 24 documents > >htdig: petete:80 37 documents > > > >3. > >htdig: 2 servers seen: > >htdig: localhost:80 24 documents > >htdig: petete:80 326 documents > > > >4. > >htdig: 2 servers seen: > >htdig: localhost:80 24 documents > >htdig: petete:80 409 documents > > > >5. > >htdig: 2 servers seen: > >htdig: localhost:80 24 documents > >htdig: petete:80 438 documents > >This is the max number of documents found, after some time > >(one day more or less), if we repeat the process we have the > >same results. > > > >In htdig.conf we have (parameters that I think could affect to > >indexation): > >start_url: http://localhost/manuales2/manuales2.html > >http://localhost/proyectos/proyectos.html http://localhost/indices > > > >limit_urls_to: ${start_url} http://petete/ > > > >exclude_urls: /cgi-bin/ .cgi /images > > > >search_rewrite_rules: http://localhost/(.*) http://petete/\\1 > > > >Please, can you help me with this problem? > > > >Thanks in advance. > > > > >--------------------------------------------------------------- > >---------- > >This SF.net email is sponsored by: Microsoft Defy all > >challenges. Microsoft(R) Visual Studio 2008. > >http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > >_______________________________________________ > >ht://Dig general mailing list: <[email protected]> > >ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html > >List information (subscribe/unsubscribe, etc.) > >https://lists.sourceforge.net/lists/listinfo/htdig-general > > > ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ ht://Dig general mailing list: <[email protected]> ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-general
