[htdig] Segmentation fault and db.words.db corruption

[Wendt, Trevor]

Need some help troubleshooting a segmentation fault error. I've read the FAQ's and searched the mailing list archives but haven't seen a specific thread regarding this.

 

When doing a query via the web I get a 500 server error with corresponding error log entry of:

     Premature end of script headers: htsearch, referer: http://....

 

Referencing the FAQ's; this can be caused by apaches cgi-bin, perl vs. cgi handler configuration. To verify if Apache was the problem I ran htsearch from the command line with the following results:

 

---------- start snip ----------

     [... cgi-bin]$ ./htsearch -vvv 'config=int&words=test'
     word: test
     tempWords: 'test:0 '
     Boolean: 'test:0 '
     Adding algorithm exact
     Adding algorithm synonyms
     Adding algorithm endings
     initial: ''
     Fuzzy on: test
        exact test
        synonyms
        endings
     searchWords: 'test:0 '
     LogicalWords: test
     Pattern: test
     Highlight Pattern: test
     perform_push @0: test
     score: push @0
     term:factor
     Segmentation fault

---------- end snip ----------

 

This Segmentation fault produced the corresponding errors:

 

---------- start snip ----------

     Premature end of script headers: htsearch
     [Thu Aug 12 10:38:24 2004] [error] [client 170.231.200.136] WordDB: CDB___memp_cmpr_read:

     unable to uncompress page at pgno = 116
     [Thu Aug 12 10:38:24 2004] [error] [client 170.231.200.136] WordDB: PANIC: Input/output error
     [Thu Aug 12 10:38:24 2004] [error] [client 170.231.200.136] DB_RUNRECOVERY: Fatal error,

     run database recovery

---------- end snip ----------

 

Upon seeing "Word DB: PANIC" messages I moved the previous version (I keep the previous versions of all db.'s around in case a run fails) to replace "db.words.db". This corrected my problems running the same "test" query through the web and via command line.

 

---------- start snip ----------

     [... cgi-bin]$ ./htsearch -vvv 'config=int&words=test'
     word: test
     tempWords: 'test:0 '
     Boolean: 'test:0 '
     Adding algorithm exact
     Adding algorithm synonyms
     Adding algorithm endings
     initial: ''
     Fuzzy on: test
        exact test
        synonyms
        endings
     searchWords: 'test:0 '
     LogicalWords: test
     Pattern: test
     Highlight Pattern: test
     perform_push @0: test
     score: push @0
     term:factor
     http://....htm
        base_score 0.00128509 date_score 0 backlink_score 0.016
        score 0.0171374(0.0171374), maxScore -1.79769e+308, minScore 1.79769e+308
     Set maxScore = score
     Set minScore = score
     http://....doc 
        base_score 9.88533e-05 date_score 0 backlink_score 0.3
        score 0.26244(0.26244), maxScore 0.0171374, minScore 0.0171374
     Set maxScore = score

     (...many, many, more....)

---------- end snip ----------

 

It seems that when running a new dig the db.words.db isn't completed properly or is being corrupted. I utilize the rundig.sh script (version updated for ht://Dig 3.2.0b3 by Geoff) and get an email report upon completion. The report looks fine and I've noted no system errors during a run. Some words can be queried by the DB fine but others will produce the problems above.

 

We rundig three times a week, always from scratch. Our db.words.db is 138mb. I have a working and non-working version of the db at this time.

 

System: Running HTDIG 3.2.0b6 on RedHat 9.0.1.EL / Apache/2.0.46 / OpenSSL 0.9.7a; Running against RedHat 9.0.1.EL / Apache/2.0.47 / OpenSSL 0.9.6b

 

Any thoughts on why this is happening so randomly and how to prevent/correct? Suggestions more than welcome.

 

Thanks.