Changelog 
---------
Date   : July 7th, 2004 10:44
Author : D. van de Vliet (david@uselab.com)
About  : Patch for httools/htmerge.cc

The groundwork for this patch was layed out by Lorenzo Campedelli.
(http://www.mail-archive.com/htdig3-dev@htdig.org/msg02780.html)
His patch was broken because WordSearchDescription is replaced by WordCursor 

DESCRIPTION
This patch seems to do the trick in reducing memory load by using the callback
interface currently in use in htdig3.2.0b6

TESTCASES
I ran two testcases on debian system
 1. merging two identical copies
 2. merging 82 small (unique) search indexes into one big index

ad. 1:
merged database was exactly the same size as a single copy

ad. 2: 
merged database was about the same size as the sum of the size of all copies.

IMPROVEMENTS
stats for merging 82 unique search indexes (each about 6Mb) into one big 
index (490Mb) on a 700Mhz PII 500Mb:
 Before patch: memory usage : 1% > 90% (resulting in swapping and slowdown)
 After patch : memory usage : 1% - 4%


BRIEF DESCRIPTION OF CHANGES
- Added one class for recordkeeping and two callback function 
  (written by Lorenzo Campedelli)
    class CallbackData
    int OverrideCallback(WordList * wl, WordDBCursor &, 
                         const WordReference * w, Object & d )
    int DeleteCallback(WordList * wl, WordDBCursor &, 
                       const WordReference * w, Object & d )

- Rewritten code merging words to use the new callback interface
  This resulted in a dramatic reduction in memory usage.
