The last complete generate (back in Sept of '03) took 4 days. The _next_ version will be using the mharc project technqiues to chop it into bits.
Interestingly IIS, and Windows 2000 handle the large directory just fine, and responsiveness is not a problem. The system is PRIMARILY accessed by a Namazu search, which ALSO works fine, with sub-second response to any query.
Rick Boatright
Earl Hood wrote:
On March 15, 2004 at 16:49, "Lyle Schofield" wrote:
Side question -- are there any practical limits to the number of
messages MHonArc can convert into an archive? I've been testing in
batches of 350, and it only takes a few minutes on my laptop. But, if I
try to dump thousands of messages in there will I hit some limit of what
can be processed? Thanks in advance.
Scaling can be a problem if your resources are limitied. When MHonArc processes an archive, all the meta-information of the archive is loaded, so archive updates for large archives will slow down over time. Therefore, doing real-time updates (i.e. add messages right as they are received) of large archives should not be done to avoid bottleneck problems.
The general solution for dealing with a large archive is to break it down into smaller archives denoted by fixed time periods (like each week, month, or year). The mharc project, <http://www.mhonarc.org/mharc/> provides such capabilites, and other users have developed their own (search the list archives).
If you are archives will grow to an indefinite size, then I highly recommend going with the smaller sub-archive approach.
--ewh
P.S. I have heard of users that have had up to 60,000 messages in a single archive. But I think you need some decent hardware and RAM to effectively work with such a beast.