On Mon, 16 Aug 1999, William Freman wrote:
> memory and disk space...what about a a quad PIII/550 Xenon with
> 2GB RAM with a 5TB RAID array on a T3. somthing such as that would take
> away a good amount of the theoretical hindermets. with those out of the
> way, would it be possible to index the web?
I still wouldn't recommend it. It's only recently that we've been
receiving feedback on scaling to huge (i.e. 500,000+ URLs) indexes. In
particular, the 3.1.x series requires the htmerge phase and that requires
sorting the word database. For even modest-sized databases, it can take an
enormous amount of RAM to sort.
The 3.2 development code should help with some of these performance
bottlenecks.
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/
------------------------------------
To unsubscribe from the htdig3-dev mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the SUBJECT of the message.
