Dual Xeon 2.0ghz, Hyperthreading enabled, 2 gigs of ram and storing index on a raid-5 mirror of 8 250 gig ultra133 ide drives.
250 threads configured Sustain about 7100kbits/sec Average 630,000 pages/hour. At this rate i should be able to add about 8-10 million pages a day and have the time to run analyze db and generate some indexes. All from a single server :) I've gotten permission to use however much incoming bandwidth i need so i'm going to try and beef up to 2-3 servers running crawl and i'll try and post some updates then. ------------------------------------------------------- This SF.net email is sponsored by: The Robotic Monkeys at ThinkGeek For a limited time only, get FREE Ground shipping on all orders of $35 or more. Hurry up and shop folks, this offer expires April 30th! http://www.thinkgeek.com/freeshipping/?cpg=12297 _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
