Actually, I am starting to think it is related to hard disks beginning to fail. We have some machines that have double or triple the load with the exact same number of tasks. One thing I am seeing is that hard disks don't just fail (ok some do), but most actually just slow down when then are starting to break down.
Dennis Kubes Doğacan Güney wrote: > Hi Dennis, > > On 7/31/07, Dennis Kubes <[EMAIL PROTECTED]> wrote: >> Is anybody doing really big indexing jobs on Nutch and Hadoop, say 50M >> or more and seeing indexer timeout jobs? > > I think we did a ~30M url indexing and didn't run into any problems. > > Did you get a task timeout? (can it be related to a slowish indexing > filter like language-identifier?) > >> Dennis >> > > ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
