Brian Whitman wrote: >>> No, it's a RAID-5 local disk >>> >>> /dev/md0 1.5T 193G 1.2T 14% /array >> >> Ok, then I don't know what's causing it. There are two options, >> really - either the file is really corrupted, or there is a subtle >> bug in Hadoop somewhere. Since this is a local FS, you can try >> removing the .crc file and see if it helps (Hadoop should rebuild >> this file when it's needed). >> >> -- > > OK-- I will check the RAID overnight and run the crawl again on a > different drive. I can't just re-run the segment merge because the > re-crawl script deletes all the segment directories whether or not > they were successfully merged. > > Relatedly, is there a way for a script to know if bin/nutch > (somecommand) completed successfully? I would like to avoid this issue > in the future :)
All commands should return proper status codes, generally 0 for success and < 0 for failure. If you find a tool that doesn't do this it's a bug and I'll fix it. -- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
