Andrzej Bialecki wrote: > The reason is that if you pack this file into your job JAR, the job jar > would become very large (presumably this 40MB is already compressed?). > Job jar needs to be copied to each tasktracker for each task, so you > will experience performance hit just because of the size of the job jar > ... whereas if this file sits on DFS and is highly replicated, its > content will always be available locally.
Note that the job jar is copied into HDFS with a highish replication (10?), and that it is only copied to each tasktracker node once per *job*, not per task. So it's only faster to manage this yourself if you have a sequence of jobs that share this data, and if the time to re-replicate it per job is significant. Doug ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
