Author: jnioche Date: Tue Dec 1 12:47:42 2015 New Revision: 1717412 URL: http://svn.apache.org/viewvc?rev=1717412&view=rev Log: NUTCH-2177 Generator produces only one partition even in distributed mode
Modified: nutch/trunk/CHANGES.txt nutch/trunk/src/java/org/apache/nutch/crawl/Generator.java Modified: nutch/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/nutch/trunk/CHANGES.txt?rev=1717412&r1=1717411&r2=1717412&view=diff ============================================================================== --- nutch/trunk/CHANGES.txt (original) +++ nutch/trunk/CHANGES.txt Tue Dec 1 12:47:42 2015 @@ -3,6 +3,8 @@ Nutch Change Log Nutch 1.11 Release 25/10/2015 (dd/mm/yyyy) Release Report: http://s.apache.org/nutch11 +* NUTCH-2177 Generator produces only one partition even in distributed mode (jnioche, snagel) + * NUTCH-2158 Upgrade to Tika 1.11 (jnioche, snagel) * NUTCH-2175 Typos in property descriptions in nutch-default.xml (Roannel Fernández Hernández via snagel) Modified: nutch/trunk/src/java/org/apache/nutch/crawl/Generator.java URL: http://svn.apache.org/viewvc/nutch/trunk/src/java/org/apache/nutch/crawl/Generator.java?rev=1717412&r1=1717411&r2=1717412&view=diff ============================================================================== --- nutch/trunk/src/java/org/apache/nutch/crawl/Generator.java (original) +++ nutch/trunk/src/java/org/apache/nutch/crawl/Generator.java Tue Dec 1 12:47:42 2015 @@ -539,9 +539,9 @@ public class Generator extends NutchTool if (numLists == -1) { // for politeness make numLists = job.getNumMapTasks(); // a partition per fetch task } - if ("local".equals(job.get("mapred.job.tracker")) && numLists != 1) { + if ("local".equals(job.get("mapreduce.framework.name")) && numLists != 1) { // override - LOG.info("Generator: jobtracker is 'local', generating exactly one partition."); + LOG.info("Generator: running in local mode, generating exactly one partition."); numLists = 1; } job.setLong(GENERATOR_CUR_TIME, curTime);