Hi, I am having a problem indexing segments in Nutch 1.11 on Hadoop.
The cluster seems to be configured correctly and every part of the crawl process is working flawlessly, however this is my first attempt at hadoop 2, so perhaps my memory settings aren't perfect. I'm also not sure where to look in the log files for more information. The same data can be indexed with Nutch in local mode, so I don't think it is a problem with the Solr configuration, and I have had Nutch 1.0.9 with Hadoop 1.2.1 on this same cluster and everything worked ok. Please let me know if I can send more information, I have spent several days working on this with no success or clue why it is happening. Thanks in advance, Jason ### Command ### /root/hadoop-2.4.0/bin/hadoop jar /root/src/apache-nutch-1.11/build/apache-nutch-1.11.job org.apache.nutch.indexer.IndexingJob crawl/crawldb -linkdb crawl/linkdb crawl/segments/20160121113335 ### Error ### 16/01/21 14:20:47 INFO mapreduce.Job: map 100% reduce 19% 16/01/21 14:20:48 INFO mapreduce.Job: map 100% reduce 26% 16/01/21 14:20:48 INFO mapreduce.Job: Task Id : attempt_1453403905213_0001_r_000001_0, Status : FAILED Error: INSTANCE 16/01/21 14:20:48 INFO mapreduce.Job: Task Id : attempt_1453403905213_0001_r_000002_0, Status : FAILED Error: INSTANCE 16/01/21 14:20:48 INFO mapreduce.Job: Task Id : attempt_1453403905213_0001_r_000000_0, Status : FAILED Error: INSTANCE 16/01/21 14:20:49 INFO mapreduce.Job: map 100% reduce 0% 16/01/21 14:20:54 INFO mapreduce.Job: Task Id : attempt_1453403905213_0001_r_000004_0, Status : FAILED Error: INSTANCE 16/01/21 14:20:55 INFO mapreduce.Job: Task Id : attempt_1453403905213_0001_r_000002_1, Status : FAILED Error: INSTANCE 16/01/21 14:20:56 INFO mapreduce.Job: Task Id : attempt_1453403905213_0001_r_000001_1, Status : FAILED Error: INSTANCE 16/01/21 14:21:00 INFO mapreduce.Job: Task Id : attempt_1453403905213_0001_r_000000_1, Status : FAILED Error: INSTANCE 16/01/21 14:21:01 INFO mapreduce.Job: Task Id : attempt_1453403905213_0001_r_000004_1, Status : FAILED Error: INSTANCE 16/01/21 14:21:02 INFO mapreduce.Job: Task Id : attempt_1453403905213_0001_r_000002_2, Status : FAILED Error: INSTANCE 16/01/21 14:21:07 INFO mapreduce.Job: Task Id : attempt_1453403905213_0001_r_000003_0, Status : FAILED Error: INSTANCE 16/01/21 14:21:08 INFO mapreduce.Job: Task Id : attempt_1453403905213_0001_r_000004_2, Status : FAILED Error: INSTANCE 16/01/21 14:21:08 INFO mapreduce.Job: Task Id : attempt_1453403905213_0001_r_000001_2, Status : FAILED Error: INSTANCE 16/01/21 14:21:11 INFO mapreduce.Job: Task Id : attempt_1453403905213_0001_r_000000_2, Status : FAILED Error: INSTANCE 16/01/21 14:21:15 INFO mapreduce.Job: Task Id : attempt_1453403905213_0001_r_000003_1, Status : FAILED Error: INSTANCE 16/01/21 14:21:16 INFO mapreduce.Job: map 100% reduce 100% 16/01/21 14:21:16 INFO mapreduce.Job: Job job_1453403905213_0001 failed with state FAILED due to: Task failed task_1453403905213_0001_r_000004 Job failed as tasks failed. failedMaps:0 failedReduces:1 16/01/21 14:21:16 INFO mapreduce.Job: Counters: 39 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=5578886 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=2277523 HDFS: Number of bytes written=0 HDFS: Number of read operations=80 HDFS: Number of large read operations=0 HDFS: Number of write operations=0 Job Counters Failed reduce tasks=15 Killed reduce tasks=2 Launched map tasks=20 Launched reduce tasks=17 Data-local map tasks=19 Rack-local map tasks=1 Total time spent by all maps in occupied slots (ms)=334664 Total time spent by all reduces in occupied slots (ms)=548199 Total time spent by all map tasks (ms)=167332 Total time spent by all reduce tasks (ms)=182733 Total vcore-seconds taken by all map tasks=167332 Total vcore-seconds taken by all reduce tasks=182733 Total megabyte-seconds taken by all map tasks=257021952 Total megabyte-seconds taken by all reduce tasks=561355776 Map-Reduce Framework Map input records=18083 Map output records=18083 Map output bytes=3140643 Map output materialized bytes=3178436 Input split bytes=2812 Combine input records=0 Spilled Records=18083 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=1182 CPU time spent (ms)=56070 Physical memory (bytes) snapshot=6087245824 Virtual memory (bytes) snapshot=34655649792 Total committed heap usage (bytes)=5412749312 File Input Format Counters Bytes Read=2274711 16/01/21 14:21:16 ERROR indexer.IndexingJob: Indexer: java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:145) at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:222) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:231) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
