I'm running Hadoop 0.20.1+133 (Cloudera distro)
I tried setting up a multi-node Hadoop cluster and on executing the command:
hadoop jar /usr/lib/hadoop/hadoop-0.20.1+133-examples.jar grep input output
'dfs[a-z.]+'
I get:
09/10/27 20:39:21 INFO mapred.FileInputFormat: Total input paths to process
: 5
09/10/27 20:39:21 INFO mapred.JobClient: Running job: job_200910272023_0002
09/10/27 20:39:22 INFO mapred.JobClient: map 0% reduce 0%
09/10/27 20:39:30 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_m_000006_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)
09/10/27 20:39:30 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_0&filter=stdout
09/10/27 20:39:30 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_0&filter=stderr
09/10/27 20:39:36 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_r_000020_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)
09/10/27 20:39:36 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_0&filter=stdout
09/10/27 20:39:36 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_0&filter=stderr
09/10/27 20:39:42 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_m_000006_1, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)
09/10/27 20:39:42 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_1&filter=stdout
09/10/27 20:39:42 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_1&filter=stderr
09/10/27 20:39:48 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_r_000020_1, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)
09/10/27 20:39:48 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_1&filter=stdout
09/10/27 20:39:48 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_1&filter=stderr
09/10/27 20:39:57 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_m_000006_2, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)
09/10/27 20:39:57 WARN mapred.JobClient: Error reading task outputhttp://
anza4.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_2&filter=stdout
09/10/27 20:39:57 WARN mapred.JobClient: Error reading task outputhttp://
anza4.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_2&filter=stderr
09/10/27 20:40:03 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_r_000020_2, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)
09/10/27 20:40:03 WARN mapred.JobClient: Error reading task outputhttp://
anza4.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_2&filter=stdout
09/10/27 20:40:03 WARN mapred.JobClient: Error reading task outputhttp://
anza4.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_2&filter=stderr
09/10/27 20:40:15 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_m_000005_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)
09/10/27 20:40:15 WARN mapred.JobClient: Error reading task outputhttp://
anza2.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_0&filter=stdout
09/10/27 20:40:15 WARN mapred.JobClient: Error reading task outputhttp://
anza2.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_0&filter=stderr
09/10/27 20:40:21 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_r_000019_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)
09/10/27 20:40:21 WARN mapred.JobClient: Error reading task outputhttp://
anza2.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_0&filter=stdout
09/10/27 20:40:21 WARN mapred.JobClient: Error reading task outputhttp://
anza2.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_0&filter=stderr
09/10/27 20:40:30 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_m_000005_1, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)
09/10/27 20:40:30 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_1&filter=stdout
09/10/27 20:40:30 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_1&filter=stderr
09/10/27 20:40:36 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_r_000019_1, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)
09/10/27 20:40:36 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_1&filter=stdout
09/10/27 20:40:36 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_1&filter=stderr
09/10/27 20:40:42 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_m_000005_2, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)
09/10/27 20:40:42 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_2&filter=stdout
09/10/27 20:40:42 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_2&filter=stderr
09/10/27 20:40:48 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_r_000019_2, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)
09/10/27 20:40:48 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_2&filter=stdout
09/10/27 20:40:48 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_2&filter=stderr
09/10/27 20:40:57 INFO mapred.JobClient: Job complete: job_200910272023_0002
09/10/27 20:40:57 INFO mapred.JobClient: Counters: 0
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1293)
at org.apache.hadoop.examples.Grep.run(Grep.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.examples.Grep.main(Grep.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:185)
Based upon a post I read to a similar issue, I changed my /etc/hosts file
to:
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
10.50.65.61 anza1.eng.blah.com anza1
10.50.65.62 anza2.eng.blah.com anza2
10.50.65.63 anza3.eng.blah.com anza3
10.50.65.64 anza4.eng.blah.com anza4
10.50.65.65 anza5.eng.blah.com anza5
Also, when I look at:
/var/log/hadoop/userlogs/attempt_200910271659_0007_r_000019_0 on a slave
STDOUT:
Error occurred during initialization of VM
Could not reserve enough space for object heap
STDERR:
Could not create the Java virtual machine.
My slaves are running on boxes with 8GB or RAM and under:
JAVA_HEAP_MAX=-Xmx1000m
And under mapred-site.xml:
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx2048m</value>
</property>
I can't figure out why the slaves are failing?
