Now theres this error showing up.
When i run a job on my 2 node cluster, it hangs at
[ ~]$ hadoop jar $HADOOP_HOME/hadoop-0.18.3-examples.jar wordcount gutenberg
gutenberg-output
09/06/06 01:50:54 INFO mapred.FileInputFormat: Total input paths to process
: 6
09/06/06 01:50:54 INFO mapred.FileInputFormat: Total input paths to process
: 6
09/06/06 01:50:54 INFO mapred.JobClient: Running job: job_200906060149_0001
09/06/06 01:50:55 INFO mapred.JobClient: map 0% reduce 0%
09/06/06 01:51:01 INFO mapred.JobClient: map 33% reduce 0%
09/06/06 01:51:02 INFO mapred.JobClient: map 66% reduce 0%
09/06/06 01:51:03 INFO mapred.JobClient: map 100% reduce 0%
*******************************************************************************
SYSLOG of reduce:
[~]$ cat
/home/utdhadoop1/Hadoop/hadoop-0.18.3/logs/userlogs/attempt_200906060156_0001_r_000000_0/syslog
2009-06-06 01:56:52,838 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=SHUFFLE, sessionId=
2009-06-06 01:56:53,254 INFO org.apache.hadoop.mapred.ReduceTask:
ShuffleRamManager: MemoryLimit=78643200, MaxSingleShuffleLimit=19660800
2009-06-06 01:56:53,259 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Thread started: Thread for merging
on-disk files
2009-06-06 01:56:53,259 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Thread waiting: Thread for merging
on-disk files
2009-06-06 01:56:53,260 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Thread started: Thread for merging in
memory files
2009-06-06 01:56:53,260 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Need another 6 map output(s) where 0 is
already in progress
2009-06-06 01:56:53,264 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0: Got 4 new map-outputs & number of
known map outputs is 4
2009-06-06 01:56:53,265 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Scheduled 2 of 4 known outputs (0 slow
hosts and 2 dup hosts)
2009-06-06 01:56:53,428 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 copy failed:
attempt_200906060156_0001_m_000000_0 from **********
2009-06-06 01:56:53,428 WARN org.apache.hadoop.mapred.ReduceTask:
java.net.NoRouteToHostException: No route to host
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1360)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1354)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1008)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1143)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1084)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:997)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:946)
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.http.HttpClient.New(HttpClient.java:323)
at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:852)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:793)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:718)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1041)
... 4 more
2009-06-06 01:56:54,264 INFO org.apache.hadoop.mapred.ReduceTask: Task
attempt_200906060156_0001_r_000000_0: Failed fetch #1 from
attempt_200906060156_0001_m_000000_0
2009-06-06 01:56:54,264 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 adding host ********** to penalty box,
next contact in 150 seconds
2009-06-06 01:56:55,265 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0: Got 2 map-outputs from previous
failures
2009-06-06 01:56:56,276 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 copy failed:
attempt_200906060156_0001_m_000003_0 from **********
2009-06-06 01:56:56,278 WARN org.apache.hadoop.mapred.ReduceTask:
java.net.NoRouteToHostException: No route to host
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1360)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1354)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1008)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1143)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1084)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:997)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:946)
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.http.HttpClient.New(HttpClient.java:323)
at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:852)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:793)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:718)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1041)
... 4 more
2009-06-06 01:56:57,271 INFO org.apache.hadoop.mapred.ReduceTask: Task
attempt_200906060156_0001_r_000000_0: Failed fetch #1 from
attempt_200906060156_0001_m_000003_0
2009-06-06 01:56:57,271 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 adding host ********** to penalty box,
next contact in 150 seconds
2009-06-06 01:56:58,271 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0: Got 2 map-outputs from previous
failures
2009-06-06 01:57:03,275 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0: Got 2 new map-outputs & number of
known map outputs is 6
2009-06-06 01:57:53,308 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Need another 6 map output(s) where 0 is
already in progress
2009-06-06 01:57:53,309 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0: Got 0 new map-outputs & number of
known map outputs is 6
2009-06-06 01:57:53,309 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Scheduled 0 of 6 known outputs (2 slow
hosts and 0 dup hosts)
2009-06-06 01:57:53,309 INFO org.apache.hadoop.mapred.ReduceTask:
Penalized(slow) Hosts:
2009-06-06 01:57:53,309 INFO org.apache.hadoop.mapred.ReduceTask: **********
Will be considered after: 90 seconds.
2009-06-06 01:57:53,309 INFO org.apache.hadoop.mapred.ReduceTask: **********
Will be considered after: 93 seconds.
2009-06-06 01:58:53,345 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Need another 6 map output(s) where 0 is
already in progress
2009-06-06 01:58:53,346 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0: Got 0 new map-outputs & number of
known map outputs is 6
2009-06-06 01:58:53,346 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Scheduled 0 of 6 known outputs (2 slow
hosts and 0 dup hosts)
2009-06-06 01:58:53,346 INFO org.apache.hadoop.mapred.ReduceTask:
Penalized(slow) Hosts:
2009-06-06 01:58:53,346 INFO org.apache.hadoop.mapred.ReduceTask: **********
Will be considered after: 30 seconds.
2009-06-06 01:58:53,346 INFO org.apache.hadoop.mapred.ReduceTask: **********
Will be considered after: 33 seconds.
2009-06-06 01:59:28,371 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Scheduled 2 of 6 known outputs (0 slow
hosts and 4 dup hosts)
2009-06-06 01:59:28,525 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 copy failed:
attempt_200906060156_0001_m_000004_0 from **********
2009-06-06 01:59:28,525 WARN org.apache.hadoop.mapred.ReduceTask:
java.net.NoRouteToHostException: No route to host
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1360)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1354)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1008)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1143)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1084)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:997)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:946)
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.http.HttpClient.New(HttpClient.java:323)
at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:852)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:793)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:718)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1041)
... 4 more
2009-06-06 01:59:29,372 INFO org.apache.hadoop.mapred.ReduceTask: Task
attempt_200906060156_0001_r_000000_0: Failed fetch #1 from
attempt_200906060156_0001_m_000004_0
2009-06-06 01:59:29,373 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 adding host ********** to penalty box,
next contact in 150 seconds
2009-06-06 01:59:30,374 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0: Got 4 map-outputs from previous
failures
2009-06-06 01:59:31,527 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 copy failed:
attempt_200906060156_0001_m_000003_0 from **********
2009-06-06 01:59:31,528 WARN org.apache.hadoop.mapred.ReduceTask:
java.net.NoRouteToHostException: No route to host
at sun.reflect.GeneratedConstructorAccessor3.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1360)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1354)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1008)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1143)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1084)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:997)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:946)
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.http.HttpClient.New(HttpClient.java:323)
at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:852)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:793)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:718)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1041)
... 4 more
2009-06-06 01:59:32,380 INFO org.apache.hadoop.mapred.ReduceTask: Task
attempt_200906060156_0001_r_000000_0: Failed fetch #2 from
attempt_200906060156_0001_m_000003_0
2009-06-06 01:59:32,381 INFO org.apache.hadoop.mapred.ReduceTask: Failed to
fetch map-output from attempt_200906060156_0001_m_000003_0 even after
MAX_FETCH_RETRIES_PER_MAP retries... reporting to the JobTracker
2009-06-06 01:59:32,381 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 adding host ********** to penalty box,
next contact in 150 seconds
2009-06-06 01:59:33,382 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0: Got 2 map-outputs from previous
failures
2009-06-06 01:59:53,394 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Need another 6 map output(s) where 0 is
already in progress
2009-06-06 01:59:53,395 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0: Got 0 new map-outputs & number of
known map outputs is 6
2009-06-06 01:59:53,395 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Scheduled 0 of 6 known outputs (2 slow
hosts and 0 dup hosts)
2009-06-06 01:59:53,395 INFO org.apache.hadoop.mapred.ReduceTask:
Penalized(slow) Hosts:
2009-06-06 01:59:53,395 INFO org.apache.hadoop.mapred.ReduceTask: **********
Will be considered after: 125 seconds.
2009-06-06 01:59:53,395 INFO org.apache.hadoop.mapred.ReduceTask: **********
Will be considered after: 128 seconds.
2009-06-06 02:00:53,424 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Need another 6 map output(s) where 0 is
already in progress
2009-06-06 02:00:53,424 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0: Got 0 new map-outputs & number of
known map outputs is 6
2009-06-06 02:00:53,424 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Scheduled 0 of 6 known outputs (2 slow
hosts and 0 dup hosts)
2009-06-06 02:00:53,424 INFO org.apache.hadoop.mapred.ReduceTask:
Penalized(slow) Hosts:
2009-06-06 02:00:53,424 INFO org.apache.hadoop.mapred.ReduceTask: **********
Will be considered after: 65 seconds.
2009-06-06 02:00:53,424 INFO org.apache.hadoop.mapred.ReduceTask: **********
Will be considered after: 68 seconds.
2009-06-06 02:01:53,454 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Need another 6 map output(s) where 0 is
already in progress
2009-06-06 02:01:53,455 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0: Got 0 new map-outputs & number of
known map outputs is 6
2009-06-06 02:01:53,455 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200906060156_0001_r_000000_0 Scheduled 0 of 6 known outputs (2 slow
hosts and 0 dup hosts)
2009-06-06 02:01:53,455 INFO org.apache.hadoop.mapred.ReduceTask:
Penalized(slow) Hosts:
2009-06-06 02:01:53,455 INFO org.apache.hadoop.mapred.ReduceTask: **********
Will be considered after: 5 seconds.
2009-06-06 02:01:53,455 INFO org.apache.hadoop.mapred.ReduceTask: **********
Will be considered after: 8 seconds.
*********************************************************************************************************************************************
Any ideas as to why .
Thanks
Asif .
