Huy, Stanislaw,Thanks a lot for your help. In the meantime i recompiled hadoop to add more traces and figured out the issue. My workstation IP had changed and the the entry in /etc/passwd' had not been updated correctly so the Reduce
task could not contact the TaskTracker.
I can now proceed with real MR jobs and start having some fun:-) Thanks again for your replies, i appreciate your help. S. On Oct 16, 2009, at 6:13 PM, Huy Phan wrote:
Hi Stephane,Your configuration file is so simple so it cannot cause any problems here :) What is see in the log is that you're using 'stephane-desktop' as the hostname of your machine, I guess Hadoop was failed to resolve the IP from this hostname.Regards, Huy Phan Stephane Brossier wrote:Stan, Huy,Thanks for your answer. Let me know if i can send you more information than that. I am bit clueless on why the reduce task tas gets this conection exception. Is my understanding correct regarding the failure: Is that the 'reduce task' failing to connect to the TaskTracker during the copy of the map output? Where do these map output live? Are they on disk somewhere?Thanks, S. The configuration i am using is: *hadoop-site.xm*l -- default suggested on the web site: <configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> <property> <name>mapred.job.tracker</name> <value>localhost:9001</value> </property> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration> I did not modify the hadoop-default.xmlAs for the errors in the logs/userlogs, i see the following-- over and over:2009-10-15 23:54:33,118 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=SHUFFLE, sessionId= 2009-10-15 23:54:33,186 INFO org.apache.hadoop.mapred.ReduceTask: ShuffleRamManager: MemoryLimit=130514944, MaxSingleShuffleLimit=32628736 2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3 Thread started: Thread for merging on-disk files 2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3 Thread started: Thread for merging in memory files 2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3 Thread waiting: Thread for merging on-disk files 2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3 Need another 11 map output(s) where 0 is already in progress 2009-10-15 23:54:33,200 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring obsolete output of FAILED map-task: 'attempt_200910152108_0001_m_000000_0' 2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring obsolete output of FAILED map-task: 'attempt_200910152108_0001_m_000001_0' 2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring obsolete output of FAILED map-task: 'attempt_200910152108_0001_m_000002_0' 2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring obsolete output of FAILED map-task: 'attempt_200910152108_0001_m_000003_0' 2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring obsolete output of FAILED map-task: 'attempt_200910152108_0001_m_000004_0' 2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring obsolete output of FAILED map-task: 'attempt_200910152108_0001_m_000005_0' 2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring obsolete output of FAILED map-task: 'attempt_200910152108_0001_m_000006_0' 2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring obsolete output of FAILED map-task: 'attempt_200910152108_0001_m_000007_0' 2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring obsolete output of FAILED map-task: 'attempt_200910152108_0001_m_000008_0' 2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring obsolete output of FAILED map-task: 'attempt_200910152108_0001_m_000009_0' 2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring obsolete output of FAILED map-task: 'attempt_200910152108_0001_m_000010_0' 2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring obsolete output of FAILED map-task: 'attempt_200910152108_0001_m_000000_1' 2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3: Got 23 new map-outputs 2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3: Got 12 obsolete map-outputs from tasktracker 2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3 Scheduled 1 outputs (0 slow hosts and0 dup hosts) 2009-10-15 23:55:03,251 WARN org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3 copy failed: attempt_200910152108_0001_m_000001_1 from stephane-desktop 2009-10-15 23:55:03,252 WARN org.apache.hadoop.mapred.ReduceTask: java.net.SocketTimeoutException: connect timed out at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun .reflect .NativeConstructorAccessorImpl .newInstance(NativeConstructorAccessorImpl.java:39) at sun .reflect .DelegatingConstructorAccessorImpl .newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at sun.net.www.protocol.http.HttpURLConnection <http://www.protocol.http.HttpURLConnection >$6.run(HttpURLConnection.java:1360)at java.security.AccessController.doPrivileged(Native Method)at sun.net.www.protocol.http.HttpURLConnection.getChainedException <http:// www .protocol .http.HttpURLConnection.getChainedException>(HttpURLConnection.java: 1354) at sun.net.www.protocol.http.HttpURLConnection.getInputStream <http://www.protocol.http.HttpURLConnection.getInputStream >(HttpURLConnection.java:1008) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.getInputStream(ReduceTask.java:1293) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.getMapOutput(ReduceTask.java:1231) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.copyOutput(ReduceTask.java:1144) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.run(ReduceTask.java:1084)Caused by: java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java: 333) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:519) at sun.net.NetworkClient.doConnect(NetworkClient.java:158)at sun.net.www.http.HttpClient.openServer <http://www.http.HttpClient.openServer >(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer <http://www.http.HttpClient.openServer >(HttpClient.java:529) at sun.net.www.http.HttpClient <http:// www.http.HttpClient>.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New <http://www.http.HttpClient.New >(HttpClient.java:306) at sun.net.www.http.HttpClient.New <http://www.http.HttpClient.New >(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient <http:// www .protocol .http.HttpURLConnection.getNewHttpClient>(HttpURLConnection.java:852) at sun.net.www.protocol.http.HttpURLConnection.plainConnect <http://www.protocol.http.HttpURLConnection.plainConnect >(HttpURLConnection.java:793) at sun.net.www.protocol.http.HttpURLConnection.connect <http://www.protocol.http.HttpURLConnection.connect >(HttpURLConnection.java:718) at sun.net.www.protocol.http.HttpURLConnection.getInputStream <http://www.protocol.http.HttpURLConnection.getInputStream >(HttpURLConnection.java:1041)... 4 more2009-10-15 23:55:04,196 INFO org.apache.hadoop.mapred.ReduceTask: Task attempt_200910152108_0001_r_000000_3: Failed fetch #1 from attempt_200910152108_0001_m_000001_1 2009-10-15 23:55:04,196 WARN org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3 adding host stephane-desktop to penalty box, next contact in 4 seconds 2009-10-15 23:55:05,196 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3: Got 1 map-outputs from previous failures 2009-10-15 23:55:10,196 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3 Scheduled 1 outputs (0 slow hosts and0 dup hosts) 2009-10-15 23:55:33,201 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3 Need another 11 map output(s) where 1 is already in progress 2009-10-15 23:55:34,200 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3: Got 0 new map-outputs 2009-10-15 23:55:34,200 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3 Scheduled 0 outputs (0 slow hosts and9 dup hosts) 2009-10-15 23:55:40,228 WARN org.apache.hadoop.mapred.ReduceTask: attempt_200910152108_0001_r_000000_3 copy failed: attempt_200910152108_0001_m_000001_1 from stephane-desktop 2009-10-15 23:55:40,228 WARN org.apache.hadoop.mapred.ReduceTask: java.net.SocketTimeoutException: connect timed outOn Oct 16, 2009, at 9:09 AM, Stanislaw Vasiljev wrote:Can you check $HADOOP_HOME/logs/userlogs/ subdirectories for attempt logs. They can contain map-reduce related errors.2009/10/16 Stephane Brossier <[email protected] <mailto:[email protected] >>:I installed hadoop on my workstation today (pseudo distributed mode) usingthe instructions: http://hadoop.apache.org/common/docs/r0.19.1/quickstart.html#DownloadThe installation is straightforward, but then when i try to run the example,all the map tasks complete,but the reduce task never completes. It looks like there is some connectionproblems??I made sure that the user 'viking' running both the hadoop java processesand the job can ssh into the machine without needed keys/passphrase.From looking at the code, in ReduceTask.java, it looks like the reduce taskis trying to contact the TaskTracker, but from the syslog below, that seems to fail: URL mapOutputLocation = new URL(event.getTaskTrackerHttp() +"/mapOutput?job=" + taskId.getJobID() +"&map=" + taskId + "&reduce=" + getPartition()); My TaskTracker is listening on port 50060 and i can access the url. What can be wrong??? Thanks, S. # Test ran:vik...@stephane-desktop:~/hadoop$ hadoop jar hadoop-*- examples.jar grepinput output 'dfs[a-z.]+'09/10/15 18:50:00 INFO mapred.FileInputFormat: Total input paths to process: 1009/10/15 18:50:00 INFO mapred.JobClient: Running job: job_200910151506_000109/10/15 18:50:01 INFO mapred.JobClient: map 0% reduce 0% 09/10/15 18:50:07 INFO mapred.JobClient: map 9% reduce 0% 09/10/15 18:50:08 INFO mapred.JobClient: map 18% reduce 0% 09/10/15 18:50:09 INFO mapred.JobClient: map 27% reduce 0% 09/10/15 18:50:10 INFO mapred.JobClient: map 36% reduce 0% 09/10/15 18:50:11 INFO mapred.JobClient: map 54% reduce 0% 09/10/15 18:50:12 INFO mapred.JobClient: map 63% reduce 0% 09/10/15 18:50:13 INFO mapred.JobClient: map 72% reduce 0% 09/10/15 18:50:14 INFO mapred.JobClient: map 81% reduce 0% 09/10/15 18:50:15 INFO mapred.JobClient: map 90% reduce 0% 09/10/15 18:50:16 INFO mapred.JobClient: map 100% reduce 0% # Syslog from the reduce task syslog logs2009-10-15 18:50:16,212 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:Initializing JVM Metrics with processName=SHUFFLE, sessionId= 2009-10-15 18:50:16,281 INFO org.apache.hadoop.mapred.ReduceTask:ShuffleRamManager: MemoryLimit=668126400, MaxSingleShuffleLimit=1670316002009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:attempt_200910151506_0001_r_000000_0 Thread started: Thread for mergingon-disk files 2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:attempt_200910151506_0001_r_000000_0 Thread started: Thread for merging inmemory files 2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:attempt_200910151506_0001_r_000000_0 Thread waiting: Thread for mergingon-disk files 2009-10-15 18:50:16,288 INFO org.apache.hadoop.mapred.ReduceTask:attempt_200910151506_0001_r_000000_0 Need another 11 map output(s) where 0is already in progress 2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910151506_0001_r_000000_0: Got 11 new map-outputs 2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:attempt_200910151506_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0dup hosts) 2009-10-15 18:50:46,354 WARN org.apache.hadoop.mapred.ReduceTask: attempt_200910151506_0001_r_000000_0 copy failed: attempt_200910151506_0001_m_000000_0 from stephane-desktop 2009-10-15 18:50:46,355 WARN org.apache.hadoop.mapred.ReduceTask: java.net.SocketTimeoutException: connect timed outat sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod) atsun .reflect .NativeConstructorAccessorImpl .newInstance(NativeConstructorAccessorImpl.java:39)atsun .reflect .DelegatingConstructorAccessorImpl .newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513)atsun.net.www.protocol.http.HttpURLConnection <http://www.protocol.http.HttpURLConnection >$6.run(HttpURLConnection.java:1360)at java.security.AccessController.doPrivileged(Native Method) atsun.net.www.protocol.http.HttpURLConnection.getChainedException <http://www.protocol.http.HttpURLConnection.getChainedException >(HttpURLConnection.java:1354)atsun.net.www.protocol.http.HttpURLConnection.getInputStream <http://www.protocol.http.HttpURLConnection.getInputStream >(HttpURLConnection.java:1008)atorg.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.getInputStream(ReduceTask.java:1293)atorg.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.getMapOutput(ReduceTask.java:1231)atorg.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.copyOutput(ReduceTask.java:1144)atorg.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.run(ReduceTask.java:1084)Caused by: java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java: 333)at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:519) at sun.net.NetworkClient.doConnect(NetworkClient.java:158)at sun.net.www.http.HttpClient.openServer <http://www.http.HttpClient.openServer >(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer <http://www.http.HttpClient.openServer >(HttpClient.java:529) at sun.net.www.http.HttpClient <http:// www.http.HttpClient>.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New <http://www.http.HttpClient.New >(HttpClient.java:306) at sun.net.www.http.HttpClient.New <http://www.http.HttpClient.New >(HttpClient.java:323)atsun.net.www.protocol.http.HttpURLConnection.getNewHttpClient <http://www.protocol.http.HttpURLConnection.getNewHttpClient >(HttpURLConnection.java:852)atsun.net.www.protocol.http.HttpURLConnection.plainConnect <http://www.protocol.http.HttpURLConnection.plainConnect >(HttpURLConnection.java:793)atsun.net.www.protocol.http.HttpURLConnection.connect <http://www.protocol.http.HttpURLConnection.connect >(HttpURLConnection.java:718)atsun.net.www.protocol.http.HttpURLConnection.getInputStream <http://www.protocol.http.HttpURLConnection.getInputStream >(HttpURLConnection.java:1041)... 4 more
