I installed hadoop on my workstation today (pseudo distributed mode) using the instructions:
http://hadoop.apache.org/common/docs/r0.19.1/quickstart.html#Download

The installation is straightforward, but then when i try to run the example, all the map tasks complete, but the reduce task never completes. It looks like there is some connection problems??

I made sure that the user 'viking' running both the hadoop java processes and the job can ssh into the machine
without needed keys/passphrase.


From looking at the code, in ReduceTask.java, it looks like the reduce task is trying to contact the TaskTracker, but from the syslog below,
that seems to fail:
   URL mapOutputLocation = new URL(event.getTaskTrackerHttp() +
"/mapOutput?job=" + taskId.getJobID() +
                                    "&map=" + taskId +
                                    "&reduce=" + getPartition());

My TaskTracker is listening on port 50060 and i can access the url.

What can be wrong???


Thanks,

S.


# Test ran:
vik...@stephane-desktop:~/hadoop$ hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+' 09/10/15 18:50:00 INFO mapred.FileInputFormat: Total input paths to process : 10 09/10/15 18:50:00 INFO mapred.JobClient: Running job: job_200910151506_0001
09/10/15 18:50:01 INFO mapred.JobClient:  map 0% reduce 0%
09/10/15 18:50:07 INFO mapred.JobClient:  map 9% reduce 0%
09/10/15 18:50:08 INFO mapred.JobClient:  map 18% reduce 0%
09/10/15 18:50:09 INFO mapred.JobClient:  map 27% reduce 0%
09/10/15 18:50:10 INFO mapred.JobClient:  map 36% reduce 0%
09/10/15 18:50:11 INFO mapred.JobClient:  map 54% reduce 0%
09/10/15 18:50:12 INFO mapred.JobClient:  map 63% reduce 0%
09/10/15 18:50:13 INFO mapred.JobClient:  map 72% reduce 0%
09/10/15 18:50:14 INFO mapred.JobClient:  map 81% reduce 0%
09/10/15 18:50:15 INFO mapred.JobClient:  map 90% reduce 0%
09/10/15 18:50:16 INFO mapred.JobClient:  map 100% reduce 0%

# Syslog from the reduce task
syslog logs
2009-10-15 18:50:16,212 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=SHUFFLE, sessionId= 2009-10-15 18:50:16,281 INFO org.apache.hadoop.mapred.ReduceTask: ShuffleRamManager: MemoryLimit=668126400, MaxSingleShuffleLimit=167031600 2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910151506_0001_r_000000_0 Thread started: Thread for merging on-disk files 2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910151506_0001_r_000000_0 Thread started: Thread for merging in memory files 2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910151506_0001_r_000000_0 Thread waiting: Thread for merging on-disk files 2009-10-15 18:50:16,288 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910151506_0001_r_000000_0 Need another 11 map output(s) where 0 is already in progress 2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910151506_0001_r_000000_0: Got 11 new map-outputs 2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200910151506_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0 dup hosts) 2009-10-15 18:50:46,354 WARN org.apache.hadoop.mapred.ReduceTask: attempt_200910151506_0001_r_000000_0 copy failed: attempt_200910151506_0001_m_000000_0 from stephane-desktop 2009-10-15 18:50:46,355 WARN org.apache.hadoop.mapred.ReduceTask: java.net.SocketTimeoutException: connect timed out at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun .reflect .NativeConstructorAccessorImpl .newInstance(NativeConstructorAccessorImpl.java:39) at sun .reflect .DelegatingConstructorAccessorImpl .newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at sun.net.www.protocol.http.HttpURLConnection $6.run(HttpURLConnection.java:1360)
        at java.security.AccessController.doPrivileged(Native Method)
at sun .net .www .protocol .http.HttpURLConnection.getChainedException(HttpURLConnection.java:1354) at sun .net .www .protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java: 1008) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.getInputStream(ReduceTask.java:1293) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.getMapOutput(ReduceTask.java:1231) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.copyOutput(ReduceTask.java:1144) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.run(ReduceTask.java:1084)
Caused by: java.net.SocketTimeoutException: connect timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:519)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
        at sun.net.www.http.HttpClient.New(HttpClient.java:306)
        at sun.net.www.http.HttpClient.New(HttpClient.java:323)
at sun .net .www .protocol .http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:852) at sun .net .www .protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java: 793) at sun .net .www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:718) at sun .net .www .protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java: 1041)
        ... 4 more

Reply via email to