Can you also post your configuration files here ?
Stanislaw Vasiljev wrote:
Can you check $HADOOP_HOME/logs/userlogs/ subdirectories for attempt
logs. They can contain map-reduce related errors.

2009/10/16 Stephane Brossier <[email protected]>:
I installed hadoop on my workstation today (pseudo distributed mode) using
the instructions:
http://hadoop.apache.org/common/docs/r0.19.1/quickstart.html#Download

The installation is straightforward, but then when i try to run the example,
all the map tasks complete,
but the reduce task never completes. It looks like there is some connection
problems??

I made sure that the user 'viking' running both the hadoop java processes
and the job can ssh into the machine
without needed keys/passphrase.


From looking at the code, in ReduceTask.java, it looks like the reduce task
is trying to contact the TaskTracker, but from the syslog below,
that seems to fail:
  URL mapOutputLocation = new URL(event.getTaskTrackerHttp() +
                                   "/mapOutput?job=" + taskId.getJobID() +
                                   "&map=" + taskId +
                                   "&reduce=" + getPartition());

My TaskTracker is listening on port 50060 and i can access the url.

What can be wrong???


Thanks,

S.


# Test ran:
vik...@stephane-desktop:~/hadoop$ hadoop jar hadoop-*-examples.jar grep
input output 'dfs[a-z.]+'
09/10/15 18:50:00 INFO mapred.FileInputFormat: Total input paths to process
: 10
09/10/15 18:50:00 INFO mapred.JobClient: Running job: job_200910151506_0001
09/10/15 18:50:01 INFO mapred.JobClient:  map 0% reduce 0%
09/10/15 18:50:07 INFO mapred.JobClient:  map 9% reduce 0%
09/10/15 18:50:08 INFO mapred.JobClient:  map 18% reduce 0%
09/10/15 18:50:09 INFO mapred.JobClient:  map 27% reduce 0%
09/10/15 18:50:10 INFO mapred.JobClient:  map 36% reduce 0%
09/10/15 18:50:11 INFO mapred.JobClient:  map 54% reduce 0%
09/10/15 18:50:12 INFO mapred.JobClient:  map 63% reduce 0%
09/10/15 18:50:13 INFO mapred.JobClient:  map 72% reduce 0%
09/10/15 18:50:14 INFO mapred.JobClient:  map 81% reduce 0%
09/10/15 18:50:15 INFO mapred.JobClient:  map 90% reduce 0%
09/10/15 18:50:16 INFO mapred.JobClient:  map 100% reduce 0%

# Syslog from the reduce task
syslog logs
2009-10-15 18:50:16,212 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=SHUFFLE, sessionId=
2009-10-15 18:50:16,281 INFO org.apache.hadoop.mapred.ReduceTask:
ShuffleRamManager: MemoryLimit=668126400, MaxSingleShuffleLimit=167031600
2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Thread started: Thread for merging
on-disk files
2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Thread started: Thread for merging in
memory files
2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Thread waiting: Thread for merging
on-disk files
2009-10-15 18:50:16,288 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Need another 11 map output(s) where 0
is already in progress
2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0: Got 11 new map-outputs
2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
dup hosts)
2009-10-15 18:50:46,354 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 copy failed:
attempt_200910151506_0001_m_000000_0 from stephane-desktop
2009-10-15 18:50:46,355 WARN org.apache.hadoop.mapred.ReduceTask:
java.net.SocketTimeoutException: connect timed out
       at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
       at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
       at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
       at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
       at
sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1360)
       at java.security.AccessController.doPrivileged(Native Method)
       at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1354)
       at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1008)
       at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1293)
       at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1231)
       at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1144)
       at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1084)
Caused by: java.net.SocketTimeoutException: connect timed out
       at java.net.PlainSocketImpl.socketConnect(Native Method)
       at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
       at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
       at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
       at java.net.Socket.connect(Socket.java:519)
       at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
       at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
       at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
       at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
       at sun.net.www.http.HttpClient.New(HttpClient.java:306)
       at sun.net.www.http.HttpClient.New(HttpClient.java:323)
       at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:852)
       at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:793)
       at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:718)
       at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1041)
       ... 4 more




Reply via email to