Can you also post your configuration files here ?
Stanislaw Vasiljev wrote:
Can you check $HADOOP_HOME/logs/userlogs/ subdirectories for attempt
logs. They can contain map-reduce related errors.
2009/10/16 Stephane Brossier <[email protected]>:
I installed hadoop on my workstation today (pseudo distributed mode) using
the instructions:
http://hadoop.apache.org/common/docs/r0.19.1/quickstart.html#Download
The installation is straightforward, but then when i try to run the example,
all the map tasks complete,
but the reduce task never completes. It looks like there is some connection
problems??
I made sure that the user 'viking' running both the hadoop java processes
and the job can ssh into the machine
without needed keys/passphrase.
From looking at the code, in ReduceTask.java, it looks like the reduce task
is trying to contact the TaskTracker, but from the syslog below,
that seems to fail:
URL mapOutputLocation = new URL(event.getTaskTrackerHttp() +
"/mapOutput?job=" + taskId.getJobID() +
"&map=" + taskId +
"&reduce=" + getPartition());
My TaskTracker is listening on port 50060 and i can access the url.
What can be wrong???
Thanks,
S.
# Test ran:
vik...@stephane-desktop:~/hadoop$ hadoop jar hadoop-*-examples.jar grep
input output 'dfs[a-z.]+'
09/10/15 18:50:00 INFO mapred.FileInputFormat: Total input paths to process
: 10
09/10/15 18:50:00 INFO mapred.JobClient: Running job: job_200910151506_0001
09/10/15 18:50:01 INFO mapred.JobClient: map 0% reduce 0%
09/10/15 18:50:07 INFO mapred.JobClient: map 9% reduce 0%
09/10/15 18:50:08 INFO mapred.JobClient: map 18% reduce 0%
09/10/15 18:50:09 INFO mapred.JobClient: map 27% reduce 0%
09/10/15 18:50:10 INFO mapred.JobClient: map 36% reduce 0%
09/10/15 18:50:11 INFO mapred.JobClient: map 54% reduce 0%
09/10/15 18:50:12 INFO mapred.JobClient: map 63% reduce 0%
09/10/15 18:50:13 INFO mapred.JobClient: map 72% reduce 0%
09/10/15 18:50:14 INFO mapred.JobClient: map 81% reduce 0%
09/10/15 18:50:15 INFO mapred.JobClient: map 90% reduce 0%
09/10/15 18:50:16 INFO mapred.JobClient: map 100% reduce 0%
# Syslog from the reduce task
syslog logs
2009-10-15 18:50:16,212 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=SHUFFLE, sessionId=
2009-10-15 18:50:16,281 INFO org.apache.hadoop.mapred.ReduceTask:
ShuffleRamManager: MemoryLimit=668126400, MaxSingleShuffleLimit=167031600
2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Thread started: Thread for merging
on-disk files
2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Thread started: Thread for merging in
memory files
2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Thread waiting: Thread for merging
on-disk files
2009-10-15 18:50:16,288 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Need another 11 map output(s) where 0
is already in progress
2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0: Got 11 new map-outputs
2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
dup hosts)
2009-10-15 18:50:46,354 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 copy failed:
attempt_200910151506_0001_m_000000_0 from stephane-desktop
2009-10-15 18:50:46,355 WARN org.apache.hadoop.mapred.ReduceTask:
java.net.SocketTimeoutException: connect timed out
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1360)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1354)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1008)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1293)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1231)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1144)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1084)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.http.HttpClient.New(HttpClient.java:323)
at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:852)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:793)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:718)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1041)
... 4 more