Despite the tremendous effort being put in Hadoop is clear and many features
seems really nice I must admit I'm a bit disappointed. It would be nice if
somebody could comment and/or explain/point out some possible mistake I have
done (if any).
I have installed v. 0.14.1 and built a cluster of 4 Linux machines. That was
easy and worked immediately. I have loaded 10 files of 50MB each. I have
then used HadoopStreaming to run some test ("bin/hadoop jar
contrib/hadoop-streaming.jar -mapper "mygrep"". mygrep is just the following
script: "grep "05:43:30"").
The only difference among such tests was the number of reduce tasks
configured in hadoop-site.xml.
Here are the result of my test runs:
=========================================================================
Test 1) mapred.map.tasks=8; mapred.reduce.tasks=1
the job completes (even if it is pretty slow) but I get only half of the
expected results (22 instead of 44)
Logs for the map seem to show all 44 matches were found:
Counters for task_200709131137_0001_m_000005_0
-------------------------------------------------------------------
Map-Reduce Framework
Map input records 906,932
Map output records 44
Map input bytes 50,000,245
Map output bytes 3,028
But logs for the reduce task show only 22 records
Counters for tip_200709131137_0001_r_000000
-----------------------------------------------------------------------
Map-Reduce Framework
Reduce input groups 22
Reduce input records 22
Reduce output records 22
=========================================================================
Test 2) mapred.map.tasks=8; mapred.reduce.tasks=2
the job completes successfully (even if takes 30 sec) and I get all 44
expected results
Counters for tip_200709131137_0002_r_000000
--------------------------------------------------------------------------------
Map-Reduce Framework
Reduce input groups 22
Reduce input records 22
Reduce output records 22
Counters for task_200709131137_0002_r_000001_0
--------------------------------------------------------------------------------
Map-Reduce Framework
Reduce input groups 22
Reduce input records 22
Reduce output records 22
=========================================================================
Test 3) mapred.map.tasks=8; mapred.reduce.tasks=4
The job doesn't complete at all and seems not to be able to recover from a
kind of looping situation:
07/09/13 11:49:14 INFO streaming.StreamJob: map 0% reduce 0%
07/09/13 11:49:21 INFO streaming.StreamJob: map 3% reduce 0%
07/09/13 11:49:23 INFO streaming.StreamJob: map 32% reduce 0%
07/09/13 11:49:24 INFO streaming.StreamJob: map 33% reduce 0%
07/09/13 11:49:27 INFO streaming.StreamJob: map 61% reduce 0%
07/09/13 11:49:30 INFO streaming.StreamJob: map 65% reduce 1%
07/09/13 11:49:31 INFO streaming.StreamJob: map 87% reduce 4%
07/09/13 11:49:33 INFO streaming.StreamJob: map 100% reduce 9%
07/09/13 11:49:34 INFO streaming.StreamJob: map 90% reduce 9%
07/09/13 11:49:37 INFO streaming.StreamJob: map 90% reduce 13%
07/09/13 11:49:40 INFO streaming.StreamJob: map 100% reduce 17%
07/09/13 11:49:41 INFO streaming.StreamJob: map 93% reduce 18%
07/09/13 11:49:42 INFO streaming.StreamJob: map 94% reduce 36%
07/09/13 11:49:45 INFO streaming.StreamJob: map 94% reduce 53%
07/09/13 11:49:46 INFO streaming.StreamJob: map 100% reduce 53%
07/09/13 11:50:20 INFO streaming.StreamJob: map 92% reduce 53%
07/09/13 11:50:22 INFO streaming.StreamJob: map 88% reduce 53%
07/09/13 11:50:24 INFO streaming.StreamJob: map 78% reduce 53%
07/09/13 11:50:25 INFO streaming.StreamJob: map 82% reduce 53%
07/09/13 11:50:26 INFO streaming.StreamJob: map 90% reduce 53%
07/09/13 11:50:30 INFO streaming.StreamJob: map 87% reduce 54%
07/09/13 11:50:31 INFO streaming.StreamJob: map 90% reduce 54%
07/09/13 11:50:32 INFO streaming.StreamJob: map 86% reduce 55%
07/09/13 11:50:33 INFO streaming.StreamJob: map 96% reduce 55%
07/09/13 11:50:35 INFO streaming.StreamJob: map 100% reduce 55%
07/09/13 11:50:40 INFO streaming.StreamJob: map 100% reduce 56%
07/09/13 11:50:41 INFO streaming.StreamJob: map 100% reduce 57%
07/09/13 11:50:42 INFO streaming.StreamJob: map 100% reduce 58%
07/09/13 11:51:31 INFO streaming.StreamJob: map 95% reduce 58%
07/09/13 11:51:34 INFO streaming.StreamJob: map 91% reduce 58%
07/09/13 11:51:40 INFO streaming.StreamJob: map 91% reduce 59%
07/09/13 11:51:41 INFO streaming.StreamJob: map 100% reduce 60%
07/09/13 11:51:44 INFO streaming.StreamJob: map 91% reduce 60%
07/09/13 11:51:50 INFO streaming.StreamJob: map 91% reduce 61%
07/09/13 11:51:51 INFO streaming.StreamJob: map 100% reduce 62%
Logs on map task side show errors like
Map output lost, rescheduling:
getMapOutput(task_200709131137_0003_m_000001_0,2) failed :
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:380)
at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1949)
..........
and Logs on reduce task side streaming errors like:
Got 0 new map outputs from tasktracker and 0 map outputs from previous
failures
2007-09-13 11:49:59,954 INFO org.apache.hadoop.mapred.ReduceTask:
task_200709131137_0003_r_000002_0 Got 12 known map output location(s);
scheduling...
--
View this message in context:
http://www.nabble.com/Hadoop-v-0.14.1%3A-3-different-test-runs---3-different-results-%21-%21--tf4435536.html#a12654323
Sent from the Hadoop Users mailing list archive at Nabble.com.
