[
https://issues.apache.org/jira/browse/MAPREDUCE-712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728282#action_12728282
]
Khaled Elmeleegy commented on MAPREDUCE-712:
--------------------------------------------
Well, 50TB/100k maps is 512 MB/map. I also tried 256MB/map and the result is
the same.
Another interesting data point is I tried the read case, using the following
command:
./hadoop jar ../hadoop-*test.jar loadgen -keepmap 0.2 -keepred 5
-inFormat org.apache.hadoop.mapred.SequenceFileInputFormat -outFormat
org.apache.hadoop.mapred.SequenceFileOutputFormat -outKey
org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -indir
/gridmix/data/WebSimulationBlock -outdir /out
My input files are those produced by the previous TextWriter job.
Interestingly, for this case the disk is clearly the bottleneck. Worker's
CPUs are ~50% idle + ~20% waiting for I/O, i.e. The CPUs are only 30% busy.
All disks are virtually saturated.
My guess for the discrepancy between the read and the write case is due to
the fact that the datanode uses sendfile for the read case. Maps don't/can't
use sendfile though for the write case. I'll try to look deeper into this.
> TextWritter example is CPU bound!!
> ----------------------------------
>
> Key: MAPREDUCE-712
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-712
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.1, 0.21.0
> Environment: ~200 nodes cluster
> Each node has the following configuration:
> Processors: 2 x Xeon L5420 2.50GHz (8 cores) - Harpertown C0, 64-bit,
> quad-core (8 CPUs)
> 4 Disks
> 16 GB RAM
> Linux 2.6
> Hadoop version: trunk
> Reporter: Khaled Elmeleegy
>
> Running the RandomTextWritter example job ( from the examples jar) pegs the
> machiens' CPUs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
