[
https://issues.apache.org/jira/browse/MAPREDUCE-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khaled Elmeleegy updated MAPREDUCE-712:
---------------------------------------
I used the following command line for the RandomTextWritter job:
./hadoop jar ../hadoop-0.21.0-dev-*examples.jar randomtextwriter -D
test.randomtextwrite.total_bytes=53687091200000 -D
test.randomtextwrite.bytes_per_map=536870912 -D
test.randomtextwrite.min_words_key=5 -D
test.randomtextwrite.max_words_key=10 -D
test.randomtextwrite.min_words_value=100 -D
test.randomtextwrite.max_words_value=10000 -D mapred.output.compress=false
-D mapred.map.output.compression.type=BLOCK -outFormat
org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat
/gridmix/data/WebSimulationBlock
The job has 100,000 maps and no reduces. I configured HDFS to have replication
factor of 1 to eliminate network traffic. Nodes were configured to have 16 map
slots and 2 reduce slots. Each task was configured to have at most 512MB of
java heap space. The jobs output file is ~50TB >> overall cluster memory,
forcing disk I/O
Since the job is doing no computation other than just writing to disk, one
would expect that the job would be totally i/o (disk) bound. Surprisingly, it
turned out to be CPU bound.
Measurements (using chukwa):
Across the cluster, workers cpu was <5% idle on average. Used disk bandwidth
was ~40 MB/s across all disks for all the nodes at the cluster, which is close
to the practical disk BW limit. The network is virtually 100% idle as one would
expect.
The CPU time was ~70% at the user space, suggesting it's mainly overhead in the
map tasks.
This suggests that there is a lot of CPU fat in the map tasks.
> TextWritter example is CPU bound!!
> ----------------------------------
>
> Key: MAPREDUCE-712
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-712
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.1, 0.21.0
> Environment: ~200 nodes cluster
> Each node has the following configuration:
> Processors: 2 x Xeon L5420 2.50GHz (8 cores) - Harpertown C0, 64-bit,
> quad-core (8 CPUs)
> 4 Disks
> 16 GB RAM
> Linux 2.6
> Hadoop version: trunk
> Reporter: Khaled Elmeleegy
>
> Running the RandomTextWritter example job ( from the examples jar) pegs the
> machiens' CPUs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
