Bartosz Gadzimski wrote:
As Arkadi said, your hdd is to slow for 2 x quad core processor. I have
the same problem and now thinking of using more boxes or very fast
drives (sas 15k).
Raymond Balmčs pisze:
Well I suspect the sort function is mono-threaded as usually they are so
only one core is used 25% is the max you will get.
I have a dual core and it only goes to 50% CPU in many of the steps ... I
assumed that some phases are mono-threaded.
Folks,
From your conversation I suspect that you are running Hadoop with
LocalJobtracker, i.e. in a single JVM - correct?
While this works ok for small datasets, you don't really benefit from
map-reduce parallelism (and you still pay the penalty for the
overheads). As your dataset grows, you will quickly reach the
scalability limits - in this case, the limit of IO throughput of a
single drive, during the sort phase of a large dataset. The excessive IO
demands can be solved by distributing the load (over many drives, and
over many machines), which is what HDFS is designed to do well.
Hadoop tasks are usually single-threaded, and additionally
LocalJobTracker implements only a primitive non-parallel model of task
execution - i.e. each task is scheduled to run sequentially in turn. If
you run the regular distributed JobTracker, Hadoop splits the load among
many tasks running in parallel.
So, the solution is this: set up a distributed Hadoop cluster, even if
it's going to consist of a single node - because then the data will be
split and processed in parallel by several JVM instances. This will also
help the operating system to schedule these processes over multiple
CPU-s. Additionally, if you still experience IO contention, consider
moving to HDFS as the filestystem, and spread it over more than 1
machine and more than 1 disk in each machine.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
