* I apologize in advance for my bad English.
Hello. I have a question about determining optimal worker count. I have
been trying SimpleShortestPathsComputation example with Hadoop 2.5.1. And
noticed that the maximum value for -w parameter differs per configuration.
Let me explain: I have been running Giraph jobs on my local Ubuntu machine,
with Intel i7 cpu (4core) and 16gb ram. I can run 8 threads simultaneously
and has 9gb of free memory.
First, I configured Hadoop can run 4 mappers with 2gb of memory for each.
When I ran SimpleShortestPathsComputation, w=2 worked well but w=3 did not
work - It froze with 'map 75%'. That is, I could not use 2 mappers for
worker.
After that, I re-configured my Hadoop distribution to run 8 mappers with
1gb of memory for each. When I re-ran SimpleShortestPathsComputation, w=5
worked well but with w=6 it froze with 'map 86%'. In this case, I could not
use 3 mappers for worker.
I already know that there is an additional mapper, which works as
BSPMaster. However, It is still mysterious for me: Is there any hidden
process I don't know? Then, how many are there? 1 or 2?
Any guidance will be welcomed.
Thanks in advance,
Dongjin
PS - When I was working with Hadoop 1.x.x, I calculated the maximum worker
count as {available mappers} - 1 and it worked perfectly. I guess there are
some differences between 1.x.x and 2.x.x that I don't know.
--
*Dongjin Lee*
*Oracle Server Technologies Group.So interested in massive-scale machine
learning.facebook: www.facebook.com/dongjin.lee.kr
<http://www.facebook.com/dongjin.lee.kr>linkedin:
kr.linkedin.com/in/dongjinleekr
<http://kr.linkedin.com/in/dongjinleekr>github:
<http://goog_969573159>github.com/dongjinleekr
<http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
<http://www.twitter.com/dongjinleekr>*
