OK. I will take more experiments and let you know. Thanks, Dongjin
On Mon, Feb 9, 2015 at 4:46 AM, Eli Reisman <[email protected]> wrote: > That's correct. There were some changes last year as to. How these worker > containers are allocated that were supposed to fix another similar issue > with the master task. Perhaps this introduced another bug. I'm not sure > there's enough info to be sure can you give more details? Anyone else > seeing this issue? And to confirm this was only when you run on a single > machine? > On Jan 17, 2015 12:09 PM, "Claudio Martella" <[email protected]> > wrote: > > > as far as I understand, with YARN there are two additional task to the > > workers: one for the master and one for the application manager. I'm not > > sure, but maybe Eli can confirm. > > > > On Fri, Jan 16, 2015 at 2:16 PM, Dongjin Lee <[email protected]> > > wrote: > > > > > * I apologize in advance for my bad English. > > > > > > Hello. I have a question about determining optimal worker count. I have > > > been trying SimpleShortestPathsComputation example with Hadoop 2.5.1. > And > > > noticed that the maximum value for -w parameter differs per > > configuration. > > > > > > Let me explain: I have been running Giraph jobs on my local Ubuntu > > machine, > > > with Intel i7 cpu (4core) and 16gb ram. I can run 8 threads > > simultaneously > > > and has 9gb of free memory. > > > > > > First, I configured Hadoop can run 4 mappers with 2gb of memory for > each. > > > When I ran SimpleShortestPathsComputation, w=2 worked well but w=3 did > > not > > > work - It froze with 'map 75%'. That is, I could not use 2 mappers for > > > worker. > > > > > > After that, I re-configured my Hadoop distribution to run 8 mappers > with > > > 1gb of memory for each. When I re-ran SimpleShortestPathsComputation, > w=5 > > > worked well but with w=6 it froze with 'map 86%'. In this case, I could > > not > > > use 3 mappers for worker. > > > > > > I already know that there is an additional mapper, which works as > > > BSPMaster. However, It is still mysterious for me: Is there any hidden > > > process I don't know? Then, how many are there? 1 or 2? > > > > > > Any guidance will be welcomed. > > > > > > Thanks in advance, > > > Dongjin > > > > > > PS - When I was working with Hadoop 1.x.x, I calculated the maximum > > worker > > > count as {available mappers} - 1 and it worked perfectly. I guess there > > are > > > some differences between 1.x.x and 2.x.x that I don't know. > > > > > > -- > > > *Dongjin Lee* > > > > > > > > > *Oracle Server Technologies Group.So interested in massive-scale > machine > > > learning.facebook: www.facebook.com/dongjin.lee.kr > > > <http://www.facebook.com/dongjin.lee.kr>linkedin: > > > kr.linkedin.com/in/dongjinleekr > > > <http://kr.linkedin.com/in/dongjinleekr>github: > > > <http://goog_969573159>github.com/dongjinleekr > > > <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr > > > <http://www.twitter.com/dongjinleekr>* > > > > > > > > > > > -- > > Claudio Martella > > > -- *Dongjin Lee* *Oracle Server Technologies Group.So interested in massive-scale machine learning.facebook: www.facebook.com/dongjin.lee.kr <http://www.facebook.com/dongjin.lee.kr>linkedin: kr.linkedin.com/in/dongjinleekr <http://kr.linkedin.com/in/dongjinleekr>github: <http://goog_969573159>github.com/dongjinleekr <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr <http://www.twitter.com/dongjinleekr>*
