as far as I understand, with YARN there are two additional task to the workers: one for the master and one for the application manager. I'm not sure, but maybe Eli can confirm.
On Fri, Jan 16, 2015 at 2:16 PM, Dongjin Lee <[email protected]> wrote: > * I apologize in advance for my bad English. > > Hello. I have a question about determining optimal worker count. I have > been trying SimpleShortestPathsComputation example with Hadoop 2.5.1. And > noticed that the maximum value for -w parameter differs per configuration. > > Let me explain: I have been running Giraph jobs on my local Ubuntu machine, > with Intel i7 cpu (4core) and 16gb ram. I can run 8 threads simultaneously > and has 9gb of free memory. > > First, I configured Hadoop can run 4 mappers with 2gb of memory for each. > When I ran SimpleShortestPathsComputation, w=2 worked well but w=3 did not > work - It froze with 'map 75%'. That is, I could not use 2 mappers for > worker. > > After that, I re-configured my Hadoop distribution to run 8 mappers with > 1gb of memory for each. When I re-ran SimpleShortestPathsComputation, w=5 > worked well but with w=6 it froze with 'map 86%'. In this case, I could not > use 3 mappers for worker. > > I already know that there is an additional mapper, which works as > BSPMaster. However, It is still mysterious for me: Is there any hidden > process I don't know? Then, how many are there? 1 or 2? > > Any guidance will be welcomed. > > Thanks in advance, > Dongjin > > PS - When I was working with Hadoop 1.x.x, I calculated the maximum worker > count as {available mappers} - 1 and it worked perfectly. I guess there are > some differences between 1.x.x and 2.x.x that I don't know. > > -- > *Dongjin Lee* > > > *Oracle Server Technologies Group.So interested in massive-scale machine > learning.facebook: www.facebook.com/dongjin.lee.kr > <http://www.facebook.com/dongjin.lee.kr>linkedin: > kr.linkedin.com/in/dongjinleekr > <http://kr.linkedin.com/in/dongjinleekr>github: > <http://goog_969573159>github.com/dongjinleekr > <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr > <http://www.twitter.com/dongjinleekr>* > -- Claudio Martella
