Thanks Varun! Like we say in Brazil. "U are the guy!" (Você é o cara!)
I have another question. You said that: "yarn.nodemanager.resource.cpu-vcores on the other hand will have to be configured as per resource capability of that particular node. " I get the configuration from my job and printed it: yarn.nodemanager.resource.cpu-vcores 8 yarn.nodemanager.resource.memory-mb 8192 So how does hadoop get this property if it is per node? Does it get the minimum of all nodes? Thanks again! On Sun, Aug 23, 2015 at 4:40 PM, Varun Saxena <vsaxena.va...@gmail.com> wrote: > The fix would be released in next version(2.8.0). > I had checked the code to find out the default value and then found it > fixed in documentation(configuration list). > > As this is an unreleased version, a URL link (of the form > https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml) > may not be available AFAIK, > However, this XML(yarn-default.xml) can be checked online in git > repository. > > Associated JIRA which fixes this is > https://issues.apache.org/jira/browse/YARN-3823 > > Regards, > Varun Saxena. > > On Mon, Aug 24, 2015 at 12:53 AM, Pedro Magalhaes <pedror...@gmail.com> > wrote: > >> Thanks Varun! >> Could plz send me the link with the fixed? >> >> On Sun, Aug 23, 2015 at 2:20 PM, Varun Saxena <vsaxena.va...@gmail.com> >> wrote: >> >>> Hi Pedro, >>> >>> Real default value of yarn.scheduler.maximum-allocation-vcores is 4. >>> The value of 32 is actually a documentation issue and has been fixed >>> recently. >>> >>> Regards, >>> Varun Saxena. >>> >>> >>> On Sun, Aug 23, 2015 at 10:39 PM, Pedro Magalhaes <pedror...@gmail.com> >>> wrote: >>> >>>> Varun, >>>> Thanks for the reply. I undestand the arn.scheduler.maximum- >>>> allocation-vcores parameter. I just asking why the default parameter >>>> is yarn.scheduler.maximum-allocation-vcores=32. And >>>> yarn.nodemanager.resource.cpu-vcores=8. >>>> >>>> In my opinion, if the yarn.scheduler.maximun-allocation-vcore is 32 tby >>>> default the yarn.nodemanager.resource.cpu-vcores would be equal or greater >>>> than 32, by default. >>>> Is this make sense? >>>> >>>> >>>> >>>> >>>> On Sun, Aug 23, 2015 at 2:00 PM, Varun Saxena <vsaxena.va...@gmail.com> >>>> wrote: >>>> >>>>> Hi Pedro, >>>>> >>>>> Actual allocation would depend on the total resource capability >>>>> advertised by NM while registering with RM. >>>>> >>>>> yarn.scheduler.maximum-allocation-vcores merely puts an upper cap on >>>>> number of vcores which can be allocated by RM i.e. any Resource >>>>> request/ask from AM which asks for vcores > 32(default value) for a >>>>> container, will be normalized back to 32. >>>>> >>>>> If there is no such node available, this allocation will not be fulfilled. >>>>> >>>>> yarn.scheduler.maximum-allocation-vcores will be configured in >>>>> resource manager and hence will be common for a cluster which can possibly >>>>> have multiple nodes with heterogeneous resource capabilities >>>>> >>>>> yarn.nodemanager.resource.cpu-vcores on the other hand will have to be >>>>> configured as per resource capability of that particular node. >>>>> >>>>> Recently there has been work done to automatically get memory and CPU >>>>> information from underlying OS(supported OS being Linux and Windows) if >>>>> configured to do so. This change would be available in 2.8 >>>>> I hope this answers your question. >>>>> >>>>> Regards, >>>>> Varun Saxena. >>>>> >>>>> On Sun, Aug 23, 2015 at 9:40 PM, Pedro Magalhaes <pedror...@gmail.com> >>>>> wrote: >>>>> >>>>>> I was looking at default parameters for: >>>>>> >>>>>> yarn.nodemanager.resource.cpu-vcores = 8 >>>>>> yarn.scheduler.maximum-allocation-vcores = 32 >>>>>> >>>>>> For me this two parameters as default doesnt make any sense. >>>>>> >>>>>> The first one say "the number of CPU cores that can be allocated for >>>>>> containers." (I imagine that is vcore) The seconds says: "The maximum >>>>>> allocation for every container request at the RM". In my opinion, the >>>>>> second one must be equal or less than the first one. >>>>>> >>>>>> How can allocate 32 vcores for a container if i have only 8 cores >>>>>> available per container? >>>>>> >>>>> >>>>> >>>> >>> >> >
