YARN cores are virtual cores which are used just to calculate available resources. But usually memory is used to manage yarn resources (not cores)
spark executor memory should be ~90% of yarn.scheduler.maximum-allocation-mb (which should be the same as yarn.nodemanager.resource.memory-mb) ~10% should go to executor memory overhead e.g. for r3.2xlarge slave (61GB / 8 cores) the setting are the following (on EMR-4.2.0) yarn settings: yarn.nodemanager.resource.memory-mb = 54272 yarn.scheduler.maximum-allocation-mb = 54272 spark settings: spark.executor.memory = 47924M spark.yarn.executor.memoryOverhead = 5324 spark.executor.cores = 8 // cores available on each slave 1024M of YARN memory is reserved on each box to run Spark AM container(s) - Spark AM container uses 896 MB of yarn memory (AM used in both client and cluster mode) On Fri, Mar 11, 2016 at 2:08 PM, Koert Kuipers <ko...@tresata.com> wrote: > you get a spark executor per yarn container. the spark executor can have > multiple cores, yes. this is configurable. so the number of partitions that > can be processed in parallel is num-executors * executor-cores. and for > processing a partition the available memory is executor-memory / > executor-cores (roughly, cores can of course borrow memory from each other > within executor). > > the relevant setting for spark-submit are: > --executor-memory > --executor-cores > --num-executors > > On Fri, Mar 11, 2016 at 4:58 PM, Mich Talebzadeh < > mich.talebza...@gmail.com> wrote: > >> Hi, >> >> Can these be clarified please >> >> >> 1. Can a YARN container use more than one core and if this is >> configurable? >> 2. A YARN container is constraint to 8MB by >> " yarn.scheduler.maximum-allocation-mb". If a YARN container is a Spark >> process will that limit also include the memory Spark going to be using? >> >> Thanks, >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> > >
