Re: Spark EMR executor-core vs Vcores

hi, all I also noticed this problem. The reason is that Yarn accounts each executor for only 1, no matter how many cores you configured. Because Yarn only uses memory as the primary metrics for resource allocation. It means that Yarn will pack as many as executors on each node as long as the

Re: Spark EMR executor-core vs Vcores

: user <user@spark.apache.org> Subject: Re: Spark EMR executor-core vs Vcores Thanks. That’s make sense. I want to know one more think , available vcore per machine is 16 but threads per node 8. Am I missing to relate here. What I m thinking now is number of vote = number of threads. On Mon, 26

Re: Spark EMR executor-core vs Vcores

yeah, for some reason (unknown to me, but you can find on aws forums) they double the actual number of cores for nodemanagers. I assume that's done to maximize utilization, but doesn't really matter to me, at least, since I only run Spark, so I, personally, set `total number of cores - 1/2`

Re: Spark EMR executor-core vs Vcores

Thanks. That’s make sense. I want to know one more think , available vcore per machine is 16 but threads per node 8. Am I missing to relate here. What I m thinking now is number of vote = number of threads. On Mon, 26 Feb 2018 at 18:45, Vadim Semenov wrote: > All used

Re: Spark EMR executor-core vs Vcores

Putting all cores won't solve the purpose alone, you'll have to mention executors as well executor memory accordingly to it.. On Tue 27 Feb, 2018, 12:15 AM Vadim Semenov, wrote: > All used cores aren't getting reported correctly in EMR, and YARN itself > has no control over

Re: Spark EMR executor-core vs Vcores

All used cores aren't getting reported correctly in EMR, and YARN itself has no control over it, so whatever you put in `spark.executor.cores` will be used, but in the ResourceManager you will only see 1 vcore used per nodemanager. On Mon, Feb 26, 2018 at 5:20 AM, Selvam Raman

Re: Spark EMR executor-core vs Vcores

Hi Fawze, Yes, it is true that i am running in yarn mode, 5 containers represents 4executor and 1 master. But i am not expecting this details as i already aware of this. What i want to know is relationship between Vcores(Emr yarn) vs executor-core(Spark). >From my slave configuration i

Re: Spark EMR executor-core vs Vcores

It's recommended to sue executor-cores of 5. Each executor here will utilize 20 GB which mean the spark job will utilize 50 cpu cores and 100GB memory. You can not run more than 4 executors because your cluster doesn't have enough memory. Use see 5 executor because 4 for the job and one for the

Re: Spark EMR executor-core vs Vcores

Master Node details: lscpu Architecture: x86_64 CPU op-mode(s):32-bit, 64-bit Byte Order:Little Endian CPU(s):4 On-line CPU(s) list: 0-3 Thread(s) per core:4 Core(s) per socket:1 Socket(s): 1 NUMA node(s): 1 Vendor ID:

Spark EMR executor-core vs Vcores

Hi, spark version - 2.0.0 spark distribution - EMR 5.0.0 Spark Cluster - one master, 5 slaves Master node - m3.xlarge - 8 vCore, 15 GiB memory, 80 SSD GB storage Slave node - m3.2xlarge - 16 vCore, 30 GiB memory, 160 SSD GB storage Cluster Metrics Apps SubmittedApps PendingApps RunningApps