Master Node details: lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 4 Core(s) per socket: 1 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 62 Model name: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz Stepping: 4 CPU MHz: 2494.066 BogoMIPS: 4988.13 Hypervisor vendor: Xen Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 25600K NUMA node0 CPU(s): 0-3
Slave Node Details: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 8 Core(s) per socket: 1 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 62 Model name: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz Stepping: 4 CPU MHz: 2500.054 BogoMIPS: 5000.10 Hypervisor vendor: Xen Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 25600K NUMA node0 CPU(s): 0-7 On Mon, Feb 26, 2018 at 10:20 AM, Selvam Raman <sel...@gmail.com> wrote: > Hi, > > spark version - 2.0.0 > spark distribution - EMR 5.0.0 > > Spark Cluster - one master, 5 slaves > > Master node - m3.xlarge - 8 vCore, 15 GiB memory, 80 SSD GB storage > Slave node - m3.2xlarge - 16 vCore, 30 GiB memory, 160 SSD GB storage > > > Cluster Metrics > Apps SubmittedApps PendingApps RunningApps CompletedContainers RunningMemory > UsedMemory TotalMemory ReservedVCores UsedVCores TotalVCores ReservedActive > NodesDecommissioning NodesDecommissioned NodesLost NodesUnhealthy > NodesRebooted > Nodes > 16 0 1 15 5 88.88 GB 90.50 GB 22 GB 5 79 1 5 > <http://localhost:8088/cluster/nodes> 0 > <http://localhost:8088/cluster/nodes/decommissioning> 0 > <http://localhost:8088/cluster/nodes/decommissioned> 5 > <http://localhost:8088/cluster/nodes/lost> 0 > <http://localhost:8088/cluster/nodes/unhealthy> 0 > <http://localhost:8088/cluster/nodes/rebooted> > I have submitted job with below configuration > --num-executors 5 --executor-cores 10 --executor-memory 20g > > > > spark.task.cpus - be default 1 > > > My understanding is there will be 5 executore each can run 10 task at a > time and task can share total memory of 20g. Here, i could see only 5 > vcores used which means 1 executor instance use 20g+10%overhead ram(22gb), > 10 core(number of threads), 1 Vcore(cpu). > > please correct me if my understand is wrong. > > how can i utilize number of vcore in EMR effectively. Will Vcore boost > performance? > > > -- > Selvam Raman > "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து" > -- Selvam Raman "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
