HI Our team has a 40 node hortonworks Hadoop cluster 2.2.4.2-2 (36 data node) with apache spark 1.2 and 1.4 installed. Each node has 64G RAM and 8 cores.
We are only able to use <= 72 executors with executor-cores=2 So we are only get 144 active tasks running pyspark programs with pyspark. [Stage 1:===============> (596 + 144) / 2042] IF we use larger number for --num-executors, the pyspark program exit with errors: ERROR YarnScheduler: Lost executor 113 on hag017.example.com: remote Rpc client disassociated I tried spark 1.4 and conf.set("dynamicAllocation.enabled", "true"). However it does not help us to increase the number of active tasks. I expect larger number of active tasks with the cluster we have. Could anyone advise on this? Thank you very much! Shaun