I also tried increasing --num-executors to numNodes * coresPerNode and using coalesce(numNodes*10,true), and it still ran all the tasks on one node. It seems like it is placing all the executors on one node (though not always the same node, which indicates it is aware of more than one!). I'm using spark-submit --master yarn --deploy-mode cluster with spark-1.0.1 built for hadoop 2.4 on HDP 2.1/Hadoop 2.4.
There's clearly just something wrong with my Hadoop configuration, or in how I'm submitting my spark job - any suggestions? Thanks, Ravi -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Memory-compute-intensive-tasks-tp9643p10209.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
