spark yarn client mode

Hi Do I need to install spark on all the yarn cluster node if I want to submit the job to yarn client? is there any way exists in which I can spawn a spark job executors on the cluster nodes where I have not installed spark. Thanks Sanjeev

Re: spark yarn client mode

Hi, No,you don't need to. However,when submitting jobs certain resources will be uploaded to hdfs,which could be a performance issue read the log and you will understand: 15/12/29 11:10:06 INFO Client: Uploading resource file:/data/spark/spark152/lib/spark-assembly-1.5.2-hadoop2.6.0.jar -> hdfs

Re: strange behavior in spark yarn-client mode

On Thu, Jan 14, 2016 at 10:17 AM, Sanjeev Verma wrote: > now it spawn a single executors with 1060M size, I am not able to understand > why this time it executes executors with 1G+overhead not 2G what I > specified. Where are you looking for the memory size for the

Re: strange behavior in spark yarn-client mode

Please reply to the list. The web ui does not show the total size of the executor's heap. It shows the amount of memory available for caching data, which is, give or take, 60% of the heap by default. On Thu, Jan 14, 2016 at 11:03 AM, Sanjeev Verma wrote: > I am

strange behavior in spark yarn-client mode

I am seeing a strange behaviour while running spark in yarn client mode.I am observing this on the single node yarn cluster.in spark-default I have configured the executors memory as 2g and started the spark shell as follows bin/spark-shell --master yarn-client which trigger the 2 executors on

Re: Spark (yarn-client mode) Hangs in final stages of Collect or Reduce

Have you checked the corresponding executor logs as well? I think information provided by you here is less to actually understand your issue. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-yarn-client-mode-Hangs-in-final-stages-of-Collect-or-Reduce