Re: problem running spark with yarn-client not using spark-submit

cc| > |"user @spark" <user@spark.apache.org> > | > | >

Re: problem running spark with yarn-client not using spark-submit

ject| |[Spam][SMG] Re: problem running spark with yarn

Re: problem running spark with yarn-client not using spark-submit

Hi, Trying to run spark with yarn-client not using spark-submit here what are you using to submit the job? spark-shell, spark-sql or anything else Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.

problem running spark with yarn-client not using spark-submit

Hello guys, Trying to run spark with yarn-client not using spark-submit here but the jobs kept failed while AM launching executor. The error collected by yarn like below. Looks like some environment setting is missing? Could someone help me out with this. Thanks in advance! HY Chung Java

Re: run multiple spark jobs yarn-client mode

Thank you for your help Mich. ThanksRajesh   Sent from Yahoo Mail. Get the app On Wednesday, May 25, 2016 3:14 PM, Mich Talebzadeh wrote: You may have some memory issues OOM etc that terminated the process. Dr Mich Talebzadeh LinkedIn  

Re: run multiple spark jobs yarn-client mode

You may have some memory issues OOM etc that terminated the process. Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: run multiple spark jobs yarn-client mode

Hi Friends, In the yarn log files of the nodemanager i can see the error below. Can i know why i am getting this error. ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: RECEIVED SIGNAL 15: SIGTERM ThanksRajesh   Sent from Yahoo Mail. Get the app On Wednesday, May 25, 2016 1:08 PM,

Re: run multiple spark jobs yarn-client mode

Yes check the yarn log files both resourcemanager and nodemanager. Also ensure that you have set up work directories consistently, especially yarn.nodemanager.local-dirs HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: run multiple spark jobs yarn-client mode

Could you check the yarn app logs ? On Wed, May 25, 2016 at 3:23 PM, wrote: > Hi, > > I am running spark streaming job on yarn-client mode. If run muliple jobs, > some of the jobs failing and giving below error message. Is there any > configuration missing? > >

run multiple spark jobs yarn-client mode

Hi, I am running spark streaming job on yarn-client mode. If run muliple jobs, some of the jobs failing and giving below error message. Is there any configuration missing? ERROR apache.spark.util.Utils - Uncaught exception in thread main java.lang.NullPointerException     at

Re: Running Spark on Yarn-Client/Cluster mode

unch Spark-Shell in yarn-client mode > > Any suggestion ? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster-mode-tp26691p26752.html > Sent from the Apache Spark U

Re: Running Spark on Yarn-Client/Cluster mode

I have updated all my nodes in the Cluster to have 4GB RAM memory , but still face the same error when trying to launch Spark-Shell in yarn-client mode Any suggestion ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster

Re: Running Spark on Yarn-Client/Cluster mode

And then issued the following command to run spark-shell in yarn client mode , spark-shell --executor-memory 512m --driver-memory 1g --num-executors 2 But i am still unable to start the Spark context and it fails with the same error. Can someone help me on explaining how to set the core, executor

Re: Running Spark on Yarn-Client/Cluster mode

0.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster-mode-tp26691p26717.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional command

Re: Running Spark on Yarn-Client/Cluster mode

Few more added information with Nodes Memory and Core ptfhadoop01v - 4GB ntpcam01v - 1GB ntpcam03v - 2GB Each of the VM has only 1 core CPU -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster-mode-tp26691p26714.html Sent

Re: Running Spark on Yarn-Client/Cluster mode

d then Again try to re-issue the same command to start Spark on Yarn-Client then it does not even start and takes me back to the error message posted earlier. <http://apache-spark-user-list.1001560.n3.nabble.com/file/n26713/Spark-error2.jpg> I have no idea on what is causing this. -- View t

Re: Running Spark on Yarn-Client/Cluster mode

560.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster-mode-tp26691p26710.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands,

Re: Running Spark on Yarn-Client/Cluster mode

RK_JAR variable in spark-env.sh file but no success. I also tried using the below command , *spark-shell --master yarn-client --conf spark.yarn.jar=hdfs://ptfhadoop01v:8020/user/spark/share/lib/spark-assembly.jar* Issuing this command gives me the following error message , Spark-Error.txt <http

Re: Running Spark on Yarn-Client/Cluster mode

sembly.jar -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster-mode-tp26691p26704.html Sent from the Apache Spark User List mailing list archive at

Running Spark on Yarn-Client/Cluster mode

y to initiate the same in yarn-client mode it always fails , the command i used is , $ *spark-shell --master yarn-client* Spark-Error.txt <http://apache-spark-user-list.1001560.n3.nabble.com/file/n26691/Spark-Error.txt> Can anyone tell me what am i doing wrong , Do i need to install spark o

Re: Spark with Yarn Client

onfiguration of spark with yarn > client on hadoop cluster . > Can somebody help me or point me document /blog/books which has deeper > understanding of above two. > Thanks, > Divya >

Spark with Yarn Client

Hi, I am trying to understand behaviour /configuration of spark with yarn client on hadoop cluster . Can somebody help me or point me document /blog/books which has deeper understanding of above two. Thanks, Divya

Re: Spark 1.5.2 - Programmatically launching spark on yarn-client mode

1.3.1 artifact / dependency leaked into your > app ? > > Cheers > > On Thu, Jan 28, 2016 at 7:36 PM, Nirav Patel <npa...@xactlycorp.com> > wrote: > >> Hi, we were using spark 1.3.1 and launching our spark jobs on yarn-client >> mode programmatically via cr

Programmatically launching spark on yarn-client mode no longer works in spark 1.5.2

Hi, we were using spark 1.3.1 and launching our spark jobs on yarn-client mode programmatically via creating a sparkConf and sparkContext object manually. It was inspired from spark self-contained application example here: https://spark.apache.org/docs/1.5.2/quick-start.html#self-contained

Re: Programmatically launching spark on yarn-client mode no longer works in spark 1.5.2

Hi, we were using spark 1.3.1 and launching our spark jobs on yarn-client >> mode programmatically via creating a sparkConf and sparkContext object >> manually. It was inspired from spark self-contained application example >> here: >> >> >> https://spark.

Re: Programmatically launching spark on yarn-client mode no longer works in spark 1.5.2

> >> I think I met this problem before, this problem might be due to some race >> conditions in exit period. The way you mentioned is still valid, this >> problem only occurs when stopping the application. >> >> Thanks >> Saisai >> >> On Fri, Ja

Spark 1.5.2 - Programmatically launching spark on yarn-client mode

Hi, we were using spark 1.3.1 and launching our spark jobs on yarn-client mode programmatically via creating a sparkConf and sparkContext object manually. It was inspired from spark self-contained application example here: https://spark.apache.org/docs/1.5.2/quick-start.html#self-contained

Re: Programmatically launching spark on yarn-client mode no longer works in spark 1.5.2

t; Hi, we were using spark 1.3.1 and launching our spark jobs on yarn-client > mode programmatically via creating a sparkConf and sparkContext object > manually. It was inspired from spark self-contained application example > here: > > > https://spark.apache.org/docs/1.5.2/quic

Re: Spark 1.5.2 - Programmatically launching spark on yarn-client mode

Looks like '--properties-file' is no longer supported. Was it possible that Spark 1.3.1 artifact / dependency leaked into your app ? Cheers On Thu, Jan 28, 2016 at 7:36 PM, Nirav Patel <npa...@xactlycorp.com> wrote: > Hi, we were using spark 1.3.1 and launching our spark jobs on ya

Re: Running Spark in Yarn-client mode

t; Regards > JB > > On 10/08/2015 07:23 AM, Sushrut Ikhar wrote: > >> Hi, >> I am new to Spark and I have been trying to run Spark in yarn-client mode. >> >> I get this error in yarn logs : >> Error: Could not find or load main class >> org.apache.s

Running Spark in Yarn-client mode

Hi, I am new to Spark and I have been trying to run Spark in yarn-client mode. I get this error in yarn logs : Error: Could not find or load main class org.apache.spark.executor.CoarseGrainedExecutorBackend Also, I keep getting these warnings: WARN YarnScheduler: Initial job has not accepted

Re: Running Spark in Yarn-client mode

Hi Sushrut, which packaging of Spark do you use ? Do you have a working Yarn cluster (with at least one worker) ? spark-hadoop-x ? Regards JB On 10/08/2015 07:23 AM, Sushrut Ikhar wrote: Hi, I am new to Spark and I have been trying to run Spark in yarn-client mode. I get this error in yarn

Re: Spark-sql(yarn-client) java.lang.NoClassDefFoundError: org/apache/spark/deploy/yarn/ExecutorLauncher

btw, user listt will be a better place for this thread. On Thu, Jun 18, 2015 at 8:19 AM, Yin Huai yh...@databricks.com wrote: Is it the full stack trace? On Thu, Jun 18, 2015 at 6:39 AM, Sea 261810...@qq.com wrote: Hi, all: I want to run spark sql on yarn(yarn-client), but ... I already

Spark-sql(yarn-client) java.lang.NoClassDefFoundError: org/apache/spark/deploy/yarn/ExecutorLauncher

Hi, all: I want to run spark sql on yarn(yarn-client), but ... I already set spark.yarn.jar and spark.jars in conf/spark-defaults.conf. ./bin/spark-sql -f game.sql --executor-memory 2g --num-executors 100 game.txt Exception in thread main java.lang.NoClassDefFoundError:

spark/yarn ignoring num-executors (python, Amazon EMR, spark-submit, yarn-client)

are accessible by Yarn (when running the applications repeatedly I've seen all five nodes in use in various attempts.) The applications are written in python and are run via spark-submit with yarn-client as the master. Example application submission: bin/spark-submit --num-executors 1 --conf