driver in queued state and not started

Hi Guys, I tried to run spark on mesos cluster. However, when I tried to submit jobs via spark-submit. The driver is in "Queued state" and not started. Which should I check? Thanks, Jared, (??) Software developer Interested in open source software, big data, Linux

Two questions about running spark on mesos

Hi Guys, Two questions about running spark on mesos. 1, Does spark configuration of conf/slaves still work when running spark on mesos? According to my observations, it seemed that conf/slaves still took effect when running spark-shell. However, it doesn't take effect when deploying

Re: A question about Spark Cluster vs Local Mode

If cluster runs out of memory, it seems that the executor will be restarted by cluster manager. Jared, (韦煜） Software developer Interested in open source software, big data, Linux From: Ascot Moss Sent: Thursday, July 28, 2016 9:48:13 AM

yarn.exceptions.ApplicationAttemptNotFoundException when trying to shut down spark applicaiton via yarn applicaiton --kill

Hi guys, When I tried to shut down spark application via "yarn application --kill". I run the spark application in yarn cluster mode in my laptop. I found following exception in log. org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: Application attempt

Re: Is it good choice to use DAO to store results generated by spark application?

data. We need to extend spark application to query and analysis data later. Thx, Jared From: Ted Yu <yuzhih...@gmail.com> Sent: Wednesday, July 20, 2016 10:34:15 PM To: Yu Wei Cc: ayan guha; Rabin Banerjee; user; Deepak Sharma Subject: Re: Is it good

Re: Is it good choice to use DAO to store results generated by spark application?

:24 PM To: Rabin Banerjee Cc: user; Yu Wei; Deepak Sharma Subject: Re: Is it good choice to use DAO to store results generated by spark application? Just as a rain check, saving data to hbase for analytics may not be the best choice. Any specific reason for not using hdfs or hive? On 20 Jul 2016

Re: Is it good choice to use DAO to store results generated by spark application?

kmc...@gmail.com> Sent: Wednesday, July 20, 2016 12:34:07 PM To: Yu Wei Cc: spark users Subject: Re: Is it good choice to use DAO to store results generated by spark application? I am using DAO in spark application to write the final computation to Cassandra and it performs well. What kinds of

Re: Is it good choice to use DAO to store results generated by spark application?

From: Ted Yu <yuzhih...@gmail.com> Sent: Wednesday, July 20, 2016 12:14:17 PM To: Andrew Ehrlich Cc: Yu Wei; user@spark.apache.org Subject: Re: Is it good choice to use DAO to store results generated by spark application? hbase-spark module is in the up-coming hbase 2.0 r

Is it good choice to use DAO to store results generated by spark application?

Hi guys, I write spark application and want to store results generated by spark application to hbase. Do I need to access hbase via java api directly? Or is it better choice to use DAO similar as traditional RDBMS? I suspect that there is major performance downgrade and other negative

Re: Is that possible to launch spark streaming application on yarn with only one machine?

Thanks very much for your help. Finally I understood the deploy mode with your explanation after trying different approach on my development environment. Thanks again. From: Yu Wei <yu20...@hotmail.com> Sent: Saturday, July 9, 2016 3:04:40 PM To: Rabin Ba

Re: Is that possible to launch spark streaming application on yarn with only one machine?

I tried to flush the information to external system in cluster mode. It works well. I suspect that in yarn cluster mode, stdout is closed. From: Rabin Banerjee <dev.rabin.baner...@gmail.com> Sent: Saturday, July 9, 2016 4:22:10 AM To: Yu Wei Cc

Re: Is that possible to launch spark streaming application on yarn with only one machine?

How could I dump data into text file? Writing to HDFS or other approach? Thanks, Jared From: Rabin Banerjee <dev.rabin.baner...@gmail.com> Sent: Thursday, July 7, 2016 7:04:29 PM To: Yu Wei Cc: Mich Talebzadeh; user; Deng Ching-Mallete Subje

Re: Is that possible to launch spark streaming application on yarn with only one machine?

From: Rabin Banerjee <dev.rabin.baner...@gmail.com> Sent: Thursday, July 7, 2016 1:04 PM To: Yu Wei Cc: Mich Talebzadeh; Deng Ching-Mallete; user@spark.apache.org Subject: Re: Is that possible to launch spark streaming application on yarn with only one machine? In yarn cluste

Re: Is that possible to launch spark streaming application on yarn with only one machine?

:46:11 PM To: Yu Wei Cc: Deng Ching-Mallete; user@spark.apache.org Subject: Re: Is that possible to launch spark streaming application on yarn with only one machine? Deploy-mode cluster don't think will work. Try --master yarn --deploy-mode client FYI * Spark Local - Spark runs on the loca

Re: It seemed JavaDStream.print() did not work when launching via yarn on a single node

) firstNum.take(num).foreach(println) if (firstNum.length > num) println("...") println() // scalastyle:on println } } foreachRDD(context.sparkContext.clean(foreachFunc), displayInnerRDDOps = false) } On Wed, Jul 6, 2016 at 9:17 PM, Yu Wei <yu20...@hotmail.com<

Re: It seemed JavaDStream.print() did not work when launching via yarn on a single node

How about DStream.print(). Does it invoke collect before print on driver? From: Sean Owen <so...@cloudera.com> Sent: Wednesday, July 6, 2016 8:20:36 PM To: Rabin Banerjee Cc: Yu Wei; user@spark.apache.org Subject: Re: It seemed JavaDStream.print() did no

It seemed JavaDStream.print() did not work when launching via yarn on a single node

Hi guys, It seemed that when launching application via yarn on single node, JavaDStream.print() did not work. However, occasionally it worked. If launch the same application in local mode, it always worked. The code is as below, SparkConf conf = new SparkConf().setAppName("Monitor");

Re: Is that possible to launch spark streaming application on yarn with only one machine?

--driver-memory 4g --executor-memory 2g --num-executors 4 target/CollAna-1.0-SNAPSHOT.jar Any advice? Thanks, Jared ________ From: Yu Wei <yu20...@hotmail.com> Sent: Tuesday, July 5, 2016 4:41 PM To: Deng Ching-Mallete Cc: user@spark.apache.org Subject:

Re: Is that possible to launch spark streaming application on yarn with only one machine?

works now. Thanks again, Jared From: odeach...@gmail.com <odeach...@gmail.com> on behalf of Deng Ching-Mallete <och...@apache.org> Sent: Tuesday, July 5, 2016 4:03:28 PM To: Yu Wei Cc: user@spark.apache.org Subject: Re: Is that possible to la

Is that possible to launch spark streaming application on yarn with only one machine?

Hi guys, I set up pseudo hadoop/yarn cluster on my labtop. I wrote a simple spark streaming program as below to receive messages with MQTTUtils. conf = new SparkConf().setAppName("Monitor"); jssc = new JavaStreamingContext(conf, Durations.seconds(1)); JavaReceiverInputDStream inputDS =

Is that possible to feed web request via spark application directly?

Hi, I'm learning spark recently. I have one question about spark. Is it possible to feed web requests via spark application directly? Is there any library to be used? Or do I need to write the results from spark to HDFS/HBase? Is one spark application only to be designed to implement one