Re: Is that possible to launch spark streaming application on yarn with only one machine?

Thanks very much for your help. Finally I understood the deploy mode with your explanation after trying different approach on my development environment. Thanks again. From: Yu Wei Sent: Saturday, July 9, 2016 3:04:40 PM To: Rabin Banerjee

Re: Is that possible to launch spark streaming application on yarn with only one machine?

I tried to flush the information to external system in cluster mode. It works well. I suspect that in yarn cluster mode, stdout is closed. From: Rabin Banerjee Sent: Saturday, July 9, 2016 4:22:10 AM To: Yu Wei Cc: Mich

Re: Is that possible to launch spark streaming application on yarn with only one machine?

Ya , I mean dump in hdfs as a file ,via yarn cluster mode . On Jul 8, 2016 3:10 PM, "Yu Wei" wrote: > How could I dump data into text file? Writing to HDFS or other approach? > > > Thanks, > > Jared > -- > *From:* Rabin Banerjee

Re: Is that possible to launch spark streaming application on yarn with only one machine?

How could I dump data into text file? Writing to HDFS or other approach? Thanks, Jared From: Rabin Banerjee Sent: Thursday, July 7, 2016 7:04:29 PM To: Yu Wei Cc: Mich Talebzadeh; user; Deng Ching-Mallete Subject: Re: Is that

Re: Is that possible to launch spark streaming application on yarn with only one machine?

In that case, I suspect that Mqtt is not getting data while you are submitting in yarn cluster . Can you please try dumping data in text file instead of printing while submitting in yarn cluster mode.? On Jul 7, 2016 12:46 PM, "Yu Wei" wrote: > Yes. Thanks for your

Re: Is that possible to launch spark streaming application on yarn with only one machine?

Yes. Thanks for your clarification. The problem I encountered is that in yarn cluster mode, no output for "DStream.print()" in yarn logs. In spark implementation org/apache/spark/streaming/dstream/DStream.scala, the logs related with "Time" was printed out. However, other information for

Re: Is that possible to launch spark streaming application on yarn with only one machine?

In yarn cluster mode , Driver is running in AM , so you can find the logs in that AM log . Open rersourcemanager UI , and check for the Job and logs. or yarn logs -applicationId In yarn client mode , the driver is the same JVM from where you are launching ,,So you are getting it in the log . On

Re: Is that possible to launch spark streaming application on yarn with only one machine?

Launching via client deploy mode, it works again. I'm still a little confused about the behavior difference for cluster and client mode on a single machine. Thanks, Jared From: Mich Talebzadeh Sent: Wednesday, July 6, 2016 9:46:11

Re: Is that possible to launch spark streaming application on yarn with only one machine?

Deploy-mode cluster don't think will work. Try --master yarn --deploy-mode client FYI - *Spark Local* - Spark runs on the local host. This is the simplest set up and best suited for learners who want to understand different concepts of Spark and those performing unit testing. -

Re: Is that possible to launch spark streaming application on yarn with only one machine?

Hi Deng, I tried the same code again. It seemed that when launching application via yarn on single node, JavaDStream.print() did not work. However, occasionally it worked. If launch the same application in local mode, it always worked. The code is as below, SparkConf conf = new

Re: Is that possible to launch spark streaming application on yarn with only one machine?

Hi Deng, Thanks for the help. Actually I need pay more attention to memory usage. I found the root cause in my problem. It seemed that it existed in spark streaming MQTTUtils module. When I use "localhost" in brokerURL, it doesn't work. After change it to "127.0.0.1", it works now. Thanks

Re: Is that possible to launch spark streaming application on yarn with only one machine?

Hi Jared, You can launch a Spark application even with just a single node in YARN, provided that the node has enough resources to run the job. It might also be good to note that when YARN calculates the memory allocation for the driver and the executors, there is an additional memory overhead

Is that possible to launch spark streaming application on yarn with only one machine?

Hi guys, I set up pseudo hadoop/yarn cluster on my labtop. I wrote a simple spark streaming program as below to receive messages with MQTTUtils. conf = new SparkConf().setAppName("Monitor"); jssc = new JavaStreamingContext(conf, Durations.seconds(1)); JavaReceiverInputDStream inputDS =