Re: Correct SparkLauncher usage

Hey Mohammad, I implemented the code using CountDownLatch, and SparkLauncher works as expected. Hope it helps. Whenever appHandle.getState() reaching one of The Final states, then countDownLatch is decreased, and execution returns back to main program. ...final CountDownLatch countDownLatch =

Re: Correct SparkLauncher usage

Sure, will look into the tests. Thanks you so much for your time! [image: --] Tariq, Mohammad [image: https://]about.me/mti [image: http://] Tariq, Mohammad about.me/mti [image: http://]

Re: Correct SparkLauncher usage

Sorry, it's kinda hard to give any more feedback from just the info you provided. I'd start with some working code like this from Spark's own unit tests:

Re: Correct SparkLauncher usage

All I want to do is submit a job, and keep on getting states as soon as it changes, and come out once the job is over. I'm sorry to be a pest of questions. Kind of having a bit of tough time making this work. [image: --] Tariq, Mohammad [image: https://]about.me/mti

Re: Correct SparkLauncher usage

Yeah, that definitely makes sense. I was just trying to make it work somehow. The problem is that it's not at all calling the listeners, hence i'm unable to do anything. Just wanted to cross check it by looping inside. But I get the point. thank you for that! I'm on YARN(cluster mode). [image:

Re: Correct SparkLauncher usage

On Thu, Nov 10, 2016 at 2:43 PM, Mohammad Tariq wrote: > @Override > public void stateChanged(SparkAppHandle handle) { > System.out.println("Spark App Id [" + handle.getAppId() + "]. State [" + > handle.getState() + "]"); > while(!handle.getState().isFinal()) {

Re: Correct SparkLauncher usage

Hi Marcelo, After a few changes I got it working. However I could not understand one thing. I need to call Thread.sleep() and then get the state explicitly in order to make it work. Also, no matter what I do my launcher program doesn't call stateChanged() or infoChanged(). Here is my code :

Re: Correct SparkLauncher usage

Then you need to look at your logs to figure out why the child app is not working. "startApplication" will by default redirect the child's output to the parent's logs. On Mon, Nov 7, 2016 at 3:42 PM, Mohammad Tariq wrote: > Hi Marcelo, > > Thank you for the prompt response.

Re: Correct SparkLauncher usage

Hi Marcelo, Thank you for the prompt response. I tried adding listeners as well, didn't work either. Looks like it isn't starting the job at all. [image: --] Tariq, Mohammad [image: https://]about.me/mti

Re: Correct SparkLauncher usage

On Mon, Nov 7, 2016 at 3:29 PM, Mohammad Tariq wrote: > I have been trying to use SparkLauncher.startApplication() to launch a Spark > app from within java code, but unable to do so. However, same piece of code > is working if I use SparkLauncher.launch(). > > Here are the

Correct SparkLauncher usage

Dear fellow Spark users, I have been trying to use *SparkLauncher.startApplication()* to launch a Spark app from within java code, but unable to do so. However, same piece of code is working if I use *SparkLauncher.launch()*. Here are the corresponding code snippets : *SparkAppHandle handle =