Re: Programmatically get status of job (WAITING/RUNNING)

Qiao, Richard wrote > Comparing #1 and #3, my understanding of “submitted” is “the jar is > submitted to executors”. With this concept, you may define your own > status. In SparkLauncher, SUBMITTED means that the Driver was able to acquire cores from Spark cluster and Launcher is waiting for

Re: Programmatically get status of job (WAITING/RUNNING)

Qiao, Richard wrote > For your question of example, the answer is yes. Perfect. I am assuming that this is true for Spark-standalone/YARN/Mesos. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To

Re: Programmatically get status of job (WAITING/RUNNING)

For your question of example, the answer is yes. “For example, if an application wanted 4 executors (spark.executor.instances=4) but the spark cluster can only provide 1 executor. This means that I will only receive 1 onExecutorAdded event. Will the application state change to

Re: Programmatically get status of job (WAITING/RUNNING)

For #2, do you mean “RUNNING” showing in “Driver” table? If yes, that is not a problem, because driver does run, while there is no executor available, as can be a status for you to catch – Driver running while no executors. Comparing #1 and #3, my understanding of “submitted” is “the jar is

Re: Programmatically get status of job (WAITING/RUNNING)

That's the Spark Master's view of the application. I don't know exactly what it means in the different run modes, I'm more familiar with YARN. But I wouldn't be surprised if, as with others, it mostly tracks the driver's state. On Thu, Dec 7, 2017 at 12:06 PM, bsikander

Re: Programmatically get status of job (WAITING/RUNNING)

See the image. I am referring to this state when I say "Application State". -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Re: Programmatically get status of job (WAITING/RUNNING)

On Thu, Dec 7, 2017 at 11:40 AM, bsikander wrote: > For example, if an application wanted 4 executors > (spark.executor.instances=4) but the spark cluster can only provide 1 > executor. This means that I will only receive 1 onExecutorAdded event. Will > the application state

Re: Programmatically get status of job (WAITING/RUNNING)

Marcelo Vanzin wrote > I'm not sure I follow you here. This is something that you are > defining, not Spark. Yes, you are right. In my code, 1) my notion of RUNNING is that both driver + executors are in RUNNING state. 2) my notion of WAITING is if any one of driver/executor is in WAITING state.

Re: Programmatically get status of job (WAITING/RUNNING)

On Tue, Dec 5, 2017 at 12:43 PM, bsikander wrote: > 2) If I use context.addSparkListener, I can customize the listener but then > I miss the onApplicationStart event. Also, I don't know the Spark's logic to > changing the state of application from WAITING -> RUNNING. I'm not

Re: Programmatically get status of job (WAITING/RUNNING)

Thank you for the reply. I am not a Spark expert but I was reading through the code and I thought that the state was changed from SUBMITTED to RUNNING only after executors (CoarseGrainedExecutorBackend) were registered.

Re: Programmatically get status of job (WAITING/RUNNING)

SparkLauncher operates at a different layer than Spark applications. It doesn't know about executors or driver or anything, just whether the Spark application was started or not. So it doesn't work for your case. The best option for your case is to install a SparkListener and monitor events. But

Re: Programmatically get status of job (WAITING/RUNNING)

So, I tried to use SparkAppHandle.Listener with SparkLauncher as you suggested. The behavior of Launcher is not what I expected. 1- If I start the job (using SparkLauncher) and my Spark cluster has enough cores available, I receive events in my class extending SparkAppHandle.Listener and I see

Re: Programmatically get status of job (WAITING/RUNNING)

In this case, the only way to check the status is via REST calls to the Spark json API, accessible at http://:/json/ -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail:

Re: Programmatically get status of job (WAITING/RUNNING)

Thank you for the reply. I am currently not using SparkLauncher to launch my driver. Rather, I am using the old fashion spark-submit and moving to SparkLauncher is not an option right now. Do I have any options there? -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Re: Programmatically get status of job (WAITING/RUNNING)

Hello Behroz, you can use a SparkListener to get updates from the underlying process (c.f. https://spark.apache.org/docs/2.2.0/api/java/org/apache/spark/scheduler/SparkListener.html ) You need first to create your own SparkAppListener class that extends it: -

Re: Programmatically get status of job (WAITING/RUNNING)

Anyone ? -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Programmatically get status of job (WAITING/RUNNING)

Hi, I have a Spark Cluster running in client mode. I programmatically submit jobs to spark cluster. Under the hood, I am using spark-submit. If my cluster is overloaded and I start a context, the driver JVM keeps on waiting for executors. The executors are in waiting state because cluster does