Hello Behroz, you can use a SparkListener to get updates from the underlying process (c.f. https://spark.apache.org/docs/2.2.0/api/java/org/apache/spark/scheduler/SparkListener.html )
You need first to create your own SparkAppListener class that extends it: ------------------------------------- private static class SparkAppListener implements SparkAppHandle.Listener, Runnable { SparkAppListener() {} @Override public void stateChanged(SparkAppHandle handle) { String sparkAppId = handle.getAppId(); SparkAppHandle.State appState = handle.getState(); log.info("Spark job with app id: " + sparkAppId + ", State changed to: " + appState); } @Override public void infoChanged(SparkAppHandle handle) {} @Override public void run() {} } ----------------------------------------- Then you can run it in a thread via a Executors.newCachedThreadPool (or with a simple New Thread(<your thread>)) ----------------------------------------- private final static ExecutorService listenerService = Executors.newCachedThreadPool(); SparkAppListener appListener = new SparkAppListener(); listenerService.execute(appListener); SparkLauncher command = new SparkLauncher() .setAppName(appName) .setSparkHome(sparkHome) .setAppResource(appResource) .setMainClass(mainClass) .setMaster(master) ......... SparkAppHandle appHandle = launcher.startApplication(appListener); ----------------------------------------- At this point, every time the state changes, you will execute the SparkAppListener.stateChanged method. Hope it helps, Davide -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org