I am setting up *Spark 2.2.0 in standalone mode* ( https://spark.apache.org/docs/latest/spark-standalone.html) and submitting spark jobs programatically using
SparkLauncher sparkAppLauncher = new SparkLauncher(userNameMap).setMaster(sparkMaster).setAppName(appName).....; SparkAppHandle sparkAppHandle = sparkAppLauncher.startApplication(); I do have* java client program that polls Job status for the jobs submitted programatically*, for which i am using following REST endpoint. curl http://192.168.1.139:8080/json/ which provide JSON response as following, { "url" : "spark://192.168.1.139:7077", "workers" : [ { "id" : "x", "host" : "x", "port" : x, "webuiaddress" : "x", "cores" : x, "coresused" : x, "coresfree" : x, "memory" : xx, "memoryused" : xx, "memoryfree" : xx, "state" : "x", "lastheartbeat" : x }, { ...}, ], "cores" : x, "coresused" : x, "memory" : x, "memoryused" : x, "activeapps" : [ ], "completedapps" : [ { "starttime" : x, "id" : "app-xx-xxxx", "name" : "abc", "user" : "xx", "memoryperslave" : x, "submitdate" : "x","state" : "FINISHED OR RUNNING", "duration" : x }, {...}], "activedrivers" : [ ], "status" : "x"} In above response, I have observed state for *completedapps is always FINISHED even if application fails*, while on UI (http://master:8080), associated driver shows a failed state, like. [image: image.png] [image: image.png] Referring to above example, Currently, My java client gets status for application (app-20190925115750-0003 <http://192.168.1.139:8080/app?appId=app-20190925115750-0003>) FINISHED, even though it got failed (encountered exception) and associated driver shows "FAILED" state.* I intend to show the final status in this case as FAILED.* It seems if i can co-relate, an application-id (app-20190925115750-0003 <http://192.168.1.139:8080/app?appId=app-20190925115750-0003>) to driver-id (driver-20190925115748-0003), I can report a "FAILED" (final) status. I could not find any co-relation between them (appID --> driver ID). *Looking forward to your suggestions to resolving this or any possible approaches to achieve this.* I have also come across some hidden REST APIs like http://xx.xx.xx.xx:6066/v1/submissions/status/driver-20190925115748-0003, which seems have a limited info returned in response. Thanks in advance. Nilkanth.
