Thanks for all responses. 1) I am not using YARN. I am using Spark Standalone. 2) yes I want to be able to kill the whole Application. 3) I want to be able to monitor the status of the Application which is running a batch query and expected to run for an hour or so, therefore, I am looking for some mechanism where I can monitor the progress like a percentage or something.
Thanks! On Wed, Dec 5, 2018 at 3:12 PM Mark Hamstra <m...@clearstorydata.com> wrote: > That will kill an entire Spark application, not a batch Job. > > On Wed, Dec 5, 2018 at 3:07 PM Priya Matpadi <pmatp...@gmail.com> wrote: > >> if you are deploying your spark application on YARN cluster, >> 1. ssh into master node >> 2. List the currently running application and retreive the application_id >> yarn application --list >> 3. Kill the application using application_id of the form >> application_xxxxx_xxxx from output of list command >> yarn application --kill <application_id> >> >> On Wed, Dec 5, 2018 at 1:42 PM kant kodali <kanth...@gmail.com> wrote: >> >>> Hi All, >>> >>> How to track batch jobs in spark? For example, is there some id or token >>> i can get after I spawn a batch job and use it to track the progress or to >>> kill the batch job itself? >>> >>> For Streaming, we have StreamingQuery.id() >>> >>> Thanks! >>> >>
