Re: Properly stop applications or jobs within the application

@sagar - YARN kill is not a reliable process for spark streaming. Regards, Dhaval Modi dhavalmod...@gmail.com On 8 March 2018 at 17:18, bsikander wrote: > I am running in Spark standalone mode. No YARN. > > anyways, yarn application -kill is a manual process. I donot want that. I > was to

Re: Properly stop applications or jobs within the application

I am running in Spark standalone mode. No YARN. anyways, yarn application -kill is a manual process. I donot want that. I was to properly kill the driver/application programatically. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Re: Properly stop applications or jobs within the application

I am assuming you are running in yarn cluster mode. Have you tried yarn application -kill application_id ? With regards, Sagar Grover Phone - 7022175584 On Thu, Mar 8, 2018 at 4:03 PM, bsikander wrote: > I have scenarios for both. > So, I want to kill both batch and

Re: Properly stop applications or jobs within the application

I have scenarios for both. So, I want to kill both batch and streaming midway, if required. Usecase: Normally, if everything is okay we don't kill the application but sometimes while accessing external resources (like Kafka) something can go wrong. In that case, the application can become useless

Re: Properly stop applications or jobs within the application

What do you mean by stopping applications? Do you want to kill a batch application mid way or are you running streaming jobs that you want to kill? With regards, Sagar Grover On Thu, Mar 8, 2018 at 1:45 PM, bsikander wrote: > Any help would be much appreciated. This seems

Re: Properly stop applications or jobs within the application

Any help would be much appreciated. This seems to be a common problem. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Properly stop applications or jobs within the application

It seems to be related to this issue from Kafka https://issues.apache.org/jira/browse/KAFKA-1894 -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Properly stop applications or jobs within the application

Hello, We are using spark-jobserver to spawn jobs in Spark cluster. We have recently faced issues with Zombie jobs in Spark cluster. This normally happens when the job is accessing some external resources like Kafka/C* and something goes wrong while consuming them. For example, if suddenly a topic