There is a problem with killing jobs gracefully in spark 2.1.0 with enabled spark.streaming.stopGracefullyOnShutdown I tested killing spark jobs in many ways and I got some conclusions. 1. With command spark-submit --master spark:// --kill driver-id results: It killed all workers almost immediately 2. With api curl -X POST http://localhost:6066/v1/submissions/kill/driverId results: the same like in 1. (I looked at code and it seems like spark-submit calls just this REST endpoint) 3. With unix kill driver-process results: it didn't kill the job at all
Then I noticed that I used param: --supervise so I repeated these all tests. It turned out that 1 and 2 ways worked in the same way like before but 3 option worked like I assumed. This mean, calling kill driver-process job , it processed all messages which left and than turned down job gracefully. It is of course some solution but quite inconvenient since I must track machine with driver instead of using simple spark REST endpoint. I have read many of issues posted on blogs and stackoverflow and a lot of people struggle with this problem. Is anybody able to explain me why there are so many issues regarding this case and why 1 and 2 method works in different way than kill. Kind Regards, Mariusz Dubielecki
