GitHub user BryanCutler opened a pull request: https://github.com/apache/spark/pull/11746
[SPARK-13602][CORE] Add shutdown hook to DriverRunner to prevent driver process leak ## What changes were proposed in this pull request? Added shutdown hook to DriverRunner to kill the driver process in case the Worker JVM exits suddenly and the `WorkerWatcher` was unable to properly catch this. Changed the `DriverRunner.kill()` method to make use of a Thread interrupt instead of setting a flag so that the process can be managed within the Thread scope. Did some cleanup to consolidate driver state management and setting of finalized vars within the running thread. ## How was this patch tested? Added unit tests toverify that final state and exception variables are set accordingly for successfull, failed, and errors in the driver process. Retrofitted existing test to verify killing of mocked process ends with the correct state and stops properly Manually tested (with deploy-mode=cluster) that the shutdown hook is called by forcibly exiting the `Worker` and various points in the code with the `WorkerWatcher` both disabled and enabled. Also, manually killed the driver and verified that the `DriverRunner` killed the process and exited properly. You can merge this pull request into a Git repository by running: $ git pull https://github.com/BryanCutler/spark DriverRunner-shutdown-hook-SPARK-13602 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/11746.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11746 ---- commit b0ce75ed40fe92f15aed1e30002ab51daaead950 Author: Bryan Cutler <cutl...@gmail.com> Date: 2016-03-15T23:40:17Z [SPARK-13602] Added shutdown hook to DriverRunner and cleanup commit 2402ecbdfe16b8cc7388cd93fd96140ef1fd8f69 Author: Bryan Cutler <cutl...@gmail.com> Date: 2016-03-15T23:41:41Z Added tests for DriverRunner for finalization states and refit existing test to work with changes commit 96c3e7f516bf4c7b054bb5651fb777a3f0b0718a Author: Bryan Cutler <cutl...@gmail.com> Date: 2016-03-15T23:54:37Z Removed usage of AtomicRef for process, it is only shared with the shutdown hook and if that is called, then does not matter at that point commit e3f17f894636c9fdb8b451458ce9470af499703c Author: Bryan Cutler <cutl...@gmail.com> Date: 2016-03-16T00:18:09Z removed import of AtomicRef ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org