GitHub user BryanCutler opened a pull request: https://github.com/apache/spark/pull/10714
[SPARK-9844][CORE] File appender race condition during shutdown When an Executor process is destroyed, the FileAppender that is asynchronously reading the stderr stream of the process can throw an IOException during read because the stream is closed. Before the ExecutorRunner destroys the process, the FileAppender thread is flagged to stop. This PR wraps the inputStream.read call of the FileAppender in a try/catch block so that if an IOException is thrown and the thread has been flagged to stop, it will safely ignore the exception. Additionally, the FileAppender thread was changed to use Utils.tryWithSafeFinally to better log any exception that do occur. Added unit tests to verify a IOException is thrown and logged if FileAppender is not flagged to stop, and that no IOException when the flag is set. You can merge this pull request into a Git repository by running: $ git pull https://github.com/BryanCutler/spark file-appender-read-ioexception-SPARK-9844 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10714.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10714 ---- commit 6efff7495a6bd630baf20b5238d51644d5abca46 Author: Bryan Cutler <cutl...@gmail.com> Date: 2016-01-12T00:22:08Z [SPARK-9844] Added check in FileAppender to not thrown an IOException if marked as stopped and changed file operation block to use tryWithSafeFinally commit f937e183d466d014f1545d823ecdf30f9becdf77 Author: Bryan Cutler <cutl...@gmail.com> Date: 2016-01-12T00:22:55Z [SPARK-9844] Added unit test for closing FileAppender InputStream asynchronously commit 1775d98c1d9c9664e0f9ce80082715938c6e0aed Author: Bryan Cutler <cutl...@gmail.com> Date: 2016-01-12T00:27:03Z fixed import order commit dc6573715a77107e7fbaaf97b4dbbc18f652b9b3 Author: Bryan Cutler <cutl...@gmail.com> Date: 2016-01-12T00:57:09Z Did not need check for log level ERROR because only logging ERRORS, check for getThrowableInformation null ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org