[GitHub] spark pull request #21936: [SPARK-24981][Core] ShutdownHook timeout causes j...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/21936


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21936: [SPARK-24981][Core] ShutdownHook timeout causes j...

[tgravescs]

Github user tgravescs commented on a diff in the pull request:

https://github.com/apache/spark/pull/21936#discussion_r207269915
  
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -571,7 +571,12 @@ class SparkContext(config: SparkConf) extends Logging {
 _shutdownHookRef = ShutdownHookManager.addShutdownHook(
   ShutdownHookManager.SPARK_CONTEXT_SHUTDOWN_PRIORITY) { () =>
   logInfo("Invoking stop() from shutdown hook")
-  stop()
+  try {
+stop()
+  } catch {
+case e: Throwable =>
+  logWarning("Ignoring Exception while stopping SparkContext", e)
--- End diff --

minor nit, could you add in "while stopping SparkContext from shutdownhook"


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21936: [SPARK-24981][Core] ShutdownHook timeout causes j...

[hthuynh2]

Github user hthuynh2 commented on a diff in the pull request:

https://github.com/apache/spark/pull/21936#discussion_r206918869
  
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -571,7 +571,12 @@ class SparkContext(config: SparkConf) extends Logging {
 _shutdownHookRef = ShutdownHookManager.addShutdownHook(
   ShutdownHookManager.SPARK_CONTEXT_SHUTDOWN_PRIORITY) { () =>
   logInfo("Invoking stop() from shutdown hook")
-  stop()
+  try {
+stop()
+  } catch {
+case e: Throwable =>
+  logWarning("Ignoring Exception while stoping SparkContext. 
Exception: " + e)
--- End diff --

@felixcheung Thanks for the comments. I updated it. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21936: [SPARK-24981][Core] ShutdownHook timeout causes j...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/21936#discussion_r206769869
  
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -571,7 +571,12 @@ class SparkContext(config: SparkConf) extends Logging {
 _shutdownHookRef = ShutdownHookManager.addShutdownHook(
   ShutdownHookManager.SPARK_CONTEXT_SHUTDOWN_PRIORITY) { () =>
   logInfo("Invoking stop() from shutdown hook")
-  stop()
+  try {
+stop()
+  } catch {
+case e: Throwable =>
+  logWarning("Ignoring Exception while stoping SparkContext. 
Exception: " + e)
--- End diff --

`stoping` -> `stopping`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21936: [SPARK-24981][Core] ShutdownHook timeout causes j...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/21936#discussion_r206770131
  
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -571,7 +571,12 @@ class SparkContext(config: SparkConf) extends Logging {
 _shutdownHookRef = ShutdownHookManager.addShutdownHook(
   ShutdownHookManager.SPARK_CONTEXT_SHUTDOWN_PRIORITY) { () =>
   logInfo("Invoking stop() from shutdown hook")
-  stop()
+  try {
+stop()
+  } catch {
+case e: Throwable =>
+  logWarning("Ignoring Exception while stoping SparkContext. 
Exception: " + e)
--- End diff --

use this format 
`logWarning("", exception)`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21936: [SPARK-24981][Core] ShutdownHook timeout causes j...

[hthuynh2]

GitHub user hthuynh2 opened a pull request:

https://github.com/apache/spark/pull/21936

[SPARK-24981][Core] ShutdownHook timeout causes job to fail when succeeded 
when SparkContext stop() not called by user program

**Description**
The issue is described in 
[SPARK-24981](https://issues.apache.org/jira/browse/SPARK-24981). 

**How does this PR fix the issue?**
This PR catch the Exception that is thrown while the Sparkcontext.stop() is 
running (when it is called by the ShutdownHookManager).

**How was this patch tested?**
I manually tested it by adding delay (60s) inside the stop(). This make the 
shutdownHookManger interrupt the thread that is running stop(). The Interrupted 
Exception was catched and the job succeed.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hthuynh2/spark SPARK_24981

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/21936.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #21936


commit 101bf2b0488ab495dbdaf7b54d9a57d9827cb833
Author: Hieu Huynh <“hieu.huynh@...>
Date:   2018-07-31T18:30:11Z

initial commit




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org