[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-153483685 Do you mind closing this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user nssalian commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-153467686 What is the status of the PR? Seems no movement for a while. @vanzin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user witgo closed the pull request at: https://github.com/apache/spark/pull/1482 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-136971893 I think it is necessary to merge the PR into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-136971880 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-136971892 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-136973684 [Test build #41928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41928/consoleFull) for PR 1482 at commit [`7271308`](https://github.com/apache/spark/commit/7271308de670c582fa97ff1db13401db379d4cb2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-137032169 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41928/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-137032164 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-137031896 [Test build #41928 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41928/console) for PR 1482 at commit [`7271308`](https://github.com/apache/spark/commit/7271308de670c582fa97ff1db13401db379d4cb2). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ExecutorExitFailure(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-136917245 What's the status of this patch? Is it still relevant after so many releases? @witgo @vanzin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-97884669 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31427/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-97884647 [Test build #31427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31427/consoleFull) for PR 1482 at commit [`ac022c9`](https://github.com/apache/spark/commit/ac022c9f71a7bc2db63b842a5b2cf9a31776c77c). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ExecutorExitFailure(` * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-97884665 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/1482#discussion_r29440041
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -280,13 +280,18 @@ private[spark] class Executor(
m
}
}
- val taskEndReason = new ExceptionFailure(t, metrics)
- execBackend.statusUpdate(taskId, TaskState.FAILED,
ser.serialize(taskEndReason))
-
- // Don't forcibly exit unless the exception was inherently
fatal, to avoid
- // stopping other tasks unnecessarily.
- if (Utils.isFatalError(t)) {
-SparkUncaughtExceptionHandler.uncaughtException(t)
+ if (Utils.inShutdown()) {
+val reason = ExecutorExitFailure(None, metrics)
+execBackend.statusUpdate(taskId, TaskState.FAILED,
ser.serialize(reason))
--- End diff --
Yes,The code work in most cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-97842633 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-97842583 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-97843150 [Test build #31427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31427/consoleFull) for PR 1482 at commit [`ac022c9`](https://github.com/apache/spark/commit/ac022c9f71a7bc2db63b842a5b2cf9a31776c77c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1482#discussion_r29381930
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -280,13 +280,18 @@ private[spark] class Executor(
m
}
}
- val taskEndReason = new ExceptionFailure(t, metrics)
- execBackend.statusUpdate(taskId, TaskState.FAILED,
ser.serialize(taskEndReason))
-
- // Don't forcibly exit unless the exception was inherently
fatal, to avoid
- // stopping other tasks unnecessarily.
- if (Utils.isFatalError(t)) {
-SparkUncaughtExceptionHandler.uncaughtException(t)
+ if (Utils.inShutdown()) {
+val reason = ExecutorExitFailure(None, metrics)
+execBackend.statusUpdate(taskId, TaskState.FAILED,
ser.serialize(reason))
+ }
+ else {
--- End diff --
nit: else goes in previous line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1482#discussion_r29381924
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -280,13 +280,18 @@ private[spark] class Executor(
m
}
}
- val taskEndReason = new ExceptionFailure(t, metrics)
- execBackend.statusUpdate(taskId, TaskState.FAILED,
ser.serialize(taskEndReason))
-
- // Don't forcibly exit unless the exception was inherently
fatal, to avoid
- // stopping other tasks unnecessarily.
- if (Utils.isFatalError(t)) {
-SparkUncaughtExceptionHandler.uncaughtException(t)
+ if (Utils.inShutdown()) {
+val reason = ExecutorExitFailure(None, metrics)
+execBackend.statusUpdate(taskId, TaskState.FAILED,
ser.serialize(reason))
--- End diff --
I guess this doesn't change what was there before, but is this safe during
shutdown?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1482#discussion_r29382009
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -701,6 +701,10 @@ private[spark] class TaskSetManager(
s${ef.className} (${ef.description}) [duplicate $dupCount])
}
+ case e :ExecutorExitFailure =
--- End diff --
nit: `case e: ...`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-97588168 Well, this has been lingering here for a while. Looks sane though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-95061047 [Test build #30738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30738/consoleFull) for PR 1482 at commit [`84ea436`](https://github.com/apache/spark/commit/84ea43603e06b6f6731aa44edc3c67f544914faa). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-95117217 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-95117546 [Test build #30750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30750/consoleFull) for PR 1482 at commit [`84ea436`](https://github.com/apache/spark/commit/84ea43603e06b6f6731aa44edc3c67f544914faa). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-95144904 [Test build #30750 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30750/consoleFull) for PR 1482 at commit [`84ea436`](https://github.com/apache/spark/commit/84ea43603e06b6f6731aa44edc3c67f544914faa). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ExecutorExitFailure(` * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-95144915 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30750/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-95078711 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30738/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-95078684 [Test build #30738 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30738/consoleFull) for PR 1482 at commit [`84ea436`](https://github.com/apache/spark/commit/84ea43603e06b6f6731aa44edc3c67f544914faa). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ExecutorExitFailure(` * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-94215412 I actually wouldn't mind working on this - it's an issue near and dear to my heart :-). What's the best way to pick up the ball here ? I could just make a new PR - the changes are pretty minimal at present. Sent with Good (www.good.com) -Original Message- From: Josh Rosen [notificati...@github.commailto:notificati...@github.com] Sent: Saturday, April 18, 2015 08:05 PM Eastern Standard Time To: apache/spark Cc: Ganelin, Ilya Subject: Re: [spark] [SPARK-2491] Don't handle uncaught exceptions from tasks that fail during executor shutdown. (#1482) @ilganelihttps://github.com/ilganeli I think that this has largely stalled because this is a somewhat complicated-to-review change (as the discussion above shows, it can be tricky to make sure that we've thought through all of the ways in which control can enter the error-handling / shutdown hook code). This issue is probably still worth fixing, but I think it will take a lot more work to review this, especially since this type of code is hard to test. â Reply to this email directly or view it on GitHubhttps://github.com/apache/spark/pull/1482#issuecomment-94215242. The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-94215242 @ilganeli I think that this has largely stalled because this is a somewhat complicated-to-review change (as the discussion above shows, it can be tricky to make sure that we've thought through all of the ways in which control can enter the error-handling / shutdown hook code). This issue is probably still worth fixing, but I think it will take a lot more work to review this, especially since this type of code is hard to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-94074662 @JoshRosen There's been no activity here for over 5 months - is this still alive? If not, I wouldn't mind finishing the job since I ran into this exact issue and could use better debug visibility. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-84830436 [Test build #28980 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28980/consoleFull) for PR 1482 at commit [`68a5086`](https://github.com/apache/spark/commit/68a50863975b90e98618018ffebcbfc18944e102). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ExecutorExitFailure(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-84830480 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28980/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-84812073 [Test build #28980 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28980/consoleFull) for PR 1482 at commit [`68a5086`](https://github.com/apache/spark/commit/68a50863975b90e98618018ffebcbfc18944e102). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-67799173 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24695/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-61362218 [Test build #22693 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22693/consoleFull) for PR 1482 at commit [`996f67d`](https://github.com/apache/spark/commit/996f67d110bcf73904d1af732fa2f0f25d524207). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-61363387 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22693/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-61363384 [Test build #22693 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22693/consoleFull) for PR 1482 at commit [`996f67d`](https://github.com/apache/spark/commit/996f67d110bcf73904d1af732fa2f0f25d524207). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ExecutorExitFailure(` * `class StreamingKMeansModel(` * `class StreamingKMeans(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-61363413 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-61363474 [Test build #22694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22694/consoleFull) for PR 1482 at commit [`996f67d`](https://github.com/apache/spark/commit/996f67d110bcf73904d1af732fa2f0f25d524207). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-61365142 [Test build #22694 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22694/consoleFull) for PR 1482 at commit [`996f67d`](https://github.com/apache/spark/commit/996f67d110bcf73904d1af732fa2f0f25d524207). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ExecutorExitFailure(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-61365144 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22694/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-54875511 @aarondav Do you have an example scenario where this patch prevents OOM reporting? Really, the only change here is wrapping the exception logging / reporting with `if (!Utils.inShutdown())`, which should always return `false` until we have a fatal error that triggers the shutdown. The actual handling the fatal error itself should be the same, as far as I can tell. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-54915526 Well, I mean, this is our attempt to tell people what happened: ``` execBackend.statusUpdate(taskId, TaskState.FAILED, ser.serialize(reason)) ``` We cannot expect that this logic was already called once, I believe, because the executor may enter shutdown mode based on a different thread receiving an OOM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-54917849 @aarondav I understand what you mean,I will submit the relevant code tomorrow. BTW,most of the OOM are present in deserialization process. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-54700201 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19887/consoleFull) for PR 1482 at commit [`6673dca`](https://github.com/apache/spark/commit/6673dca28dd3f163045d24feb65b4fe6f8f65c56). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-54700237 Could this suppress the OOM message from getting to the driver? I thought there was special handling for these messages (error code). This could result in executor died rather than executor oom'd messages. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2491] Don't handle uncaught exceptions ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1482#issuecomment-54701544 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19887/consoleFull) for PR 1482 at commit [`6673dca`](https://github.com/apache/spark/commit/6673dca28dd3f163045d24feb65b4fe6f8f65c56). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
