[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1225 [SPARK-2286][UI] Report exception/errors for failed tasks that are not ExceptionFailure Also added inline doc for each TaskEndReason. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark SPARK-2286 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1225.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1225 commit 38c73916e28c97cee49b2dab03f4dcc76d6da82f Author: Reynold Xin r...@apache.org Date: 2014-06-26T06:27:36Z [SPARK-2286][UI] Report exception/errors for failed tasks that are not ExceptionFailure. Also added inline doc for each TaskEndReason. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1225#discussion_r14226878 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -30,27 +30,67 @@ import org.apache.spark.storage.BlockManagerId @DeveloperApi sealed trait TaskEndReason +/** + * :: DeveloperApi :: + * Task succeeded. + */ @DeveloperApi case object Success extends TaskEndReason --- End diff -- Want to rename this while you're at it? (It's developer API) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47191925 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1225#discussion_r14226907 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -30,27 +30,67 @@ import org.apache.spark.storage.BlockManagerId @DeveloperApi sealed trait TaskEndReason +/** + * :: DeveloperApi :: + * Task succeeded. + */ @DeveloperApi case object Success extends TaskEndReason --- End diff -- I can do that if we are not going to backport this into branch-1.0. I think we should though ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1225#discussion_r14227125
--- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala ---
@@ -58,10 +98,19 @@ case class ExceptionFailure(
* it was fetched.
*/
@DeveloperApi
-case object TaskResultLost extends TaskEndReason
+case object TaskResultLost extends TaskFailedReason {
+ override def toErrorString: String =
+TaskResultLost (result lost from block manager)
--- End diff --
nit: move to line above
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47192750 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47192746 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1225#discussion_r14227540
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -283,12 +283,7 @@ private[ui] class StagePage(parent: JobProgressTab)
extends WebUIPage(stage) {
/td
}}
td
- {exception.map { e =
-span
- {e.className} ({e.description})br/
- {fmtStackTrace(e.stackTrace)}
-/span
- }.getOrElse()}
+ {errorMessage.map { e = pre{e}/pre }.getOrElse()}
--- End diff --
It does. Looks like the stack trace in the details thing on the stage list.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47193773 This is a pretty straightforward change. Looks good to me providing that the stack trace looks alright and tests pass. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47194199 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47194200 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16139/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47198182 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47198281 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47198289 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47199293 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16141/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47199288 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47200164 There was a unit test failure that my latest push fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47200218 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47200208 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47201642 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16148/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47201641 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47203899 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47203901 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16152/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1225#discussion_r14253406
--- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala ---
@@ -30,27 +30,69 @@ import org.apache.spark.storage.BlockManagerId
@DeveloperApi
sealed trait TaskEndReason
+/**
+ * :: DeveloperApi ::
+ * Task succeeded.
+ */
@DeveloperApi
case object Success extends TaskEndReason
+/**
+ * :: DeveloperApi ::
+ * Various possible reasons why a task failed.
+ */
+@DeveloperApi
+sealed trait TaskFailedReason extends TaskEndReason {
+ /** Error message displayed in the web UI. */
+ def toErrorString: String
+}
+
+/**
+ * :: DeveloperApi ::
+ * A [[org.apache.spark.scheduler.ShuffleMapTask]] that completed
successfully earlier, but we
+ * lost the executor before the stage completed. This means Spark needs to
reschedule the task
+ * to be re-executed on a different executor.
+ */
@DeveloperApi
-case object Resubmitted extends TaskEndReason // Task was finished earlier
but we've now lost it
+case object Resubmitted extends TaskFailedReason {
+ override def toErrorString: String = Resubmitted (resubmitted due to
lost executor)
+}
+/**
+ * :: DeveloperApi ::
+ * Task failed to fetch shuffle data from a remote node. Probably means we
have lost the remote
+ * executors the task is trying to fetch from, and thus needs to rerun the
previous stage.
--- End diff --
super nit: needs - need
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1225#issuecomment-47279406 Thanks guys, I'm going to merge this and make the change kay suggested. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2286][UI] Report exception/errors for f...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1225 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
