[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145947026 [Test build #43278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43278/console) for PR 8950 at commit [`d7c42c9`](https://github.com/apache/spark/commit/d7c42c9e8599fd1dc7ab4bc732b1af29647cf6b9). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145947179 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43278/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145947177 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-146027302 LGTM. Merging this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8950 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145928461 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145928440 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145930082 [Test build #43278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43278/consoleFull) for PR 8950 at commit [`d7c42c9`](https://github.com/apache/spark/commit/d7c42c9e8599fd1dc7ab4bc732b1af29647cf6b9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145928029 > In the case of "Failed", unless the details is opened, there is not indication of failure. So it might be better to show "Failed due to error: $exceptionMessage", and the full stacktrace in the detail. Added it. https://cloud.githubusercontent.com/assets/1000778/10315755/b84cbfb8-6c8e-11e5-9cea-0b2b408afd05.png;> > In case of "Failed due to Spark error", does it make sense to have the error show up in both places? This seems unnecessary. The job error column has already shown it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145738954 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145738897 [Test build #43270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43270/console) for PR 8950 at commit [`ca68ac8`](https://github.com/apache/spark/commit/ca68ac858462fff107d8a5ce7a5af3cf9416aca3). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145738955 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43270/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145743591 This looks cool! Also I like that you used "details" for that. In the case of "Failed", unless the details is opened, there is not indication of failure. So it might be better to show "Failed due to error: $exceptionMessage", and the full stacktrace in the detail. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/8950#discussion_r41217146 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala --- @@ -265,16 +323,20 @@ private[ui] class BatchPage(parent: StreamingTab) extends WebUIPage("batch") { * Generate the job table for the batch. */ private def generateJobTable(batchUIData: BatchUIData): Seq[Node] = { -val outputOpIdToSparkJobIds = batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).toSeq. - sortBy(_._1). // sorted by OutputOpId +val outputOpIdToSparkJobIds = batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId). map { case (outputOpId, outputOpIdAndSparkJobIds) => // sort SparkJobIds for each OutputOpId (outputOpId, outputOpIdAndSparkJobIds.map(_.sparkJobId).sorted) } +val outputOps = (0 until batchUIData.numOutputOp).map { outputOpId => + val status = batchUIData.failureReason.getOrElse(outputOpId, "Succeeded") --- End diff -- I don't think so. E.g., the use may write the following codes: ``` stream.foreachRDD { rdd => try { rdd.foreach(...) } catch { ... } } ``` In this case, if a Spark job (rdd.foreach) fails, we cannot say the output op fails. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/8950#discussion_r41219078 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala --- @@ -265,16 +323,20 @@ private[ui] class BatchPage(parent: StreamingTab) extends WebUIPage("batch") { * Generate the job table for the batch. */ private def generateJobTable(batchUIData: BatchUIData): Seq[Node] = { -val outputOpIdToSparkJobIds = batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).toSeq. - sortBy(_._1). // sorted by OutputOpId +val outputOpIdToSparkJobIds = batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId). map { case (outputOpId, outputOpIdAndSparkJobIds) => // sort SparkJobIds for each OutputOpId (outputOpId, outputOpIdAndSparkJobIds.map(_.sparkJobId).sorted) } +val outputOps = (0 until batchUIData.numOutputOp).map { outputOpId => + val status = batchUIData.failureReason.getOrElse(outputOpId, "Succeeded") --- End diff -- Here is a possible logic. If no failure reason, then "Succeeded" Else if failure reason contains "SparkException" then "Failed due to Spark job error" Else then "Failed" This should work fine for most cases, where the user is not doing fancy things like catching exceptions themselves and ignoring/rethrowing them. Isnt it? Consider your example. If the user catches Spark job exception (most probably SparkException) and rethrows it, the above logic should identify it as Spark job error and say "Failed due to Spark job error". On the other hand, if the user catches and ignore exception, then failure reason will be empty and the output will be marked as "Succeeded" (even though Spark job error will not be empty, which is okay). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/8950#discussion_r41188944 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala --- @@ -265,16 +323,20 @@ private[ui] class BatchPage(parent: StreamingTab) extends WebUIPage("batch") { * Generate the job table for the batch. */ private def generateJobTable(batchUIData: BatchUIData): Seq[Node] = { -val outputOpIdToSparkJobIds = batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).toSeq. - sortBy(_._1). // sorted by OutputOpId +val outputOpIdToSparkJobIds = batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId). map { case (outputOpId, outputOpIdAndSparkJobIds) => // sort SparkJobIds for each OutputOpId (outputOpId, outputOpIdAndSparkJobIds.map(_.sparkJobId).sorted) } +val outputOps = (0 until batchUIData.numOutputOp).map { outputOpId => + val status = batchUIData.failureReason.getOrElse(outputOpId, "Succeeded") --- End diff -- Cant we distinguish based on the whether there is a Spark job error in any of the jobs associated with the output op --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/8950#discussion_r41220303 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala --- @@ -265,16 +323,20 @@ private[ui] class BatchPage(parent: StreamingTab) extends WebUIPage("batch") { * Generate the job table for the batch. */ private def generateJobTable(batchUIData: BatchUIData): Seq[Node] = { -val outputOpIdToSparkJobIds = batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).toSeq. - sortBy(_._1). // sorted by OutputOpId +val outputOpIdToSparkJobIds = batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId). map { case (outputOpId, outputOpIdAndSparkJobIds) => // sort SparkJobIds for each OutputOpId (outputOpId, outputOpIdAndSparkJobIds.map(_.sparkJobId).sorted) } +val outputOps = (0 until batchUIData.numOutputOp).map { outputOpId => + val status = batchUIData.failureReason.getOrElse(outputOpId, "Succeeded") --- End diff -- Sound great. I will update the logic. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145732880 New screenshot: ![3](https://cloud.githubusercontent.com/assets/1000778/10299535/bd291c50-6c1d-11e5-92c7-131c0630fad8.png) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145733024 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145733038 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-145734108 [Test build #43270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43270/consoleFull) for PR 8950 at commit [`ca68ac8`](https://github.com/apache/spark/commit/ca68ac858462fff107d8a5ce7a5af3cf9416aca3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/8950#discussion_r41128898 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala --- @@ -265,16 +323,20 @@ private[ui] class BatchPage(parent: StreamingTab) extends WebUIPage("batch") { * Generate the job table for the batch. */ private def generateJobTable(batchUIData: BatchUIData): Seq[Node] = { -val outputOpIdToSparkJobIds = batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).toSeq. - sortBy(_._1). // sorted by OutputOpId +val outputOpIdToSparkJobIds = batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId). map { case (outputOpId, outputOpIdAndSparkJobIds) => // sort SparkJobIds for each OutputOpId (outputOpId, outputOpIdAndSparkJobIds.map(_.sparkJobId).sorted) } +val outputOps = (0 until batchUIData.numOutputOp).map { outputOpId => + val status = batchUIData.failureReason.getOrElse(outputOpId, "Succeeded") --- End diff -- > In fact, there should be "Failed" and "Failed due to Spark job error". In the first case, the output op error will be below the "Failed" . I think now we cannot distinguish these two errors. They are both caught in `org.apache.spark.streaming.scheduler.Job.run`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/8950#discussion_r41077235 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala --- @@ -265,16 +323,20 @@ private[ui] class BatchPage(parent: StreamingTab) extends WebUIPage("batch") { * Generate the job table for the batch. */ private def generateJobTable(batchUIData: BatchUIData): Seq[Node] = { -val outputOpIdToSparkJobIds = batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).toSeq. - sortBy(_._1). // sorted by OutputOpId +val outputOpIdToSparkJobIds = batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId). map { case (outputOpId, outputOpIdAndSparkJobIds) => // sort SparkJobIds for each OutputOpId (outputOpId, outputOpIdAndSparkJobIds.map(_.sparkJobId).sorted) } +val outputOps = (0 until batchUIData.numOutputOp).map { outputOpId => + val status = batchUIData.failureReason.getOrElse(outputOpId, "Succeeded") --- End diff -- If we are showing "Succeeded" for successful output ops, then we should also show "Failed" for failed ones as well. In fact, there should be "Failed" and "Failed due to Spark job error". In the first case, the output op error will be below the "Failed" . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-144460172 /cc @tdas --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/8950 [SPARK-10885][Streaming]Display the failed output op in Streaming UI This PR implements the following features for both `master` and `branch-1.5`. 1. Display the failed output op count in the batch list 2. Display the failure reason of output op in the batch detail page Screenshots: https://cloud.githubusercontent.com/assets/1000778/10198387/5b2b97ec-67ce-11e5-81c2-f818b9d2f3ad.png;> https://cloud.githubusercontent.com/assets/1000778/10198388/5b76ac14-67ce-11e5-8c8b-de2683c5b485.png;> There are still two remaining problems in the UI. 1. If an output operation doesn't run any spark job, we cannot get the its duration since now it's the sum of all jobs' durations. 2. If an output operation doesn't run any spark job, we cannot get the description since it's the latest job's call site. We need to add new `StreamingListenerEvent` about output operations to fix them. So I'd like to fix them only for `master` in another PR. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zsxwing/spark batch-failure Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8950.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8950 commit f940ccf4fb3c614940320ea94539e70fbd2bfacf Author: zsxwingDate: 2015-09-30T10:17:04Z Display the failed output op in Streaming UI 1. Display the failed output op count in the batch list 2. Display the failure reason of output op in the batch detail page --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-144460802 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-144460866 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-144461863 [Test build #43133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43133/consoleFull) for PR 8950 at commit [`f940ccf`](https://github.com/apache/spark/commit/f940ccf4fb3c614940320ea94539e70fbd2bfacf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-144474528 [Test build #43133 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43133/console) for PR 8950 at commit [`f940ccf`](https://github.com/apache/spark/commit/f940ccf4fb3c614940320ea94539e70fbd2bfacf). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-144474660 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43133/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8950#issuecomment-144474659 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org