[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145947026
  
  [Test build #43278 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43278/console)
 for   PR 8950 at commit 
[`d7c42c9`](https://github.com/apache/spark/commit/d7c42c9e8599fd1dc7ab4bc732b1af29647cf6b9).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145947179
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43278/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145947177
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-146027302
  
LGTM. Merging this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/8950


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145928461
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145928440
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145930082
  
  [Test build #43278 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43278/consoleFull)
 for   PR 8950 at commit 
[`d7c42c9`](https://github.com/apache/spark/commit/d7c42c9e8599fd1dc7ab4bc732b1af29647cf6b9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145928029
  
> In the case of "Failed", unless the details is opened, there is not 
indication of failure. So it might be better to show "Failed due to error: 
$exceptionMessage", and the full stacktrace in the detail.

Added it.
https://cloud.githubusercontent.com/assets/1000778/10315755/b84cbfb8-6c8e-11e5-9cea-0b2b408afd05.png;>

> In case of "Failed due to Spark error", does it make sense to have the 
error show up in both places?

This seems unnecessary. The job error column has already shown it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145738954
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145738897
  
  [Test build #43270 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43270/console)
 for   PR 8950 at commit 
[`ca68ac8`](https://github.com/apache/spark/commit/ca68ac858462fff107d8a5ce7a5af3cf9416aca3).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145738955
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43270/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145743591
  
This looks cool! Also I like that you used "details" for that. In the case 
of "Failed", unless the details is opened, there is not indication of failure. 
So it might be better to show "Failed due to error: $exceptionMessage", and the 
full stacktrace in the detail. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/8950#discussion_r41217146
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala ---
@@ -265,16 +323,20 @@ private[ui] class BatchPage(parent: StreamingTab) 
extends WebUIPage("batch") {
* Generate the job table for the batch.
*/
   private def generateJobTable(batchUIData: BatchUIData): Seq[Node] = {
-val outputOpIdToSparkJobIds = 
batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).toSeq.
-  sortBy(_._1). // sorted by OutputOpId
+val outputOpIdToSparkJobIds = 
batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).
   map { case (outputOpId, outputOpIdAndSparkJobIds) =>
 // sort SparkJobIds for each OutputOpId
 (outputOpId, outputOpIdAndSparkJobIds.map(_.sparkJobId).sorted)
   }
+val outputOps = (0 until batchUIData.numOutputOp).map { outputOpId =>
+  val status = batchUIData.failureReason.getOrElse(outputOpId, 
"Succeeded")
--- End diff --

I don't think so. E.g., the use may write the following codes:

```
stream.foreachRDD { rdd =>
  try {
rdd.foreach(...)
  } catch {
 ...
  }
}
```

In this case, if a Spark job (rdd.foreach) fails, we cannot say the output 
op fails.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/8950#discussion_r41219078
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala ---
@@ -265,16 +323,20 @@ private[ui] class BatchPage(parent: StreamingTab) 
extends WebUIPage("batch") {
* Generate the job table for the batch.
*/
   private def generateJobTable(batchUIData: BatchUIData): Seq[Node] = {
-val outputOpIdToSparkJobIds = 
batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).toSeq.
-  sortBy(_._1). // sorted by OutputOpId
+val outputOpIdToSparkJobIds = 
batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).
   map { case (outputOpId, outputOpIdAndSparkJobIds) =>
 // sort SparkJobIds for each OutputOpId
 (outputOpId, outputOpIdAndSparkJobIds.map(_.sparkJobId).sorted)
   }
+val outputOps = (0 until batchUIData.numOutputOp).map { outputOpId =>
+  val status = batchUIData.failureReason.getOrElse(outputOpId, 
"Succeeded")
--- End diff --

Here is a possible logic.
If no failure reason, 
then "Succeeded" 
Else if failure reason contains "SparkException"
then "Failed due to Spark job error"
Else 
then "Failed" 

This should work fine for most cases, where the user is not doing fancy 
things like catching exceptions themselves and ignoring/rethrowing them. Isnt 
it? 

Consider your example. If the user catches Spark job exception (most 
probably SparkException) and rethrows it, the above logic should identify it as 
Spark job error and say "Failed due to Spark job error". On the other hand, if 
the user catches and ignore exception, then failure reason will be empty and 
the output will be marked as "Succeeded" (even though Spark job error will not 
be empty, which is okay).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/8950#discussion_r41188944
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala ---
@@ -265,16 +323,20 @@ private[ui] class BatchPage(parent: StreamingTab) 
extends WebUIPage("batch") {
* Generate the job table for the batch.
*/
   private def generateJobTable(batchUIData: BatchUIData): Seq[Node] = {
-val outputOpIdToSparkJobIds = 
batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).toSeq.
-  sortBy(_._1). // sorted by OutputOpId
+val outputOpIdToSparkJobIds = 
batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).
   map { case (outputOpId, outputOpIdAndSparkJobIds) =>
 // sort SparkJobIds for each OutputOpId
 (outputOpId, outputOpIdAndSparkJobIds.map(_.sparkJobId).sorted)
   }
+val outputOps = (0 until batchUIData.numOutputOp).map { outputOpId =>
+  val status = batchUIData.failureReason.getOrElse(outputOpId, 
"Succeeded")
--- End diff --

Cant we distinguish based on the whether there is a Spark job error in any 
of the jobs associated with the output op


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/8950#discussion_r41220303
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala ---
@@ -265,16 +323,20 @@ private[ui] class BatchPage(parent: StreamingTab) 
extends WebUIPage("batch") {
* Generate the job table for the batch.
*/
   private def generateJobTable(batchUIData: BatchUIData): Seq[Node] = {
-val outputOpIdToSparkJobIds = 
batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).toSeq.
-  sortBy(_._1). // sorted by OutputOpId
+val outputOpIdToSparkJobIds = 
batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).
   map { case (outputOpId, outputOpIdAndSparkJobIds) =>
 // sort SparkJobIds for each OutputOpId
 (outputOpId, outputOpIdAndSparkJobIds.map(_.sparkJobId).sorted)
   }
+val outputOps = (0 until batchUIData.numOutputOp).map { outputOpId =>
+  val status = batchUIData.failureReason.getOrElse(outputOpId, 
"Succeeded")
--- End diff --

Sound great. I will update the logic.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145732880
  
New screenshot:

![3](https://cloud.githubusercontent.com/assets/1000778/10299535/bd291c50-6c1d-11e5-92c7-131c0630fad8.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145733024
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145733038
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-145734108
  
  [Test build #43270 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43270/consoleFull)
 for   PR 8950 at commit 
[`ca68ac8`](https://github.com/apache/spark/commit/ca68ac858462fff107d8a5ce7a5af3cf9416aca3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/8950#discussion_r41128898
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala ---
@@ -265,16 +323,20 @@ private[ui] class BatchPage(parent: StreamingTab) 
extends WebUIPage("batch") {
* Generate the job table for the batch.
*/
   private def generateJobTable(batchUIData: BatchUIData): Seq[Node] = {
-val outputOpIdToSparkJobIds = 
batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).toSeq.
-  sortBy(_._1). // sorted by OutputOpId
+val outputOpIdToSparkJobIds = 
batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).
   map { case (outputOpId, outputOpIdAndSparkJobIds) =>
 // sort SparkJobIds for each OutputOpId
 (outputOpId, outputOpIdAndSparkJobIds.map(_.sparkJobId).sorted)
   }
+val outputOps = (0 until batchUIData.numOutputOp).map { outputOpId =>
+  val status = batchUIData.failureReason.getOrElse(outputOpId, 
"Succeeded")
--- End diff --

> In fact, there should be "Failed" and "Failed due to Spark job error". In 
the first case, the output op error will be below the "Failed" .

I think now we cannot distinguish these two errors. They are both caught in 
`org.apache.spark.streaming.scheduler.Job.run`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/8950#discussion_r41077235
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala ---
@@ -265,16 +323,20 @@ private[ui] class BatchPage(parent: StreamingTab) 
extends WebUIPage("batch") {
* Generate the job table for the batch.
*/
   private def generateJobTable(batchUIData: BatchUIData): Seq[Node] = {
-val outputOpIdToSparkJobIds = 
batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).toSeq.
-  sortBy(_._1). // sorted by OutputOpId
+val outputOpIdToSparkJobIds = 
batchUIData.outputOpIdSparkJobIdPairs.groupBy(_.outputOpId).
   map { case (outputOpId, outputOpIdAndSparkJobIds) =>
 // sort SparkJobIds for each OutputOpId
 (outputOpId, outputOpIdAndSparkJobIds.map(_.sparkJobId).sorted)
   }
+val outputOps = (0 until batchUIData.numOutputOp).map { outputOpId =>
+  val status = batchUIData.failureReason.getOrElse(outputOpId, 
"Succeeded")
--- End diff --

If we are showing "Succeeded" for successful output ops, then we should 
also show "Failed" for failed ones as well. In fact, there should be "Failed" 
and "Failed due to Spark job error". In the first case, the output op error 
will be below the "Failed" .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-144460172
  
/cc @tdas 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[zsxwing]

GitHub user zsxwing opened a pull request:

https://github.com/apache/spark/pull/8950

[SPARK-10885][Streaming]Display the failed output op in Streaming UI

This PR implements the following features for both `master` and 
`branch-1.5`.
1. Display the failed output op count in the batch list
2. Display the failure reason of output op in the batch detail page

Screenshots:
https://cloud.githubusercontent.com/assets/1000778/10198387/5b2b97ec-67ce-11e5-81c2-f818b9d2f3ad.png;>
https://cloud.githubusercontent.com/assets/1000778/10198388/5b76ac14-67ce-11e5-8c8b-de2683c5b485.png;>

There are still two remaining problems in the UI.
1. If an output operation doesn't run any spark job, we cannot get the its 
duration since now it's the sum of all jobs' durations.
2. If an output operation doesn't run any spark job, we cannot get the 
description since it's the latest job's call site.

We need to add new `StreamingListenerEvent` about output operations to fix 
them. So I'd like to fix them only for `master` in another PR.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zsxwing/spark batch-failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8950.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8950


commit f940ccf4fb3c614940320ea94539e70fbd2bfacf
Author: zsxwing 
Date:   2015-09-30T10:17:04Z

Display the failed output op in Streaming UI

1. Display the failed output op count in the batch list
2. Display the failure reason of output op in the batch detail page




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-144460802
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-144460866
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-144461863
  
  [Test build #43133 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43133/consoleFull)
 for   PR 8950 at commit 
[`f940ccf`](https://github.com/apache/spark/commit/f940ccf4fb3c614940320ea94539e70fbd2bfacf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-144474528
  
  [Test build #43133 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43133/console)
 for   PR 8950 at commit 
[`f940ccf`](https://github.com/apache/spark/commit/f940ccf4fb3c614940320ea94539e70fbd2bfacf).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-144474660
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43133/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10885][Streaming]Display the failed out...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8950#issuecomment-144474659
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org