Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/8818#discussion_r40040978
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala
---
@@ -177,9 +177,12 @@ class JobScheduler(val ssc: StreamingContext) extends
Logging {
jobSet.totalDelay / 1000.0, jobSet.time.toString,
jobSet.processingDelay / 1000.0
))
-
listenerBus.post(StreamingListenerBatchCompleted(jobSet.toBatchInfo))
+
listenerBus.post(StreamingListenerBatchCompleted(jobSet.toBatchInfo()))
}
case Failure(e) =>
+ val jobSet = jobSets.get(job.time)
+ val errorMessage = Some(Utils.exceptionString(e))
+
listenerBus.post(StreamingListenerBatchCompleted(jobSet.toBatchInfo(errorMessage)))
--- End diff --
The jobset may not have completed at this point! There may be multiple
streaming jobs in the jobset, and when one streaming job fail, other streaming
job may pass. So even if there is a failure, you have to keep other "jobs" in
the batch that can finish successfully.
So the right thing to do is to keep track of the failures in the JobSet, at
the granularity of jobs. Eventually we may want to show the details failure
information - e.g. 1 out of 3 output operations failed in Spark Streaming
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
