Github user victor-wong commented on the issue:
https://github.com/apache/spark/pull/19824
@CodingCat
Thank you:)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user CodingCat commented on the issue:
https://github.com/apache/spark/pull/19824
if you just worry about
> As I was using the StreamingListenerBatchCompleted to do some metadata
checkpointing stuff, which should be done only when the batch succeeded.
If
Github user victor-wong commented on the issue:
https://github.com/apache/spark/pull/19824
@CodingCat
> One thing to note is that mute an event is a behavior change
I agree with that, so we should be careful about changing the current
behavior. I will close the PR later.
Github user CodingCat commented on the issue:
https://github.com/apache/spark/pull/19824
One thing to note is that mute an event is a behavior change, if a user has
introduced some customized listener to capture all completed batches and also
extract failed job info, he/she will see
Github user CodingCat commented on the issue:
https://github.com/apache/spark/pull/19824
#16542 has guaranteed that the failed batch can be re-executed, and I
didnât check if reverting the change in #16542 plus your new change can
guarantee the same thing...
Suppose it
Github user victor-wong commented on the issue:
https://github.com/apache/spark/pull/19824
@CodingCat please checkout the difference between the two PR.
`
if (jobSet.hasCompleted) {
- jobSets.remove(jobSet.time)
-
Github user CodingCat commented on the issue:
https://github.com/apache/spark/pull/19824
`What I want to say is that if a Job is failed, we should consider the
Batch as not completed.` isn't #16542 doing the same thing?
---
Github user victor-wong commented on the issue:
https://github.com/apache/spark/pull/19824
@CodingCat Yes, this PR wants to solve the same issue in
https://github.com/apache/spark/pull/16542, but I think this is a better way to
solve it.
If a Job failed, I think we should not
Github user victor-wong commented on the issue:
https://github.com/apache/spark/pull/19824
@viirya Sorry for the misleading title, I have changed it now.
---
-
To unsubscribe, e-mail:
