Github user squito commented on the issue:
https://github.com/apache/spark/pull/16620
@kayousterhout I think your fix is correct, but I actually think its a
bigger change in behavior, one that has been explicitly argued *against* in the
past. I think the idea is that if you've got a bunch of tasks completing from
an old attempt for a stage, you dont' want to throw all that work away, as
@jinxing64 mentioned.
You may have a large number of resources tied up computing tasks from a
previous attempt, and the results are completely correct, but you still throw
those results away. (Its especially bad since we're [still not canceling tasks
from previous attempts](https://issues.apache.org/jira/browse/SPARK-2666).) I
do think that code would be simplified with the change you are suggesting --
late task completions from an earlier stage have been the cause of more bugs in
the past. and this is all only happening when there is a fetch failure, not
when everything is running smoothly.
But I do think its a rather large change in behavior which we should weigh
carefully. I was even worried that the change I was proposing would lead to
some cases where tasks would get fully computed, and then the results would get
thrown away, but it was necessary for correctness.
Also I should mention, that I'm not even 100% sure about this -- I have to
admit I find the epoch logic to be confusing and perhaps with a careful read,
we'll see there really isn't much more that is getting thrown away than was
already from epochs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
