mridulm commented on code in PR #38371:
URL: https://github.com/apache/spark/pull/38371#discussion_r1009018882
##########
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala:
##########
@@ -3089,13 +3089,14 @@ class DAGSchedulerSuite extends SparkFunSuite with
TempLocalSparkContext with Ti
submit(finalRdd, Array(0, 1), properties = new Properties())
// Finish the first 2 shuffle map stages.
- completeShuffleMapStageSuccessfully(0, 0, 2)
+ completeShuffleMapStageSuccessfully(0, 0, 2, Seq("hostA", "hostB"))
assert(mapOutputTracker.findMissingPartitions(shuffleId1) ===
Some(Seq.empty))
completeShuffleMapStageSuccessfully(1, 0, 2, Seq("hostB", "hostD"))
assert(mapOutputTracker.findMissingPartitions(shuffleId2) ===
Some(Seq.empty))
- // Executor lost on hostB, both of stage 0 and 1 should be reran.
+ // FetchFailed on stage 2, both of stage 1 and 2 should be reran. Besides,
executor lost on
+ // hostB, both of stage 0 and 1 should be reran.
Review Comment:
The comment could read - "Executor lost on hostB, both of stage 0 and 1
should be reran - as part of recomputation of stage 2" : since there is output
on hostB for stage 0 (in completeShuffleMapStageSuccessfully), and stage 1.
Something like "Executor lost on hostB, both of stage 0, 1 and 2 should be
reran due to missing outputs"
Will that clarify ?
##########
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala:
##########
@@ -3089,13 +3089,14 @@ class DAGSchedulerSuite extends SparkFunSuite with
TempLocalSparkContext with Ti
submit(finalRdd, Array(0, 1), properties = new Properties())
// Finish the first 2 shuffle map stages.
- completeShuffleMapStageSuccessfully(0, 0, 2)
+ completeShuffleMapStageSuccessfully(0, 0, 2, Seq("hostA", "hostB"))
assert(mapOutputTracker.findMissingPartitions(shuffleId1) ===
Some(Seq.empty))
completeShuffleMapStageSuccessfully(1, 0, 2, Seq("hostB", "hostD"))
assert(mapOutputTracker.findMissingPartitions(shuffleId2) ===
Some(Seq.empty))
- // Executor lost on hostB, both of stage 0 and 1 should be reran.
+ // FetchFailed on stage 2, both of stage 1 and 2 should be reran. Besides,
executor lost on
+ // hostB, both of stage 0 and 1 should be reran.
Review Comment:
The comment could read - "Executor lost on hostB, both of stage 0 and 1
should be reran - as part of recomputation of stage 2" : since there is output
on hostB for stage 0 (in completeShuffleMapStageSuccessfully), and stage 1.
Something like "Executor lost on hostB, both of stage 0, 1 and 2 should be
reran due to missing partitions"
Will that clarify ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
