Venkata krishnan Sowrirajan created SPARK-61465:
---------------------------------------------------
Summary: Avoid map-output epoch bump on an empty host cleanup when
all shuffles are reliably stored and push-based shuffle is on
Key: SPARK-61465
URL: https://issues.apache.org/jira/browse/SPARK-61465
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.3.1
Reporter: Venkata krishnan Sowrirajan
In SPARK-59138, `handleExecutorLost` skips map-output cleanup when
`ShuffleDriverComponents.supportsReliableStorage()` is true and no registered
shuffle is unreliable. This matches the behavior before that change. With
push-based shuffle on, a loss that names a worker host still runs
`MapOutputTrackerMaster.removeOutputsOnHost(host, respectReliablyStored =
true)` so the host's merge results are dropped.
If that host held no map output and no merge results, the pass returns
`CleanupOutcome(metadataChanged = false, preservedReliable = false)`.
`shouldBumpEpoch` is then true, so the map-output epoch advances. Every
executor drops its cached map and merge statuses and refetches them from the
driver, even though nothing changed. Before SPARK-59138 this path did nothing
for these applications.
We can't just stop bumping on an empty pass. The tracker relies on "a no-op
loss still bumps the epoch" for executor-failure fencing whenever unreliable
shuffles are registered.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]