venkata91 opened a new pull request, #58687: URL: https://github.com/apache/spark/pull/58687
Stacked on apache/spark#58437. The final commit is the storage-aware dynamic-allocation change; the preceding commits belong to the dependency. A Spark JIRA ID is still needed before this is marked ready for review. ### What changes were proposed in this pull request? - Carry the resolved per-shuffle reliable-storage bit through both regular and streaming shuffle output trackers. - Let `ExecutorMonitor` query that bit when processing `SparkListenerJobStart`. - Exclude reliably stored shuffles from dynamic-allocation shuffle tracking while retaining existing tracking for local or unknown-storage shuffles. ### Why are the changes needed? When shuffle tracking and a reliable remote shuffle service are both enabled, Spark currently tracks every shuffle-producing executor without consulting the shuffle storage location. Executors that wrote remotely stored shuffle data therefore remain pinned by `spark.dynamicAllocation.shuffleTracking.timeout`, even though their output survives executor removal. This is particularly costly for mixed fallback configurations: local fallback shuffles need tracking, while remote shuffles do not. The per-shuffle reliability state introduced by apache/spark#58437 provides the distinction, but `ExecutorMonitor` does not currently consume it. ### Does this PR introduce _any_ user-facing change? Yes. With shuffle tracking enabled, executors holding only reliably stored shuffle output become eligible for the normal idle timeout. Executors holding local or unknown-storage shuffle output retain the existing active-shuffle and shuffle-timeout protection. ### How was this patch tested? Added coverage that reliably stored shuffles use the executor idle timeout while local shuffles remain protected by shuffle tracking, plus streaming-tracker coverage for the per-shuffle reliability bit. ``` build/sbt "core/testOnly org.apache.spark.StreamingShuffleOutputTrackerSuite org.apache.spark.scheduler.DAGSchedulerSuite org.apache.spark.scheduler.dynalloc.ExecutorMonitorSuite" core/scalastyle ``` 272 tests passed; core scalastyle reported no errors or warnings. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot (GPT-5.6 Sol Fast) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
