viirya commented on PR #58097: URL: https://github.com/apache/spark/pull/58097#issuecomment-5612379825
Thanks for the review. Pushed the changes as `e569cd39841`. The main change is a more conservative eligibility policy: RDD boundaries, cache construction/reads, and mixed AQE shuffle plans retain regular execution. In particular, this gives up the materialized-prefix/pipelined-tail optimization for now, so repeated actions can use the existing regular-shuffle recovery path. I also added job-scoped epoch cleanup, including protection against late tasks recreating completed state and against one action’s cleanup removing another action’s active queues. Concurrent sharing of a pipelined exchange remains unsupported, and multi-job consumers still recompute their pipelined producers. Those limitations, along with the slot requirements, are now documented and covered by tests. Validation: 269 core/scheduler tests and 34 SQL tests passed. Scalastyle passed for the affected modules. Details below in the individual threads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
