[
https://issues.apache.org/jira/browse/SPARK-59776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59776:
-----------------------------------
Labels: pull-request-available (was: )
> Dynamic allocation shuffle tracking never releases executors when SQL shuffle
> file cleanup is enabled
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-59776
> URL: https://issues.apache.org/jira/browse/SPARK-59776
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
> Affects Versions: 4.3.0
> Reporter: Saurabh Damle
> Priority: Major
> Labels: pull-request-available
>
> When `spark.sql.classic.shuffleDependency.fileCleanup.enabled` (or the
> connect/thriftserver
> variants) is enabled, SQLExecution.cleanupShuffleDependencies removes shuffle
> files by calling
> `sc.shuffleDriverComponents.removeShuffle` directly. This bypasses
> ContextCleaner, so
> `CleanerListener.shuffleCleaned` is never invoked and ExecutorMonitor
> continues to consider
> those shuffles active.
> With `spark.dynamicAllocation.shuffleTracking.enabled=true` and the default
> (infinite)
> `spark.dynamicAllocation.shuffleTracking.timeout`, executors that hold the
> (already deleted)
> shuffle data are never released after actions like collect() / toPandas().
> Proposed fix: notify ContextCleaner listeners when SQL cleanup removes
> shuffle files, so
> ExecutorMonitor drops the shuffle IDs.
> Reported in https://github.com/apache/spark/issues/59014
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
