[
https://issues.apache.org/jira/browse/SPARK-35570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-35570.
----------------------------------
Resolution: Invalid
> Shuffle file leak with external shuffle service enable
> ------------------------------------------------------
>
> Key: SPARK-35570
> URL: https://issues.apache.org/jira/browse/SPARK-35570
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Shuffle
> Affects Versions: 3.1.2
> Reporter: ZiyueGuan
> Priority: Major
>
> Unlike rdd block, external shuffle service doesn't offer a cleaning up of
> shuffle file. The cleaning up of shuffle file mainly rely on alive executors
> to response the request from context cleaner. As long as the executor exit,
> the shuffle file left will not be cleaned until application exits. For
> streaming application or long running application, disk may run out.
> I'm confused that shuffle file was left like above while the lifecycle of rdd
> block was properly handled. Is there any difference between them?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
