[ https://issues.apache.org/jira/browse/HUDI-945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
sivabalan narayanan updated HUDI-945: ------------------------------------- Sprint: Cont' improve - 2022/02/14 > Cleanup spillable map files eagerly as part of close > ---------------------------------------------------- > > Key: HUDI-945 > URL: https://issues.apache.org/jira/browse/HUDI-945 > Project: Apache Hudi > Issue Type: Bug > Components: writer-core > Reporter: Balaji Varadarajan > Assignee: sivabalan narayanan > Priority: Major > Labels: core-flow-ds, pull-request-available, sev:high > Fix For: 0.11.0 > > > Currently, files used by external spillable map are deleted on exits. For > spark-streaming/deltastreamer continuous-mode cases which runs several > iterations, it is better to eagerly delete files on closing the handles using > it. > We need to eagerly delete the files on following cases: > # HoodieMergeHandle > # HoodieMergedLogRecordScanner > # SpillableMapBasedFileSystemView -- This message was sent by Atlassian Jira (v8.20.1#820001)