kbuci opened a new issue, #19428: URL: https://github.com/apache/hudi/issues/19428
Currently, the HUDI delete_partition will replace an entire partition by finding all file ids in each target partition, and marking them as replaced. Later on when this replacecommit instant is eligible for clean, the clean table service will find all these replaced file ids and will delete them from DFS and remove their entried from MDT partitions (like FILES). This is correct and the same with how clustering/insert_overwrite replacecommits are handled. We have seen cases where, if we perform a delete_partition on many large partitions (large meaning 10,000+), then the subsequent clean will need more spark resources (since it has to process many file entries). For append-only datasets where we just insert files in new partitions and use delete_partition to delete old expired TTLed partitions, it would be ideal if we could avoid this cost for `clean`ing. As an optimization, could we update HUDI such that we could mark an entire partition as "replaced"? The idea is that all relevant flows that read instants for replaced file ids would be updated to avoiding needing to read/process each individual file id entry in a replaced partition. - The filesystem view APIs would return 0 file ids for a partition if it sees that the partition is marked as replaced in a replacecommit and no later instant has added new file ids to the partition (instead of having to update in the internal mapping of `partition -> [replace file id list]`) - Clean planner would not attempt to find each file id in a replaced partition - it can create a smaller set of each partition that has been marked for replace and hasn't had a later instant writing new files to it. Then the clean execution can issue DFS delete call for all files in the partition. And remove all related records in MDT partitions cc @nsivabalan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
