rangareddy commented on issue #15479: URL: https://github.com/apache/hudi/issues/15479#issuecomment-5056513619
This improvement was reviewed as part of the JIRA-migrated backlog triage. The ask - use a mapPartitions-style plan instead of a per-partition map() during clean planning for tables with very large partition counts - is still valid on current master. CleanPlanActionExecutor still drives planning through context.map(...) bounded by cleanerParallelism (it now batches partitions in chunks to bound timeline-server memory, but does not use a mapPartitions plan). The only linked PR, #6890, was closed unmerged, so this was never implemented. Fix scope is larger than a trivial patch and needs rebase/rework. Keeping open and tracking as stream:fix / clean_archive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
