zhang-arvin commented on PR #19806: URL: https://github.com/apache/hudi/pull/19806#issuecomment-6059319325
Thanks @danny0405 — that's a clear direction, and it confirms this PR is the wrong shape. Reading it back, the general fix lives in the file-group-id scheme rather than the call sites: 1. drop the hard-coded bucket-num-suffixed file group id, so a partition's file groups can be recreated without colliding with retired/replaced ids; 2. make file-group-id creation atomic for a given (partition, bucket) so concurrent tasks from multiple jobs don't race on it; 3. invalidate the Flink bucket write function's cached `bucket_num -> file group id` mapping as soon as a replace commit lands for that partition. Agreed a per-operation fix in the delete-partition caller doesn't cover that. @hudi-agent's inline note points at the same thing from the other side: bumping the generation once only survives a single delete/re-insert cycle — a second cycle retires the bumped id and re-inserts into it again, so the loss comes back. Given that, I'd rather not land this as-is. Which would you prefer: (a) I convert this PR to a draft and rework it into the file-group-id change above, or (b) I close it and open a separate design issue/PR for the general solution, referenced from #19793? Happy to draft the design in #19793 first either way if that helps the triage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
