deepakpanda93 commented on issue #8567: URL: https://github.com/apache/hudi/issues/8567#issuecomment-4844937422
This is fixed in the 1.0.0 GA release. Root cause: a timed-out/failed compaction left a dangling pending compaction instant in the data-table timeline. Metadata-table (MDT) compaction stalls behind any pending/inflight data-table instant, so MDT stopped compacting and the log files grew unbounded — and the failed compaction wasn't being rolled back to let later runs recover. Fix: HUDI-7510 / [#10874](https://github.com/apache/hudi/pull/10874) ("Loosen the compaction scheduling and rollback check for MDT") loosens the gating so MDT compaction no longer blocks indefinitely on a pending data-table operation and the failed/pending compaction is handled. It's in 1.0.0 GA and all later 1.x releases. (As @danny0405 noted earlier in this thread, there wasn't a clean fix for the 0.x line — the resolution is to move to 1.x.) The 1.0 timeline rework (completion-time-based timeline) and failed-write rollback improvements further harden this area. Closing as fixed in 1.0.0. If you still see MDT/log files growing due to a stuck compaction on a recent 1.x release, please reopen with a sorted .hoodie/ listing and we'll investigate. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
