chinmay291 opened a new issue, #20548: URL: https://github.com/apache/hudi/issues/20548
### Feature Description **What the feature achieves:** An opt-in way for count-based compaction triggers to ignore delta commits that wrote no records, for both merge-on-read data tables and the metadata table. **Why this feature is needed:** The count-based compaction triggers (`NUM_COMMITS`, `NUM_COMMITS_AFTER_LAST_REQUEST`, `NUM_AND_TIME`, `NUM_OR_TIME`) count every completed delta commit since the last compaction, including delta commits that wrote no records. Streaming writers produce such commits routinely. For example, Hudi Streamer makes an empty commit whenever the source checkpoint advances but no records match, just to persist the new checkpoint. With a short sync interval, there can be many empty commits for every commit that carries data. This hurts most on the metadata table, which is compacted inline: - Every data table commit, empty or not, produces a metadata table delta commit. Even for an empty data commit, that delta commit writes a record to the `files` partition. - Empty data commits therefore count towards `hoodie.metadata.compact.max.delta.commits` like real ones. - With the record-level index enabled, every metadata table compaction rewrites all record index file groups. For large tables this is expensive, and inline compaction blocks the writer while it runs. As a result, the metadata table can be compacted several times more often than its actual write volume warrants. For example, with `hoodie.metadata.compact.max.delta.commits=10`, a writer that makes 3 data commits and 7 empty commits triggers a compaction even though only 3 delta commits carried changes. Raising `max.delta.commits` reduces the frequency, but it also delays compaction for tables with real write activity, because it doesn't tell empty commits from real ones. ### User Experience **How users will use this feature:** New configs, all off by default: | Config | Default | Purpose | |---|---|---| | `hoodie.compact.inline.ignore.empty.commits` | `false` | MoR data tables: don't count empty delta commits towards `hoodie.compact.inline.max.delta.commits` | | `hoodie.metadata.compact.ignore.empty.commits` | `false` | Metadata table: don't count delta commits whose data table commit wrote no records | | `hoodie.compact.inline.max.total.delta.commits` / `hoodie.metadata.compact.max.total.delta.commits` | `-1` (5 x `max.delta.commits`) | Safety cap: once total delta commits, empty ones included, reach this, compaction triggers anyway. Bounds the number of log files and the length of the active timeline | Behaviour when enabled: - **Data table:** a delta commit is empty when it wrote and deleted no records. - **Metadata table:** a delta commit is empty when the data table commit at the same instant time wrote and deleted no records. The metadata table's own write stats can't tell, because empty data commits still write a `files` partition record. - Cleans, rollbacks, and any instant whose metadata can't be read still count, so the trigger errs towards compacting. - **Archival:** while the flag is on, the archiver keeps every delta commit since the latest compaction, so archiving can't shrink the count and postpone compaction indefinitely. - Time-based triggers and the log compaction trigger are unchanged. ### Hudi RFC Requirements **RFC PR link:** N/A **Why RFC is/isn't needed:** The change is opt-in and limited to how the existing compaction trigger counts delta commits. It adds no storage format, timeline or API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
