chinmay291 opened a new pull request, #20549:
URL: https://github.com/apache/hudi/pull/20549

   ### Describe the issue this Pull Request addresses
   
   Closes #20548
   
   Count-based compaction triggers count every delta commit since the last 
compaction, including delta commits that wrote no files. Streaming writers make 
such commits routinely, for example when only the source checkpoint advances. 
On the metadata table, every empty data table commit still creates a delta 
commit, so the metadata table is compacted far more often than its write volume 
needs. With the record-level index enabled, each of those inline compactions 
rewrites every record index file group.
   
   ### Summary and Changelog
   
   Adds an opt-in setting, off by default, that excludes empty delta commits 
from count-based compaction triggers.
   
   - **New configs**
     - `hoodie.compact.inline.ignore.empty.commits` (default `false`) and 
`hoodie.compact.inline.max.total.delta.commits` (default `-1`, meaning 5 x 
`hoodie.compact.inline.max.delta.commits`) in `HoodieCompactionConfig`.
     - `hoodie.metadata.compact.ignore.empty.commits` and 
`hoodie.metadata.compact.max.total.delta.commits` in `HoodieMetadataConfig`, 
passed to the metadata table write config in `HoodieMetadataWriteUtils`.
   - **`CompactionUtils.filterOutEmptyDeltaCommits`**
     - **Data table:** a delta commit is empty when it has no write stats.
     - **Metadata table:** a delta commit is empty when the data table commit 
at the same instant time has no write stats. The metadata table's own write 
stats can't tell, because an empty data commit still writes a `files` partition 
record.
     - Cleans, rollbacks, and instants without a readable data table commit 
count as non-empty, so the trigger errs towards compacting.
   - **`ScheduleCompactionActionExecutor.needCompact`**: when enabled, 
`NUM_COMMITS`, `NUM_COMMITS_AFTER_LAST_REQUEST`, `NUM_AND_TIME` and 
`NUM_OR_TIME` use the count of non-empty delta commits. Once total delta 
commits reach the cap, every delta commit counts again. The log compaction and 
time-only triggers are unchanged.
   - **`TimelineArchiverV1` / `TimelineArchiverV2`**: when enabled, every delta 
commit since the latest compaction is retained 
(`CompactionUtils.getEarliestInstantToRetainForCompaction` overload). Otherwise 
archival could shrink the non-empty count and postpone compaction indefinitely.
   
   ### Impact
   
   - New opt-in configs only. With the default (`false`), behaviour is 
unchanged.
   - When enabled, scheduling a compaction reads the commit metadata of each 
delta commit since the last compaction. On the metadata table it also reads the 
data table's active timeline. The number of reads is bounded by the 
total-delta-commits cap.
   
   ### Risk Level
   
   Low. The feature is off by default. When enabled, the trigger errs towards 
compacting whenever an instant can't be classified, and the cap bounds how long 
compaction can be postponed. Tests cover data and metadata table filtering, the 
trigger count with and without the cap, archival retention, and an end-to-end 
metadata table scenario: empty data commits don't trigger the compaction when 
the flag is on, and do when it's off.
   
   ### Documentation Update
   
   Config descriptions are included in the new `ConfigProperty` definitions. A 
website update for the new configs can follow once the approach is agreed.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to