deepakpanda93 commented on code in PR #19572: URL: https://github.com/apache/hudi/pull/19572#discussion_r3756119406
########## website/docs/compaction.md: ########## @@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the command line. The progr The retry options (`--retry`, `--retry-last-failed-job`, `--job-max-processing-time-ms`) are only effective in single-run mode, not in service mode. Service mode has implicit retry semantics via its continuous monitoring loop. A warning will be logged if `--retry-last-failed-job` is enabled but `--job-max-processing-time-ms` is not set to a positive value. ::: +## Log Compaction + +Log compaction is a minor compaction for Merge-on-Read tables. Rather than merging log files into a new base file, it +stitches several small log blocks into a larger one within the same file group. A file group that receives frequent +small updates can therefore be kept efficient without paying the cost of rewriting its base file. Readers skip the log Review Comment: Fair, and taken. The section now leads with the mechanism and then the cost: > The stitched block is appended rather than replacing anything: the superseded blocks stay on disk until the next full compaction and clean. Log compaction therefore trades extra storage for fewer blocks to merge on read. A single run can also emit more than one block if the merged output exceeds the log block size. That covers both your points — the RFC-48 line about merged blocks only being cleaned by a full compaction, and Scenario 3's multi-block output, so "into a larger one" is no longer stated as an absolute. On the framing divergence you noted with `tech-specs.md:669`: I have left the tech spec alone here, since this PR is already being narrowed and I would rather not edit a second page's wording on the same pass. The two now differ in emphasis — the spec sells write amplification, this section sells fewer blocks on read and names the storage cost — which is defensible but not ideal. Happy to align them in a follow-up if you think the spec should also carry the cost side. 7a99aa130ab0. ########## website/docs/compaction.md: ########## @@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the command line. The progr The retry options (`--retry`, `--retry-last-failed-job`, `--job-max-processing-time-ms`) are only effective in single-run mode, not in service mode. Service mode has implicit retry semantics via its continuous monitoring loop. A warning will be logged if `--retry-last-failed-job` is enabled but `--job-max-processing-time-ms` is not set to a positive value. ::: +## Log Compaction + +Log compaction is a minor compaction for Merge-on-Read tables. Rather than merging log files into a new base file, it +stitches several small log blocks into a larger one within the same file group. A file group that receives frequent +small updates can therefore be kept efficient without paying the cost of rewriting its base file. Readers skip the log +blocks that have already been stitched, so read amplification is reduced as well. Log compaction appears on the timeline +as a `logcompaction` action. + +| Config Name | Default | Description | +|---|---|---| +| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the log compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br /><br />`Config Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction can be scheduled once the number of log blocks crosses this threshold. Effective only when log compaction is enabled via `hoodie.log.compaction.inline`.<br /><br />`Config Param: LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` | Review Comment: Both points confirmed and the row is rewritten essentially to your suggestion. 7a99aa130ab0. > Log compaction can be scheduled once a file slice has at least this many log files, or at least this many log blocks. Applies to any scheduling attempt, whether triggered inline or programmatically. You are right that I inherited the "effective only when enabled via `hoodie.log.compaction.inline`" clause from the upstream javadoc but then rewrote the sentence around it, which made me responsible for it. And it did directly contradict the note three lines below about `scheduleLogCompaction` — that is the kind of internal inconsistency I should have caught before pushing, since both lines were mine. The file-count half is the part I would not have found: `isFileSliceEligibleForLogCompaction` returning on log file count before counting blocks makes "number of log blocks crosses this threshold" wrong on its own terms, independent of the trigger question. ########## website/docs/compaction.md: ########## @@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the command line. The progr The retry options (`--retry`, `--retry-last-failed-job`, `--job-max-processing-time-ms`) are only effective in single-run mode, not in service mode. Service mode has implicit retry semantics via its continuous monitoring loop. A warning will be logged if `--retry-last-failed-job` is enabled but `--job-max-processing-time-ms` is not set to a positive value. ::: +## Log Compaction + +Log compaction is a minor compaction for Merge-on-Read tables. Rather than merging log files into a new base file, it +stitches several small log blocks into a larger one within the same file group. A file group that receives frequent +small updates can therefore be kept efficient without paying the cost of rewriting its base file. Readers skip the log +blocks that have already been stitched, so read amplification is reduced as well. Log compaction appears on the timeline +as a `logcompaction` action. + +| Config Name | Default | Description | +|---|---|---| +| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the log compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br /><br />`Config Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction can be scheduled once the number of log blocks crosses this threshold. Effective only when log compaction is enabled via `hoodie.log.compaction.inline`.<br /><br />`Config Param: LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` | + +:::note +`hoodie.log.compaction.inline` is the only built-in way to schedule log compaction on a data table. There is no +asynchronous log compaction service, SQL procedure, Hudi CLI command, or standalone utility for it, unlike compaction. +Programmatic scheduling is available through the write client's `scheduleLogCompaction` and `logCompact` methods. Review Comment: Both gaps confirmed and covered, scoped to the two copies that survive. 7a99aa130ab0. The note now reads: > `hoodie.log.compaction.inline` is the only built-in way to schedule log compaction on a data table. There is no asynchronous log compaction service for the data table, and no SQL procedure, Hudi CLI command or standalone utility, unlike compaction. It is also not exposed through Flink options, so Flink cannot schedule it for a data table. Programmatic scheduling is available through the write client's `scheduleLogCompaction` and `logCompact` methods. and the metadata-table paragraph now carries the async path: > The metadata table runs its own log compaction, controlled by a separate pair of configs. Unlike the data table, it can be delegated to an async pipeline by setting `hoodie.metadata.table.service.manager.enabled` together with `hoodie.metadata.table.service.manager.actions=logcompaction`. Worth noting what went wrong on my side: I had checked for an `AsyncLogCompactService` class and found none, then generalised that to "no async, full stop". The metadata table's async path runs through the table service manager configs rather than a service class, so my search could not have found it. The Flink gap is the sharper catch — the second half of this page being Flink offline compaction means a Flink MOR user lands directly on advice that cannot apply to them, and nothing said so. Since the section is now restricted to `next` + 1.2.0, the version qualification you noted (neither exists at 1.1.1 or earlier) no longer needs stating. ########## website/docs/compaction.md: ########## @@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the command line. The progr The retry options (`--retry`, `--retry-last-failed-job`, `--job-max-processing-time-ms`) are only effective in single-run mode, not in service mode. Service mode has implicit retry semantics via its continuous monitoring loop. A warning will be logged if `--retry-last-failed-job` is enabled but `--job-max-processing-time-ms` is not set to a positive value. ::: +## Log Compaction + +Log compaction is a minor compaction for Merge-on-Read tables. Rather than merging log files into a new base file, it +stitches several small log blocks into a larger one within the same file group. A file group that receives frequent +small updates can therefore be kept efficient without paying the cost of rewriting its base file. Readers skip the log +blocks that have already been stitched, so read amplification is reduced as well. Log compaction appears on the timeline +as a `logcompaction` action. + +| Config Name | Default | Description | +|---|---|---| +| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the log compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br /><br />`Config Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction can be scheduled once the number of log blocks crosses this threshold. Effective only when log compaction is enabled via `hoodie.log.compaction.inline`.<br /><br />`Config Param: LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` | + +:::note +`hoodie.log.compaction.inline` is the only built-in way to schedule log compaction on a data table. There is no +asynchronous log compaction service, SQL procedure, Hudi CLI command, or standalone utility for it, unlike compaction. +Programmatic scheduling is available through the write client's `scheduleLogCompaction` and `logCompact` methods. +::: + +The metadata table runs its own log compaction, controlled by a separate pair of configs: Review Comment: Point 2 is now in the section; point 1 became moot. 7a99aa130ab0. > While a log compaction is pending on the metadata table, major compaction of the metadata table is not scheduled, since metadata partitions such as the record level index rely on processing-time ordering. See [HUDI-7533](https://issues.apache.org/jira/browse/HUDI-7533). That is a real operational trap — enabling MDT log compaction quietly changing when MDT major compaction can be scheduled is not something a reader would infer from either config's description. Point 1, the `hoodie.metadata.optimized.log.blocks.scan.enable` pairing, was scoped to the pre-1.2.0 copies, and those are now reverted, so there is nowhere left for that sentence to go. If the same gap exists at 1.2.0 for the MDT reader I would want to add it — but `HoodieBackedTableMetadata` passing `metadataConfig.isOptimizedLogBlocksScanEnabled()` while the data-table branch was removed at 1.2.0 is a combination I have not verified, so I have not written anything about it rather than guess. Worth a separate look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
