voonhous commented on code in PR #19572: URL: https://github.com/apache/hudi/pull/19572#discussion_r3757355581
########## website/docs/compaction.md: ########## @@ -283,6 +283,27 @@ Offline compaction needs to submit the Flink task on the command line. The progr The retry options (`--retry`, `--retry-last-failed-job`, `--job-max-processing-time-ms`) are only effective in single-run mode, not in service mode. Service mode has implicit retry semantics via its continuous monitoring loop. A warning will be logged if `--retry-last-failed-job` is enabled but `--job-max-processing-time-ms` is not set to a positive value. ::: +## Log Compaction + +Log compaction is a minor compaction for Merge-on-Read tables. Rather than merging log files into a new base file, it +stitches several small log blocks into a larger one within the same file group. A file group that receives frequent +small updates can therefore be kept efficient without paying the cost of rewriting its base file. Readers skip the log +blocks that have already been stitched, so read amplification is reduced as well. Log compaction appears on the timeline +as a `logcompaction` action. + +| Config Name | Default | Description | +|---|---|---| +| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the log compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br /><br />`Config Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction can be scheduled once the number of log blocks crosses this threshold. Effective only when log compaction is enabled via `hoodie.log.compaction.inline`.<br /><br />`Config Param: LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.enable` | `false` (Optional) | By enabling log compaction through this config, log compaction also gets enabled for the metadata table.<br /><br />`Config Param: ENABLE_LOG_COMPACTION`<br />`Since Version: 0.14.0` | Review Comment: Verified independently and resolving. `hoodie.log.compaction.enable` no longer appears in any config table. Its only occurrence in the section is the `:::caution` at line 328, which states outright that it is not a switch to set on your table. I confirmed the underlying behaviour at release-1.0.0, 1.0.1, 1.0.2, 1.1.1, 1.2.0 and master: `ENABLE_LOG_COMPACTION` is read only via `metadataWriteConfig.isLogCompactionEnabled()`, and `HoodieMetadataWriteUtils.createMetadataWriteConfig` unconditionally overwrites it from `hoodie.metadata.log.compaction.enable`. The caution is accurate in every version the section now ships to. Your original read was right -- a reader could well have set it expecting an effect on their MOR data. The fix goes further than the clarifier you suggested by dropping the row entirely and documenting the two real metadata-table knobs instead. ########## website/docs/compaction.md: ########## @@ -283,6 +283,27 @@ Offline compaction needs to submit the Flink task on the command line. The progr The retry options (`--retry`, `--retry-last-failed-job`, `--job-max-processing-time-ms`) are only effective in single-run mode, not in service mode. Service mode has implicit retry semantics via its continuous monitoring loop. A warning will be logged if `--retry-last-failed-job` is enabled but `--job-max-processing-time-ms` is not set to a positive value. ::: +## Log Compaction + +Log compaction is a minor compaction for Merge-on-Read tables. Rather than merging log files into a new base file, it +stitches several small log blocks into a larger one within the same file group. A file group that receives frequent +small updates can therefore be kept efficient without paying the cost of rewriting its base file. Readers skip the log +blocks that have already been stitched, so read amplification is reduced as well. Log compaction appears on the timeline +as a `logcompaction` action. + +| Config Name | Default | Description | +|---|---|---| +| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the log compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br /><br />`Config Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction can be scheduled once the number of log blocks crosses this threshold. Effective only when log compaction is enabled via `hoodie.log.compaction.inline`.<br /><br />`Config Param: LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.enable` | `false` (Optional) | By enabling log compaction through this config, log compaction also gets enabled for the metadata table.<br /><br />`Config Param: ENABLE_LOG_COMPACTION`<br />`Since Version: 0.14.0` | + +:::note +Log compaction is scheduled inline through the configs above. Unlike compaction, it has no dedicated SQL procedure, Review Comment: Addressed and resolving. The note now states explicitly that there is no asynchronous log compaction service for the data table, no SQL procedure, Hudi CLI command or standalone utility, and that it is not exposed through Flink options -- while naming programmatic scheduling through `scheduleLogCompaction` / `logCompact`. I verified each of those against release-1.2.0 and master. The cross-page inconsistency you flagged is fixed too: `write_operations.md` step 9 now reads "If `hoodie.log.compaction.inline` is enabled, log compaction may also run", so it no longer implies a broader trigger than the section it links to. One async point is still open, but it is a different one and is tracked on a separate thread: the metadata table's log compaction can be delegated to an external table service manager, which Hudi does not ship, so that advice needs a stated prerequisite. It does not affect the data-table claim you asked about here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
