hudi-agent commented on code in PR #19572: URL: https://github.com/apache/hudi/pull/19572#discussion_r3748849439
########## website/docs/compaction.md: ########## @@ -283,6 +283,27 @@ Offline compaction needs to submit the Flink task on the command line. The progr The retry options (`--retry`, `--retry-last-failed-job`, `--job-max-processing-time-ms`) are only effective in single-run mode, not in service mode. Service mode has implicit retry semantics via its continuous monitoring loop. A warning will be logged if `--retry-last-failed-job` is enabled but `--job-max-processing-time-ms` is not set to a positive value. ::: +## Log Compaction + +Log compaction is a minor compaction for Merge-on-Read tables. Rather than merging log files into a new base file, it +stitches several small log blocks into a larger one within the same file group. A file group that receives frequent +small updates can therefore be kept efficient without paying the cost of rewriting its base file. Readers skip the log +blocks that have already been stitched, so read amplification is reduced as well. Log compaction appears on the timeline +as a `logcompaction` action. + +| Config Name | Default | Description | +|---|---|---| +| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the log compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br /><br />`Config Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction can be scheduled once the number of log blocks crosses this threshold. Effective only when log compaction is enabled via `hoodie.log.compaction.inline`.<br /><br />`Config Param: LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.enable` | `false` (Optional) | By enabling log compaction through this config, log compaction also gets enabled for the metadata table.<br /><br />`Config Param: ENABLE_LOG_COMPACTION`<br />`Since Version: 0.14.0` | Review Comment: 🤖 The description for `hoodie.log.compaction.enable` (correctly copied from the config docs) states that it enables log compaction *for the metadata table*, whereas `hoodie.log.compaction.inline` above governs log compaction on the data table. Grouping all three under one table, followed by a note that says "Log compaction is scheduled inline through the configs above," could lead a reader to treat `hoodie.log.compaction.enable` as the master on/off switch for data-table log compaction and set it expecting an effect on their MOR data — when per its own description it only applies to the metadata table. It might help to visually separate this config or add a one-line clarifier (e.g. "applies to the metadata table's log compaction, not the data table"). @yihua could you confirm the intended scope of `hoodie.log.compaction.enable` so the grouping doesn't mislead? <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## website/docs/compaction.md: ########## @@ -283,6 +283,27 @@ Offline compaction needs to submit the Flink task on the command line. The progr The retry options (`--retry`, `--retry-last-failed-job`, `--job-max-processing-time-ms`) are only effective in single-run mode, not in service mode. Service mode has implicit retry semantics via its continuous monitoring loop. A warning will be logged if `--retry-last-failed-job` is enabled but `--job-max-processing-time-ms` is not set to a positive value. ::: +## Log Compaction + +Log compaction is a minor compaction for Merge-on-Read tables. Rather than merging log files into a new base file, it +stitches several small log blocks into a larger one within the same file group. A file group that receives frequent +small updates can therefore be kept efficient without paying the cost of rewriting its base file. Readers skip the log +blocks that have already been stitched, so read amplification is reduced as well. Log compaction appears on the timeline +as a `logcompaction` action. + +| Config Name | Default | Description | +|---|---|---| +| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the log compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br /><br />`Config Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction can be scheduled once the number of log blocks crosses this threshold. Effective only when log compaction is enabled via `hoodie.log.compaction.inline`.<br /><br />`Config Param: LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.enable` | `false` (Optional) | By enabling log compaction through this config, log compaction also gets enabled for the metadata table.<br /><br />`Config Param: ENABLE_LOG_COMPACTION`<br />`Since Version: 0.14.0` | + +:::note +Log compaction is scheduled inline through the configs above. Unlike compaction, it has no dedicated SQL procedure, Review Comment: 🤖 This note states log compaction "is scheduled inline through the configs above" and has "no dedicated SQL procedure, Hudi CLI command, or standalone utility." The write_operations.md pointer, by contrast, says log compaction "may also run" more generally. Could you confirm there is truly no asynchronous log-compaction scheduling path? Hudi does support async table services, and if async log compaction can be scheduled (even indirectly), the "scheduled inline" framing would be incomplete and users running async services might miss it. If inline is genuinely the only path, it may be worth stating that explicitly so readers know async is not an option here. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
