voonhous commented on code in PR #19572: URL: https://github.com/apache/hudi/pull/19572#discussion_r3757486831
########## website/docs/compaction.md: ########## @@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the command line. The progr The retry options (`--retry`, `--retry-last-failed-job`, `--job-max-processing-time-ms`) are only effective in single-run mode, not in service mode. Service mode has implicit retry semantics via its continuous monitoring loop. A warning will be logged if `--retry-last-failed-job` is enabled but `--job-max-processing-time-ms` is not set to a positive value. ::: +## Log Compaction + +Log compaction is a minor compaction for Merge-on-Read tables. Rather than merging log files into a new base file, it +stitches several small log blocks into a larger one within the same file group. A file group that receives frequent +small updates can therefore be kept efficient without paying the cost of rewriting its base file. Readers skip the log +blocks that have already been stitched, so read amplification is reduced as well. Log compaction appears on the timeline +as a `logcompaction` action. + +| Config Name | Default | Description | +|---|---|---| +| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the log compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br /><br />`Config Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction can be scheduled once the number of log blocks crosses this threshold. Effective only when log compaction is enabled via `hoodie.log.compaction.inline`.<br /><br />`Config Param: LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` | + +:::note +`hoodie.log.compaction.inline` is the only built-in way to schedule log compaction on a data table. There is no +asynchronous log compaction service, SQL procedure, Hudi CLI command, or standalone utility for it, unlike compaction. +Programmatic scheduling is available through the write client's `scheduleLogCompaction` and `logCompact` methods. Review Comment: Pushed the correction to your branch directly as `3fe708da1547` rather than round-tripping it -- hope that is alright, and revert or reword freely if you would rather own the phrasing. What changed, identically in both copies: - The paragraph no longer offers delegation as an option. It now says the writer that maintains the metadata table executes it, and keeps your Flink sentence, which is accurate. - The caution is inverted. Instead of "deploy a table service manager and make it reachable at `hoodie.table.service.manager.uris`", it now says `hoodie.metadata.table.service.manager.actions` lists `logcompaction` but cannot be used to move log compaction off the writer, and states the stall consequence. The URI pointer had to go: `delegateToTableServiceManager` is the only site that constructs a `HoodieTableServiceManagerClient`, so it is the only place that config is ever read, and it is reachable only in a form that cannot dispatch a log compaction. Sending readers there would have had them debugging an endpoint the code never contacts. No new links, headings or anchors, so no build or anchor impact; prose wraps at 120 to match the file. I did not re-run `npm run build` for a prose-only change inside existing admonitions -- worth a rebuild on your side before merge if you want belt and braces. Resolving this thread. The only remaining open one is the pre-existing `metadata.md` dangling anchor, which we agreed to leave for a follow-up and is not a blocker. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
