deepakpanda93 commented on code in PR #19572: URL: https://github.com/apache/hudi/pull/19572#discussion_r3749114219
########## website/docs/compaction.md: ########## @@ -283,6 +283,27 @@ Offline compaction needs to submit the Flink task on the command line. The progr The retry options (`--retry`, `--retry-last-failed-job`, `--job-max-processing-time-ms`) are only effective in single-run mode, not in service mode. Service mode has implicit retry semantics via its continuous monitoring loop. A warning will be logged if `--retry-last-failed-job` is enabled but `--job-max-processing-time-ms` is not set to a positive value. ::: +## Log Compaction + +Log compaction is a minor compaction for Merge-on-Read tables. Rather than merging log files into a new base file, it +stitches several small log blocks into a larger one within the same file group. A file group that receives frequent +small updates can therefore be kept efficient without paying the cost of rewriting its base file. Readers skip the log +blocks that have already been stitched, so read amplification is reduced as well. Log compaction appears on the timeline +as a `logcompaction` action. + +| Config Name | Default | Description | +|---|---|---| +| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the log compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br /><br />`Config Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction can be scheduled once the number of log blocks crosses this threshold. Effective only when log compaction is enabled via `hoodie.log.compaction.inline`.<br /><br />`Config Param: LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.enable` | `false` (Optional) | By enabling log compaction through this config, log compaction also gets enabled for the metadata table.<br /><br />`Config Param: ENABLE_LOG_COMPACTION`<br />`Since Version: 0.14.0` | Review Comment: You were right to push on this, and the answer is sharper than either of us had it. Fixed in d4016ac05ffb — I removed the config rather than annotate it. Traced through `release-1.2.0`: 1. `HoodieWriteConfig.isLogCompactionEnabled()` reads `HoodieCompactionConfig.ENABLE_LOG_COMPACTION` (`hoodie.log.compaction.enable`). 2. Its **only** caller is `HoodieBackedTableMetadataWriter`, and it reads it off **`metadataWriteConfig`**, not the data table's write config. No data-table code path reads it at all — the inline data-table trigger in `BaseHoodieTableServiceClient` uses `config.inlineLogCompactionEnabled()` (`hoodie.log.compaction.inline`). 3. That `metadataWriteConfig` value is not the user's either. `HoodieMetadataWriteUtils.createMetadataWriteConfig` sets it explicitly: ```java .withLogCompactionEnabled(writeConfig.isLogCompactionEnabledOnMetadata()) .withLogCompactionBlocksThreshold(writeConfig.getMetadataLogCompactBlocksThreshold()) ``` where `isLogCompactionEnabledOnMetadata()` reads `hoodie.metadata.log.compaction.enable`. So **setting `hoodie.log.compaction.enable` on a table does nothing**, on the data table or the metadata table — it is internal plumbing that Hudi overwrites from the `hoodie.metadata.*` key. Your concern that a reader might set it "expecting an effect on their MOR data" was well founded; the underlying doc string is accurate but easy to misread, which is exactly how it ended up in my table. The section now lists only `hoodie.log.compaction.inline` and `hoodie.log.compaction.blocks.threshold` for the data table, adds the two real user-facing knobs for the metadata table: | Config | Default | Since | |---|---|---| | `hoodie.metadata.log.compaction.enable` | `false` | 0.14.0 | | `hoodie.metadata.log.compaction.blocks.threshold` | `5` | 0.14.0 | and carries a caution so nobody who finds `hoodie.log.compaction.enable` in the configuration reference is left guessing: > `hoodie.log.compaction.enable` also appears in the configuration reference, but it is not a switch to set on your table. Hudi applies it internally to the metadata table's own write config, deriving its value from `hoodie.metadata.log.compaction.enable`. Setting it on a data table has no effect: use `hoodie.log.compaction.inline` for the data table, and `hoodie.metadata.log.compaction.enable` for the metadata table. @yihua a confirmation would still be welcome, since this reads like a config that was never meant to be user-settable. ########## website/docs/compaction.md: ########## @@ -283,6 +283,27 @@ Offline compaction needs to submit the Flink task on the command line. The progr The retry options (`--retry`, `--retry-last-failed-job`, `--job-max-processing-time-ms`) are only effective in single-run mode, not in service mode. Service mode has implicit retry semantics via its continuous monitoring loop. A warning will be logged if `--retry-last-failed-job` is enabled but `--job-max-processing-time-ms` is not set to a positive value. ::: +## Log Compaction + +Log compaction is a minor compaction for Merge-on-Read tables. Rather than merging log files into a new base file, it +stitches several small log blocks into a larger one within the same file group. A file group that receives frequent +small updates can therefore be kept efficient without paying the cost of rewriting its base file. Readers skip the log +blocks that have already been stitched, so read amplification is reduced as well. Log compaction appears on the timeline +as a `logcompaction` action. + +| Config Name | Default | Description | +|---|---|---| +| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the log compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br /><br />`Config Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction can be scheduled once the number of log blocks crosses this threshold. Effective only when log compaction is enabled via `hoodie.log.compaction.inline`.<br /><br />`Config Param: LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` | +| `hoodie.log.compaction.enable` | `false` (Optional) | By enabling log compaction through this config, log compaction also gets enabled for the metadata table.<br /><br />`Config Param: ENABLE_LOG_COMPACTION`<br />`Since Version: 0.14.0` | + +:::note +Log compaction is scheduled inline through the configs above. Unlike compaction, it has no dedicated SQL procedure, Review Comment: Good catch — "scheduled inline" was incomplete, and the two pages were inconsistent as you noticed. Reworded in d4016ac05ffb. Checked both halves against `release-1.2.0`: **No async service exists.** `AsyncCompactService`, `SparkAsyncCompactService` and `SparkStreamingAsyncCompactService` all drive `compactor.compact(instantTime)` — regular compaction only. There is no `AsyncLogCompactService` anywhere in the tree. (RFC-48 even ships an `async_logcompaction_issues.jpeg`, which suggests async was considered and hit problems.) Also re-confirmed there is no SQL procedure, Hudi CLI command, or standalone utility, unlike compaction which has all three. **But programmatic scheduling does exist**, so "inline only" was too strong: `BaseHoodieTableServiceClient` exposes `scheduleLogCompaction(...)` and `logCompact(instantTime, shouldComplete)` publicly, plus `runAnyPendingLogCompactions(...)`. A caller driving the write client directly can schedule and execute it outside the inline path. The note now reads: > `hoodie.log.compaction.inline` is the only built-in way to schedule log compaction on a data table. There is no asynchronous log compaction service, SQL procedure, Hudi CLI command, or standalone utility for it, unlike compaction. Programmatic scheduling is available through the write client's `scheduleLogCompaction` and `logCompact` methods. That states explicitly that async is not an option, which is what you asked for, without claiming inline is the only mechanism in existence. The `write_operations.md` pointer keeps its looser "may also run" phrasing deliberately — it is a one-line signpost in the write-path list, and the detail now lives one click away in the section it links to. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
