deepakpanda93 commented on code in PR #19572:
URL: https://github.com/apache/hudi/pull/19572#discussion_r3757349206
##########
website/docs/compaction.md:
##########
@@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
The retry options (`--retry`, `--retry-last-failed-job`,
`--job-max-processing-time-ms`) are only effective in single-run mode, not in
service mode. Service mode has implicit retry semantics via its continuous
monitoring loop. A warning will be logged if `--retry-last-failed-job` is
enabled but `--job-max-processing-time-ms` is not set to a positive value.
:::
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group. A file group that receives frequent
+small updates can therefore be kept efficient without paying the cost of
rewriting its base file. Readers skip the log
+blocks that have already been stitched, so read amplification is reduced as
well. Log compaction appears on the timeline
+as a `logcompaction` action.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the
log compaction service is triggered after each write. While being simpler
operationally, this adds extra latency on the write path.<br /><br />`Config
Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` |
+| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction
can be scheduled once the number of log blocks crosses this threshold.
Effective only when log compaction is enabled via
`hoodie.log.compaction.inline`.<br /><br />`Config Param:
LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` |
+
+:::note
+`hoodie.log.compaction.inline` is the only built-in way to schedule log
compaction on a data table. There is no
+asynchronous log compaction service, SQL procedure, Hudi CLI command, or
standalone utility for it, unlike compaction.
+Programmatic scheduling is available through the write client's
`scheduleLogCompaction` and `logCompact` methods.
Review Comment:
You're right, and it's worse than an omission — the sentence as written
would have led someone into a stall. Fixed in a32981e52be9.
I verified each part against `release-1.2.0` rather than taking the report
as read, and all of it holds:
**Delegation stops execution.** `HoodieBackedTableMetadataWriter` schedules
the log compaction and then, when delegated, logs `Skipping execution of log
compaction on MDT as it is delegated to table service manager` in place of
`writeClient.logCompact(...)`. The pending-services path does the same —
`Skipping pending log compactions on MDT as they are delegated to table service
manager` guards `runAnyPendingLogCompactions()` — so nothing catches up on a
later commit either.
**Nothing ships to execute them.** Searching the tree for
`TableServiceManager` at `release-1.2.0` returns exactly two files:
`HoodieTableServiceManagerClient` and `HoodieTableServiceManagerConfig`. No
server. And the config class still says so itself:
```java
* TODO: enable docs gen by adding {@link ConfigClassProperty} after TSM is
landed (HUDI-3475)
```
with `hoodie.table.service.manager.uris` defaulting to
`http://localhost:9091`.
So the combined outcome you describe — instants accumulating, and
`validateCompactionScheduling` then blocking MDT major compaction too — follows
directly. That interlock is the part that makes it a footgun rather than a
no-op: the failure is silent and it takes out a *different* table service than
the one the user was configuring.
The paragraph now reads:
> The metadata table runs its own log compaction, controlled by a separate
pair of configs. Its execution can be delegated to an external table service
manager by setting `hoodie.metadata.table.service.manager.enabled=true`
together with `hoodie.metadata.table.service.manager.actions=logcompaction`.
Flink does not need this: its compaction pipeline schedules and executes
metadata table log compaction directly.
with a caution after the config table:
> Delegating only stops the writer from running log compaction inline. Hudi
does not ship the table service manager itself, so one must be deployed and
reachable at `hoodie.table.service.manager.uris` (default
`http://localhost:9091`); otherwise pending `logcompaction` instants accumulate
on the metadata table and, per the note below, metadata table compaction stops
being scheduled as well.
I also dropped "delegated to an async pipeline" for "delegated to an
external table service manager" — the original phrasing implied Hudi runs the
pipeline for you, which is precisely the wrong impression.
**On the Flink point**, I took it but checked it first, and the call chain
is slightly different from your summary in a way that does not change the
conclusion. `CompactionUtil.scheduleMetadataCompaction` tries
`writeClient.scheduleCompaction(...)` first and returns if that schedules
something; only if it does not, and `isLogCompactionEnabled()` is true, does it
fall through to `writeClient.scheduleLogCompaction(...)`. So MDT log compaction
is scheduled by the Flink pipeline as a fallback after compaction, not
unconditionally — but either way no TSM is involved, so "Flink does not need
this" is accurate and the prerequisite genuinely is Spark-side only.
Two things I did not add, to keep the caution readable: that the delegation
check is `shouldDelegateToTableServiceManager(config,
ActionType.logcompaction)` specifically, and that
`hoodie.table.service.manager.uris` is absent from the generated TSM section of
`configurations.md` for the HUDI-3475 reason. Say the word if either belongs in
the text.
Build clean, warning set byte-identical to baseline, both copies carry
identical hunks.
##########
website/docs/compaction.md:
##########
@@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
The retry options (`--retry`, `--retry-last-failed-job`,
`--job-max-processing-time-ms`) are only effective in single-run mode, not in
service mode. Service mode has implicit retry semantics via its continuous
monitoring loop. A warning will be logged if `--retry-last-failed-job` is
enabled but `--job-max-processing-time-ms` is not set to a positive value.
:::
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group. A file group that receives frequent
+small updates can therefore be kept efficient without paying the cost of
rewriting its base file. Readers skip the log
+blocks that have already been stitched, so read amplification is reduced as
well. Log compaction appears on the timeline
+as a `logcompaction` action.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the
log compaction service is triggered after each write. While being simpler
operationally, this adds extra latency on the write path.<br /><br />`Config
Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` |
+| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction
can be scheduled once the number of log blocks crosses this threshold.
Effective only when log compaction is enabled via
`hoodie.log.compaction.inline`.<br /><br />`Config Param:
LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` |
+
+:::note
+`hoodie.log.compaction.inline` is the only built-in way to schedule log
compaction on a data table. There is no
+asynchronous log compaction service, SQL procedure, Hudi CLI command, or
standalone utility for it, unlike compaction.
+Programmatic scheduling is available through the write client's
`scheduleLogCompaction` and `logCompact` methods.
+:::
+
+The metadata table runs its own log compaction, controlled by a separate pair
of configs:
Review Comment:
Thanks for closing that one out with the actual check — that is the answer I
wanted and could not justify writing myself.
Useful to have it confirmed that #17520 removed the branch for the MDT
reader as well as the data-table reader, and that the
`HoodieBackedTableMetadata:551` call site is gone at 1.2.0 with only a
vestigial builder field left in `HoodieMergedLogRecordReader`. So
`hoodie.metadata.optimized.log.blocks.scan.enable` cannot gate the skip on
either path there, and the pairing caveat has nothing left to attach to now
that the pre-1.2.0 copies are reverted.
Nothing to change here. The HUDI-7533 sentence stays as written, and it now
sits directly below the new table-service-manager caution from the other
thread, which reads well since the two restrictions compound.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
