voonhous commented on code in PR #19572:
URL: https://github.com/apache/hudi/pull/19572#discussion_r3757140205
##########
website/docs/compaction.md:
##########
@@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
The retry options (`--retry`, `--retry-last-failed-job`,
`--job-max-processing-time-ms`) are only effective in single-run mode, not in
service mode. Service mode has implicit retry semantics via its continuous
monitoring loop. A warning will be logged if `--retry-last-failed-job` is
enabled but `--job-max-processing-time-ms` is not set to a positive value.
:::
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group. A file group that receives frequent
+small updates can therefore be kept efficient without paying the cost of
rewriting its base file. Readers skip the log
+blocks that have already been stitched, so read amplification is reduced as
well. Log compaction appears on the timeline
+as a `logcompaction` action.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the
log compaction service is triggered after each write. While being simpler
operationally, this adds extra latency on the write path.<br /><br />`Config
Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` |
+| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction
can be scheduled once the number of log blocks crosses this threshold.
Effective only when log compaction is enabled via
`hoodie.log.compaction.inline`.<br /><br />`Config Param:
LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` |
+
+:::note
+`hoodie.log.compaction.inline` is the only built-in way to schedule log
compaction on a data table. There is no
+asynchronous log compaction service, SQL procedure, Hudi CLI command, or
standalone utility for it, unlike compaction.
+Programmatic scheduling is available through the write client's
`scheduleLogCompaction` and `logCompact` methods.
Review Comment:
The async path does exist and the config keys are right, but the sentence as
written is a footgun: following it without a table service manager deployed
will stall metadata table compaction.
**Delegation stops execution; nothing in Hudi picks it up.** At
`release-1.2.0`, `HoodieBackedTableMetadataWriter:2235-2244` still *schedules*
the MDT log compaction, then skips running it:
```java
Option<String> scheduledLogCompaction =
writeClient.scheduleLogCompaction(Option.empty());
if (scheduledLogCompaction.isPresent()) {
LOG.info("Log compaction is scheduled for timestamp {}",
scheduledLogCompaction.get());
if (shouldDelegateToTableServiceManager(metadataWriteConfig,
ActionType.logcompaction)) {
LOG.info("Skipping execution of log compaction on MDT as it is delegated
to table service manager.");
} else {
writeClient.logCompact(scheduledLogCompaction.get(), true);
}
}
```
`runPendingTableServicesOperationsAndRefreshTimeline` at `:2170-2176` skips
`runAnyPendingLogCompactions()` on the same condition, so already-pending
instants are not picked up on a later commit either.
**The executor is an external service that Hudi does not ship.**
`hoodie.table.service.manager.uris` defaults to `http://localhost:9091`, and
the only thing in the tree is `HoodieTableServiceManagerClient` -- there is no
server at `release-1.2.0` or on `apache/master`.
`HoodieTableServiceManagerConfig`'s own javadoc says so:
```java
/**
* Configurations used by the Hudi Table Service Manager.
*
* TODO: enable docs gen by adding {@link ConfigClassProperty} after TSM is
landed (HUDI-3475)
*/
```
That TODO is also why these keys are absent from the generated TSM section
of `configurations.md`.
**And it interlocks with the note you added two paragraphs down.**
`validateCompactionScheduling` (`HoodieBackedTableMetadataWriter:2281-2296`)
returns false while any pending log compaction *or* compaction instant exists.
So the combined outcome of following this sentence without a TSM is:
`logcompaction.requested` instants accumulate on the MDT timeline, nothing
executes them, and MDT major compaction stops being scheduled too.
Not asking you to drop the sentence -- the information is worth having.
Please make the prerequisite explicit, something like:
> The metadata table runs its own log compaction, controlled by a separate
pair of configs. Unlike the data table, its execution can be delegated to an
external table service manager by setting
`hoodie.metadata.table.service.manager.enabled=true` together with
`hoodie.metadata.table.service.manager.actions=logcompaction`. Note that this
only stops the writer from executing log compaction inline: a table service
manager must be deployed and reachable at `hoodie.table.service.manager.uris`,
otherwise pending `logcompaction` instants accumulate and, per the note below,
metadata table compaction stops being scheduled as well.
One thing that would strengthen it if you want it: on Flink at 1.2.0 the
MDT's log compaction is executed by the existing compaction pipeline
(`MetadataTableCompactionPlanHandler` /
`CompactionUtil.scheduleMetadataCompaction`), with no TSM involved. So "async
MDT log compaction" means two different things depending on the engine, and
only the TSM route carries this prerequisite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
