voonhous commented on code in PR #19572:
URL: https://github.com/apache/hudi/pull/19572#discussion_r3755493025
##########
website/versioned_docs/version-1.0.1/compaction.md:
##########
@@ -227,6 +227,41 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
| `--service` | `false` (Optional) | Whether to start a monitoring service
that checks and schedules new compaction task in configured interval. |
| `--min-compaction-interval-seconds` | `600(s)` (optional) | The checking
interval for service mode, by default 10 minutes. |
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group. A file group that receives frequent
+small updates can therefore be kept efficient without paying the cost of
rewriting its base file. Readers skip the log
+blocks that have already been stitched, so read amplification is reduced as
well. Log compaction appears on the timeline
+as a `logcompaction` action.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the
log compaction service is triggered after each write. While being simpler
operationally, this adds extra latency on the write path.<br /><br />`Config
Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` |
Review Comment:
**Blocker, same as the `version-1.0.0` copy.**
`hoodie.log.compaction.inline` is unusable on 1.0.1:
`WriteOperationType.LOG_COMPACT` serializes as `"logcompact"` and `fromValue`
has no matching case, so it throws `HoodieException("Invalid value of Type.")`.
Fixed in 1.0.2 by HUDI-9220 / #13029 (`fb77b1da4114`), which renamed the value
to `log_compact` and added the case.
```console
$ git show
release-1.0.1:hudi-common/src/main/java/org/apache/hudi/common/model/WriteOperationType.java
| sed -n '104,106p'
default:
throw new HoodieException("Invalid value of Type.");
```
**Please drop this section from `version-1.0.1/compaction.md`** (and
`version-1.0.0/`), or gate it behind a caution naming HUDI-9220.
##########
website/versioned_docs/version-1.1.1/compaction.md:
##########
@@ -252,6 +252,41 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
| `--service` | `false` (Optional) | Whether to
start a monitoring service that checks and schedules new compaction task in
configured interval.
|
| `--min-compaction-interval-seconds` | `600(s)` (optional) | The checking
interval for service mode, by default 10 minutes.
|
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group. A file group that receives frequent
+small updates can therefore be kept efficient without paying the cost of
rewriting its base file. Readers skip the log
+blocks that have already been stitched, so read amplification is reduced as
well. Log compaction appears on the timeline
Review Comment:
**The read-amplification claim is not true on this version, nor on
1.0.0/1.0.1/1.0.2.** It only becomes true at 1.2.0.
The `COMPACTED_BLOCK_TIMES` skip lives only in `scanInternalV2`, and
`scanInternal` chooses the path at runtime:
```console
$ git show
release-1.1.1-rc2:hudi-common/src/main/java/org/apache/hudi/common/table/log/BaseHoodieLogRecordReader.java
| sed -n '196,201p'
if (enableOptimizedLogBlocksScan) {
scanInternalV2(keySpecOpt, skipProcessingBlocks);
} else {
scanInternalV1(keySpecOpt);
```
`hoodie.optimized.log.blocks.scan.enable` defaults to `false`. On the Spark
data-table read path it cannot be switched on at all in these releases:
`HoodieFileGroupReader` has no optimized-scan wiring in 1.0.x, and in 1.1.1 its
builder field is `private Boolean enableOptimizedLogBlockScan = false;` (line
382), which makes the props lookup at line 515 dead code. The only non-MDT
caller of the setter is `PartitionBucketIndexManager.scala:245`, a procedure,
not the query path.
The writer side states this outright,
`FileGroupReaderBasedAppendHandle.java:90`:
```java
// instead of using config.enableOptimizedLogBlocksScan(), we set to true as
log compaction blocks only supported in scanV2
```
So with stock settings on 1.0.x/1.1.1 a reader loads both the original
blocks and their stitched copy: read amplification goes **up**, not down. The
branch was removed only at 1.2.0 by #17520 ("migrate to ScanV2Internal API and
remove ENABLE_OPTIMIZED_LOG_BLOCKS_SCAN config"), which is what makes the
sentence correct for `next` and 1.2.0.
**Please drop "Readers skip the log blocks that have already been stitched,
so read amplification is reduced as well." from the 1.0.0, 1.0.1, 1.0.2 and
1.1.1 copies**, and keep it only in `website/docs/compaction.md` and
`version-1.2.0/compaction.md`. Combined with the HUDI-9220 blocker on
1.0.0/1.0.1, restricting the whole section to `next` + 1.2.0 is the simpler
call and matches how you already excluded 0.14/0.15.
##########
website/docs/compaction.md:
##########
@@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
The retry options (`--retry`, `--retry-last-failed-job`,
`--job-max-processing-time-ms`) are only effective in single-run mode, not in
service mode. Service mode has implicit retry semantics via its continuous
monitoring loop. A warning will be logged if `--retry-last-failed-job` is
enabled but `--job-max-processing-time-ms` is not set to a positive value.
:::
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group. A file group that receives frequent
+small updates can therefore be kept efficient without paying the cost of
rewriting its base file. Readers skip the log
Review Comment:
**The section describes only the upside.** Log compaction costs storage and
write amplification, and that is the thing a user most needs to know before
enabling it: the superseded blocks are not deleted.
From RFC-48, the design doc this section links to:
> Merged LogBlocks are only cleaned once a complete or major compaction is
executed on the file group to form a new base file. So, there won't be any
changes required from the cleaner service.
The stitched block is an append, not a rewrite, and the originals stay on
disk until the next full compaction plus clean. (RFC-48 Scenario 3 also notes a
single log compaction can emit more than one block when the merged output
exceeds the block size, so "into a larger one" is not always literally one
block.)
Worth noting `website/learn/tech-specs.md:669` frames the benefit as
"stitches together log files into a single large log file, thus **reducing
write amplification**", which is RFC-48's stated motivation. This section drops
write amplification entirely and keeps only the read claim, so the two pages
now sell the feature on different grounds.
**Please add a clause to this paragraph**, roughly: "The stitched block is
appended rather than replacing anything; the original blocks stay on disk until
the next full compaction and clean, so log compaction trades extra storage for
fewer blocks to merge on read."
##########
website/versioned_docs/version-1.0.0/compaction.md:
##########
@@ -226,3 +226,38 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
| `--seq` | `LIFO` (Optional) | The order in which compaction tasks are
executed. Executing from the latest compaction plan by default. `LIFO`:
executing from the latest plan. `FIFO`: executing from the oldest plan. |
| `--service` | `false` (Optional) | Whether to start a monitoring service
that checks and schedules new compaction task in configured interval. |
| `--min-compaction-interval-seconds` | `600(s)` (optional) | The checking
interval for service mode, by default 10 minutes. |
+
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group. A file group that receives frequent
+small updates can therefore be kept efficient without paying the cost of
rewriting its base file. Readers skip the log
+blocks that have already been stitched, so read amplification is reduced as
well. Log compaction appears on the timeline
+as a `logcompaction` action.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the
log compaction service is triggered after each write. While being simpler
operationally, this adds extra latency on the write path.<br /><br />`Config
Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` |
Review Comment:
**Blocker for this versioned copy.** Inline log compaction does not work on
1.0.0 or 1.0.1, so this row tells users to switch on something that fails.
`WriteOperationType.LOG_COMPACT` serializes as `"logcompact"` on these two
releases, but `fromValue` has no matching case and falls through to `default:
throw new HoodieException("Invalid value of Type.")`.
```console
$
P=hudi-common/src/main/java/org/apache/hudi/common/model/WriteOperationType.java
$ for t in release-1.0.0 release-1.0.1 release-1.0.2; do printf "%-16s " $t;
\
git show "$t:$P" | grep -o 'LOG_COMPACT("[a-z_]*")'; \
git show "$t:$P" | grep -c 'case "log_compact"'; done
release-1.0.0 LOG_COMPACT("logcompact") 0
release-1.0.1 LOG_COMPACT("logcompact") 0
release-1.0.2 LOG_COMPACT("log_compact") 1
```
This is HUDI-9220, "Cannot find write operation type if run inline log
compaction" (#13029, commit `fb77b1da4114`), fix version 1.0.2. The JIRA
reports it failing an entire Spark streaming micro-batch on a MOR table with
OCC.
**Please drop the whole `## Log Compaction` section from
`version-1.0.0/compaction.md` and `version-1.0.1/compaction.md`.** If you would
rather keep it, it needs a `:::caution` naming HUDI-9220 and stating that
`hoodie.log.compaction.inline` is unusable before 1.0.2.
##########
website/docs/compaction.md:
##########
@@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
The retry options (`--retry`, `--retry-last-failed-job`,
`--job-max-processing-time-ms`) are only effective in single-run mode, not in
service mode. Service mode has implicit retry semantics via its continuous
monitoring loop. A warning will be logged if `--retry-last-failed-job` is
enabled but `--job-max-processing-time-ms` is not set to a positive value.
:::
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group. A file group that receives frequent
+small updates can therefore be kept efficient without paying the cost of
rewriting its base file. Readers skip the log
+blocks that have already been stitched, so read amplification is reduced as
well. Log compaction appears on the timeline
+as a `logcompaction` action.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the
log compaction service is triggered after each write. While being simpler
operationally, this adds extra latency on the write path.<br /><br />`Config
Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` |
+| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction
can be scheduled once the number of log blocks crosses this threshold.
Effective only when log compaction is enabled via
`hoodie.log.compaction.inline`.<br /><br />`Config Param:
LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` |
+
+:::note
+`hoodie.log.compaction.inline` is the only built-in way to schedule log
compaction on a data table. There is no
+asynchronous log compaction service, SQL procedure, Hudi CLI command, or
standalone utility for it, unlike compaction.
+Programmatic scheduling is available through the write client's
`scheduleLogCompaction` and `logCompact` methods.
Review Comment:
Two gaps in this note, both scoped to the `next` and `version-1.2.0` copies
only.
**Async does exist for the metadata table from 1.2.0.** This same version's
`configurations.md:624-625` documents:
> `hoodie.metadata.table.service.manager.actions` -- Comma-separated list of
table service actions on the metadata table that should be delegated to the
table service manager. Currently supported actions are: compaction,
**logcompaction**.
> `hoodie.metadata.table.service.manager.enabled` -- ... This prevents the
current writer from executing compaction/logcompaction on the metadata table,
**allowing a separate async pipeline to handle them**.
Flink does it too: `CompactionUtil.scheduleMetadataCompaction` calls
`writeClient.scheduleLogCompaction`, and
`MetadataTableCompactionPlanHandler.collectCompactionOperations` picks up the
pending plan. Neither exists at 1.1.1 or earlier
(`TABLE_SERVICE_MANAGER_ACTIONS` has no match at `release-1.1.1`), so the 1.0.x
and 1.1.1 copies are fine as written.
**Flink cannot do data-table log compaction at all.** There is no
`FlinkOptions` key for it in any version -- `git grep -i
"log.compaction\|logcompact" release-1.2.0 --
'.../configuration/FlinkOptions.java'` is empty, same on master. The second
half of this page is Flink offline compaction, so a Flink MOR user lands here
with no signal that this is Spark-only.
Suggested rewrite for the `next` and 1.2.0 copies: keep the first sentence,
then
> There is no asynchronous log compaction service for the data table, and no
SQL procedure, Hudi CLI command or standalone utility, unlike compaction; nor
is it exposed through Flink options, so Flink MOR data tables cannot schedule
it. The metadata table's log compaction can be delegated to an async pipeline
via `hoodie.metadata.table.service.manager.enabled` with
`.actions=logcompaction`. Programmatic scheduling is available through the
write client's `scheduleLogCompaction` and `logCompact` methods.
##########
website/docs/compaction.md:
##########
@@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
The retry options (`--retry`, `--retry-last-failed-job`,
`--job-max-processing-time-ms`) are only effective in single-run mode, not in
service mode. Service mode has implicit retry semantics via its continuous
monitoring loop. A warning will be logged if `--retry-last-failed-job` is
enabled but `--job-max-processing-time-ms` is not set to a positive value.
:::
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group. A file group that receives frequent
+small updates can therefore be kept efficient without paying the cost of
rewriting its base file. Readers skip the log
+blocks that have already been stitched, so read amplification is reduced as
well. Log compaction appears on the timeline
+as a `logcompaction` action.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the
log compaction service is triggered after each write. While being simpler
operationally, this adds extra latency on the write path.<br /><br />`Config
Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` |
+| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction
can be scheduled once the number of log blocks crosses this threshold.
Effective only when log compaction is enabled via
`hoodie.log.compaction.inline`.<br /><br />`Config Param:
LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` |
+
+:::note
+`hoodie.log.compaction.inline` is the only built-in way to schedule log
compaction on a data table. There is no
+asynchronous log compaction service, SQL procedure, Hudi CLI command, or
standalone utility for it, unlike compaction.
+Programmatic scheduling is available through the write client's
`scheduleLogCompaction` and `logCompact` methods.
+:::
+
+The metadata table runs its own log compaction, controlled by a separate pair
of configs:
Review Comment:
Two caveats missing from the metadata-table half, both of which bite on the
versions this section is backported to.
**1. Same scanV2 gap as the data table.**
`HoodieBackedTableMetadata.java:551` passes
`metadataConfig.isOptimizedLogBlocksScanEnabled()`, and
`HoodieMetadataConfig.ENABLE_OPTIMIZED_LOG_BLOCKS_SCAN`
(`hoodie.metadata.optimized.log.blocks.scan.enable`) defaults to `false` at
1.0.0 through 1.1.1. Setting `hoodie.metadata.log.compaction.enable=true` on
those versions without also setting that produces stitched MDT blocks the MDT
reader never skips. Nothing in the code validates or warns about the pairing.
**2. Enabling MDT log compaction gates MDT major compaction.**
`CompactionUtil.canScheduleMetadataCompaction` and
`HoodieBackedTableMetadataWriter.validateCompactionScheduling` both return
false whenever any pending compaction or log-compaction instant exists, with
the comment that "metadata items such as RLI only has proc-time ordering
semantics". HUDI-7533 to lift this restriction is still open.
**Please add a sentence covering (1) to the pre-1.2.0 copies, and a short
note for (2).**
##########
website/docs/compaction.md:
##########
@@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
The retry options (`--retry`, `--retry-last-failed-job`,
`--job-max-processing-time-ms`) are only effective in single-run mode, not in
service mode. Service mode has implicit retry semantics via its continuous
monitoring loop. A warning will be logged if `--retry-last-failed-job` is
enabled but `--job-max-processing-time-ms` is not set to a positive value.
:::
+## Log Compaction
Review Comment:
Nit, feel free to ignore. This content already exists at
`website/learn/tech-specs.md:667-674` as a `### Log Compaction` section with
the same substance and the same RFC-48 link. The PR description's "net-new
content" claim holds for `compaction.md` and `write_operations.md`, but the
site now describes log compaction in five places with no cross-links and in two
different units:
```text
timeline.md:33 "merge multiple small log FILES into a bigger log FILE
in the same file slice"
hudi_stack.md:42 "log files are compacted into small set of log FILES
(log compaction)"
tech-specs.md:109 "consolidates a set of ... log FILES into another log
FILE within the same file group"
tech-specs.md:669 "stitches together log FILES into a single large log
FILE"
this section "stitches several small log BLOCKS into a larger one"
```
Both units are defensible since the scheduler triggers on either count, but
it is worth picking one. The cheap win: link `timeline.md:33`'s
`**LOGCOMPACTION**` bullet to `compaction.md#log-compaction` in the six
versions you are already touching. That bullet is the most likely entry point
and currently dead-ends.
##########
website/docs/write_operations.md:
##########
@@ -128,7 +128,7 @@ The following is an inside look on the Hudi write path and
the sequence of event
6. Update [Index](indexes.md): Now that the write is performed, we will go
back and update the index.
7. Commit: Finally we commit all of these changes atomically. ([Post-commit
callback](platform_services_post_commit_callback.md) can be configured.)
8. [Clean](cleaning.md) (if needed): Following the commit, cleaning is invoked
if needed.
-9. [Compaction](compaction.md): If you are using MOR tables, compaction will
either run inline, or be scheduled asynchronously
+9. [Compaction](compaction.md): If you are using MOR tables, compaction will
either run inline, or be scheduled asynchronously. [Log
compaction](compaction.md#log-compaction) may also run, stitching small log
blocks together without rewriting the base file.
Review Comment:
Nit: "may also run" sits next to compaction's "will either run inline, or be
scheduled asynchronously". Compaction is on by default; log compaction is not
-- `hoodie.log.compaction.inline` defaults to `false` and is the only
data-table trigger (`HoodieWriteConfig.inlineLogCompactionEnabled()`, called
only from `BaseHoodieTableServiceClient.runTableServicesInline`). As written a
reader infers it happens out of the box.
```suggestion
9. [Compaction](compaction.md): If you are using MOR tables, compaction will
either run inline, or be scheduled asynchronously. If
`hoodie.log.compaction.inline` is enabled, [log
compaction](compaction.md#log-compaction) may also run, stitching small log
blocks together without rewriting the base file.
```
##########
website/docs/compaction.md:
##########
@@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
The retry options (`--retry`, `--retry-last-failed-job`,
`--job-max-processing-time-ms`) are only effective in single-run mode, not in
service mode. Service mode has implicit retry semantics via its continuous
monitoring loop. A warning will be logged if `--retry-last-failed-job` is
enabled but `--job-max-processing-time-ms` is not set to a positive value.
:::
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group. A file group that receives frequent
+small updates can therefore be kept efficient without paying the cost of
rewriting its base file. Readers skip the log
+blocks that have already been stitched, so read amplification is reduced as
well. Log compaction appears on the timeline
+as a `logcompaction` action.
Review Comment:
**The completed instant is not a `logcompaction` action.** `logcompaction`
is only the requested and inflight state; the transition to complete writes
`DELTA_COMMIT_ACTION`, in every version this PR touches:
```console
$
P=hudi-common/src/main/java/org/apache/hudi/common/table/timeline/versioning/v2/ActiveTimelineV2.java
$ git show release-1.2.0:$P | grep -A5
"transitionLogCompactionInflightToComplete("
ValidationUtils.checkArgument(inflightInstant.getAction().equals(HoodieTimeline.LOG_COMPACTION_ACTION));
ValidationUtils.checkArgument(inflightInstant.isInflight());
HoodieInstant commitInstant =
instantGenerator.createNewInstant(HoodieInstant.State.COMPLETED,
DELTA_COMMIT_ACTION, inflightInstant.requestedTime());
```
Identical at release-1.0.0, 1.0.1, 1.0.2 and 1.1.1. RFC-48 says the same
("can issue a .deltacommit on the timeline after completion").
As written, someone who enables log compaction and then lists `.hoodie/`
looking for a completed `logcompaction` instant will not find one and will
conclude it never ran.
```suggestion
blocks that have already been stitched, so read amplification is reduced as
well. Log compaction is scheduled on the
timeline as a `logcompaction` action (`.logcompaction.requested` /
`.logcompaction.inflight`); on completion it is
committed as a `deltacommit`.
```
##########
website/docs/compaction.md:
##########
@@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
The retry options (`--retry`, `--retry-last-failed-job`,
`--job-max-processing-time-ms`) are only effective in single-run mode, not in
service mode. Service mode has implicit retry semantics via its continuous
monitoring loop. A warning will be logged if `--retry-last-failed-job` is
enabled but `--job-max-processing-time-ms` is not set to a positive value.
:::
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group. A file group that receives frequent
+small updates can therefore be kept efficient without paying the cost of
rewriting its base file. Readers skip the log
+blocks that have already been stitched, so read amplification is reduced as
well. Log compaction appears on the timeline
+as a `logcompaction` action.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the
log compaction service is triggered after each write. While being simpler
operationally, this adds extra latency on the write path.<br /><br />`Config
Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` |
+| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction
can be scheduled once the number of log blocks crosses this threshold.
Effective only when log compaction is enabled via
`hoodie.log.compaction.inline`.<br /><br />`Config Param:
LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` |
Review Comment:
Two problems in this row.
**1. It contradicts the note three lines below.** "Effective only when log
compaction is enabled via `hoodie.log.compaction.inline`" versus a note saying
programmatic scheduling through `scheduleLogCompaction` also exists. The
threshold gates *scheduling*, not the *trigger*, and none of its read sites are
guarded by `inlineLogCompactionEnabled()`:
```console
$ git grep -n "getLogCompactionBlocksThreshold()" apache/master -- '*/main/*'
.../ScheduleCompactionActionExecutor.java:251: boolean shouldLogCompact =
numDeltaCommitsSince >= config.getLogCompactionBlocksThreshold();
.../plan/generators/HoodieLogCompactionPlanGenerator.java:98: if
(numLogFiles >= writeConfig.getLogCompactionBlocksThreshold()) {
.../plan/generators/HoodieLogCompactionPlanGenerator.java:113: return
totalBlocks >= writeConfig.getLogCompactionBlocksThreshold();
```
**2. It is not only log blocks.**
`HoodieLogCompactionPlanGenerator.isFileSliceEligibleForLogCompaction` returns
true on log **file** count before it counts blocks at all, and both comparisons
are `>=`.
Both sentences are inherited from the upstream javadoc in
`HoodieCompactionConfig`, so the PR did not invent them. But the PR chose to
rewrite the description rather than copy it verbatim, so it is worth getting
right here.
```suggestion
| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction
can be scheduled once a file slice has at least this many log files, or at
least this many log blocks. Applies to any log compaction scheduling attempt,
whether triggered inline or programmatically.<br /><br />`Config Param:
LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` |
```
##########
website/docs/compaction.md:
##########
@@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
The retry options (`--retry`, `--retry-last-failed-job`,
`--job-max-processing-time-ms`) are only effective in single-run mode, not in
service mode. Service mode has implicit retry semantics via its continuous
monitoring loop. A warning will be logged if `--retry-last-failed-job` is
enabled but `--job-max-processing-time-ms` is not set to a positive value.
:::
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group. A file group that receives frequent
+small updates can therefore be kept efficient without paying the cost of
rewriting its base file. Readers skip the log
+blocks that have already been stitched, so read amplification is reduced as
well. Log compaction appears on the timeline
+as a `logcompaction` action.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the
log compaction service is triggered after each write. While being simpler
operationally, this adds extra latency on the write path.<br /><br />`Config
Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` |
+| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction
can be scheduled once the number of log blocks crosses this threshold.
Effective only when log compaction is enabled via
`hoodie.log.compaction.inline`.<br /><br />`Config Param:
LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` |
+
+:::note
+`hoodie.log.compaction.inline` is the only built-in way to schedule log
compaction on a data table. There is no
+asynchronous log compaction service, SQL procedure, Hudi CLI command, or
standalone utility for it, unlike compaction.
+Programmatic scheduling is available through the write client's
`scheduleLogCompaction` and `logCompact` methods.
+:::
+
+The metadata table runs its own log compaction, controlled by a separate pair
of configs:
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.metadata.log.compaction.enable` | `false` (Optional) | Enables log
compaction for the metadata table.<br /><br />`Config Param:
ENABLE_LOG_COMPACTION_ON_METADATA_TABLE`<br />`Since Version: 0.14.0` |
+| `hoodie.metadata.log.compaction.blocks.threshold` | `5` (Optional) | Number
of log blocks above which log compaction is scheduled on the metadata table.<br
/><br />`Config Param: LOG_COMPACT_BLOCKS_THRESHOLD`<br />`Since Version:
0.14.0` |
Review Comment:
Optional, and a pre-existing bug rather than one you introduced -- but this
PR is the natural place to fix it. `website/docs/metadata.md:151` links to:
```text
[compaction](compaction.md#delegating-mdt-compaction-to-an-external-platform)
```
That heading exists in no version of `compaction.md`. `docusaurus.config.js`
sets `onBrokenLinks: "throw"` and `onBrokenMarkdownLinks: "warn"` but leaves
`onBrokenAnchors` unset, so it defaults to `warn` and degrades silently --
which is why your byte-identical-warning-set check did not surface it.
Since you are already writing MDT log-compaction prose here, promoting this
block to `### Metadata Table Log Compaction` and adding a `### Delegating MDT
Compaction to an External Platform` subsection (in the `next` and 1.2.0 copies)
covering `hoodie.metadata.table.service.manager.enabled` / `.actions` would
close the dangling anchor and the async gap in one edit, and give the MDT block
its own TOC entry and deep link. Entirely your call whether to take that on
here.
##########
website/docs/compaction.md:
##########
@@ -283,6 +283,41 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
The retry options (`--retry`, `--retry-last-failed-job`,
`--job-max-processing-time-ms`) are only effective in single-run mode, not in
service mode. Service mode has implicit retry semantics via its continuous
monitoring loop. A warning will be logged if `--retry-last-failed-job` is
enabled but `--job-max-processing-time-ms` is not set to a positive value.
:::
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group. A file group that receives frequent
+small updates can therefore be kept efficient without paying the cost of
rewriting its base file. Readers skip the log
+blocks that have already been stitched, so read amplification is reduced as
well. Log compaction appears on the timeline
+as a `logcompaction` action.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the
log compaction service is triggered after each write. While being simpler
operationally, this adds extra latency on the write path.<br /><br />`Config
Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` |
+| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction
can be scheduled once the number of log blocks crosses this threshold.
Effective only when log compaction is enabled via
`hoodie.log.compaction.inline`.<br /><br />`Config Param:
LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` |
+
+:::note
+`hoodie.log.compaction.inline` is the only built-in way to schedule log
compaction on a data table. There is no
+asynchronous log compaction service, SQL procedure, Hudi CLI command, or
standalone utility for it, unlike compaction.
+Programmatic scheduling is available through the write client's
`scheduleLogCompaction` and `logCompact` methods.
+:::
+
+The metadata table runs its own log compaction, controlled by a separate pair
of configs:
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.metadata.log.compaction.enable` | `false` (Optional) | Enables log
compaction for the metadata table.<br /><br />`Config Param:
ENABLE_LOG_COMPACTION_ON_METADATA_TABLE`<br />`Since Version: 0.14.0` |
+| `hoodie.metadata.log.compaction.blocks.threshold` | `5` (Optional) | Number
of log blocks above which log compaction is scheduled on the metadata table.<br
/><br />`Config Param: LOG_COMPACT_BLOCKS_THRESHOLD`<br />`Since Version:
0.14.0` |
+
+:::caution
+`hoodie.log.compaction.enable` also appears in the configuration reference,
but it is not a switch to set on your table.
+Hudi applies it internally to the metadata table's own write config, deriving
its value from
+`hoodie.metadata.log.compaction.enable`. Setting it on a data table has no
effect: use
+`hoodie.log.compaction.inline` for the data table, and
`hoodie.metadata.log.compaction.enable` for the metadata table.
Review Comment:
Nit on the PR description rather than this hunk, anchored here because this
is the part the description no longer covers. The body still says "the
**three** configs" and that the `hoodie.log.compaction.enable` row "stays as
narrow as" the generated doc "rather than inventing a broader meaning". After
`d4016ac` the diff documents five configs, and this `:::caution` flatly
contradicts the generated description ("it is not a switch to set on your table
... Setting it on a data table has no effect").
The caution itself is correct, and removing the row rather than annotating
it was the right call. I verified `ENABLE_LOG_COMPACTION` is read only via
`metadataWriteConfig.isLogCompactionEnabled()`, and that
`HoodieMetadataWriteUtils.createMetadataWriteConfig` unconditionally overwrites
it from `hoodie.metadata.log.compaction.enable`, at release-1.0.0 (`:147`),
1.0.1 (`:149`), 1.0.2 (`:150`), 1.1.1 (`:219`), 1.2.0 (`:250`) and master
(`:307`).
**Please refresh the description to match the five-config diff, and re-state
the `npm run build` / warning-parity verification against `d4016ac`** -- as
written it reads as having been run against the earlier revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
