This is an automated email from the ASF dual-hosted git repository.
nsivabalan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 2f835a59e625 feat(agent): add Hudi optimizer catalogs and Agent
(#20197)
2f835a59e625 is described below
commit 2f835a59e625d042ff35efda4ed7ec13d835b0d0
Author: Sivabalan Narayanan <[email protected]>
AuthorDate: Sun Oct 25 22:16:50 2026 -0700
feat(agent): add Hudi optimizer catalogs and Agent (#20197)
A recommendation Agent under hudi-ai-operator/skills/hudi-optimizer for a
deployment that works
but costs more than it should. It takes writer properties, a Spark event
log and read access to
the table, runs the health checker, layout analyzer and event log
analyzer itself, and reports
an observation only when at least two inputs agree, stating confidence
from how many did.
observation-catalog.md: thirty "works, but could be better" rows built on
derived expectations,
configured intent against measured reality, each naming the JSON fields
that corroborate it,
the config key and direction, a risk tier and a verification step.
capability-catalog.md: seven
"adopt this feature" rows with signals for and against, sizing and
adoption cost. Durable
changes such as index type, table type or key generator are refused and
handed to a design
decision rather than recommended as tuning. Tools run through
spark-submit; exit codes and the
analyzers' attribution caveats, including the simple index having no
sub-phase, are documented.
Part of #19263.
---
hudi-ai-operator/skills/hudi-optimizer/README.md | 212 +++++
hudi-ai-operator/skills/hudi-optimizer/SKILL.md | 339 ++++++++
.../references/capability-catalog.md | 576 +++++++++++++
.../references/observation-catalog.md | 910 +++++++++++++++++++++
4 files changed, 2037 insertions(+)
diff --git a/hudi-ai-operator/skills/hudi-optimizer/README.md
b/hudi-ai-operator/skills/hudi-optimizer/README.md
new file mode 100644
index 000000000000..647bb2a15dab
--- /dev/null
+++ b/hudi-ai-operator/skills/hudi-optimizer/README.md
@@ -0,0 +1,212 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# hudi-optimizer
+
+Finds what a working Apache Hudi deployment is leaving on the table. Bring
your writer properties, a
+Spark event log, and a table path; get back what the run cost, what it did not
need to cost, and what
+to change — with the risk of each change stated.
+
+## What it does
+
+1. **Collects three inputs** — writer properties, a Spark event log, and the
table — and says which
+ it has, because the answer's trustworthiness depends on it.
+2. **Corroborates before it recommends.** An observation needs **at least two
of the three inputs to
+ agree**. One signal is a hypothesis and gets reported as one.
+3. **Runs the read-only tools itself** rather than asking you to run three
commands and paste three
+ outputs.
+4. **Matches against two catalogs** — thirty observations ("this works, but
could be better") and
+ seven capability rows ("adopt this feature", each able to say *don't*).
+5. **Refuses durable changes.** Anything that alters what is on storage —
table type, index type,
+ partitioning — is reported with its measurements and handed to
`hudi-architect` for an ADR, never
+ recommended as a tuning step.
+
+It recommends. It does not run table services, write to tables, or edit
configuration files.
+
+## What it is not
+
+Not a monitor. You run it, you get a report. There is no daemon, no baseline
store, no alerting.
+
+Not a Spark profiler either. Generic stage analysis can tell you that stage 14
was slow with 16,000
+tasks; it cannot tell you stage 14 was bloom-index probing inside an upsert,
which is why generic
+advice stays generic. The value here is the join from Spark stages onto
**Hudi's own phases**, and
+then onto a catalog that knows what each phase costs and what moves it.
+
+Not a failure diagnoser. If the job is failing, that is `hudi-rca`. If you are
designing a table, that
+is `hudi-architect`.
+
+## Install
+
+Copy this directory into your Claude Code skills location:
+
+```bash
+# From the root of a Hudi checkout.
+SKILL=hudi-ai-operator/skills/hudi-optimizer
+
+# User-level (available in every project)
+mkdir -p ~/.claude/skills && cp -r "$SKILL" ~/.claude/skills/
+
+# Or project-level, in the repo where your pipelines live
+mkdir -p .claude/skills && cp -r "$SKILL" .claude/skills/
+```
+
+Then in Claude Code:
+
+```
+/hudi-optimizer
+```
+
+Point it at a table and hand it what you have.
+
+## What evidence to bring, and how to capture it
+
+| Input | How to get it | Why it matters |
+|---|---|---|
+| **Writer properties** | The `--props` file or `--hoodie-conf` values your
writer actually runs with | **The one most often skipped and least skippable.**
237 of 939 Hudi configs are performance levers and 93 of those are
conditionally applied — advice given without the config may be silently inert |
+| **Spark event log** | Set `spark.eventLog.enabled=true` and
`spark.eventLog.dir` on the job, then collect the file | Where the time
actually went, resolved to Hudi phases |
+| **Table path** | A path the skill can read: local, `s3://`, `gs://`,
`abfs://` | File sizes, partition skew, service cadence, timeline state |
+
+### Capturing the event log
+
+```
+spark-submit \
+ --conf spark.eventLog.enabled=true \
+ --conf spark.eventLog.dir=file:///tmp/spark-events \
+ ... your Hudi job ...
+```
+
+The file lands in that directory named after the application ID. It may be
compressed (`.gz`, `.lz4`,
+`.snappy`, `.zstd`) or a directory of rolling parts; all are handled.
+
+**Ask for the event log, not a link to the Spark UI.** The UI is ephemeral and
cannot be attached. The
+event log is a file containing everything the UI shows, and it is what makes
this skill possible.
+
+### Two things worth doing before you capture
+
+- **Run with `hoodie.metadata.streaming.write.enabled=false` for one
diagnostic run** if you can.
+ That config defaults to `true` on Spark, and on that default **a large share
of the write DAG is
+ unattributable** — on a measured run, 60.5% of stage wall clock landed in
`UNKNOWN`. Turning it off
+ for one run gives a fully attributed breakdown; turning it back on costs
nothing. See the limits
+ below.
+- **Redact before pasting.** Properties files carry bucket names, hostnames
and occasionally
+ credentials.
+
+### If you can only bring one thing
+
+Bring the writer properties. With properties plus the table path the skill can
run two of its three
+tools and reach genuinely corroborated observations. With an event log alone
it can describe where
+time went and very little else.
+
+## Scope and limits
+
+**Covers:** file sizing and small-file handling, parallelism and skew, spill,
index cost and index
+economics, table-service cadence (compaction, cleaning, archival, clustering),
timeline growth,
+metadata-table cost and which index partitions earn it, marker overhead,
driver-side file-system-view
+sizing, and the capability question — whether the record-level index, bucket
index, clustering,
+column stats, secondary index, the metadata table or Merge-on-Read would suit
*this* table.
+
+**Does not cover:** live cluster access, continuous monitoring, applying
anything, or query-plan
+analysis. Several rows depend on query evidence the skill cannot obtain from
write-side telemetry;
+those rows say so and ask rather than guessing.
+
+### The attribution limit, stated plainly
+
+A phase share names **where work was forced, not where it logically belongs.**
Hudi sets the Spark job
+description before the action and Spark evaluates lazily, so a bucket
routinely carries the cost of an
+untagged step that ran before it. The analyzer reports this per run in a
`caveats[]` array and the
+skill repeats it. Three cases are known and verified in source: source read
bundles transformation and
+record creation; deduplication bundles with index tagging; and the write
bucket can absorb work not
+forced until the commit action ran.
+
+**And the bigger one:** with `hoodie.metadata.streaming.write.enabled=true`,
the Spark default, those
+stages are not merely fused — they are **unlabelled**, because Hudi never sets
a job description on
+that path. The skill says so rather than dividing up the remainder as though
it were the whole
+picture. This is an accepted v1 limitation; the tooling is accurate for
non-streaming writes and
+0.x-style DAGs and degrades on the 1.x streaming default.
+
+### Four rows whose answer may be "do nothing"
+
+OBS-019 (the cleaner correctly finding nothing to clean), OBS-002 (small-file
handling that is live
+and correctly declining to act), OBS-005 (a benign commit guard firing), and
every capability row
+whose counter-signals fire. They are in the catalog because the natural
reaction — change the config —
+turns a non-problem into a problem.
+
+### Durable changes are refused by construction
+
+Three rows point at changes to what is on storage: index type (CAP-001),
bucket index and its count
+(CAP-002), and table type (CAP-006). The skill reports the measurements, the
sizing estimate, and the
+counter-signals, then **hands off to `hudi-architect`**. It does not emit a
config block for them, and
+it says why out loud rather than quietly skipping them. An index-type change
on a live 10 TB table is
+a design decision with a migration behind it, not an optimization.
+
+## Engine coverage
+
+The recipes here come from Spark deployments, because that is where the
evidence came from — the
+event-log analyzer reads Spark event logs, and the measurements behind these
rows were taken on Spark
+runs.
+
+The **core patterns apply to Flink too**. File sizing, service cadence,
timeline growth,
+metadata-table cost, index economics and the capability tradeoffs are
table-layer mechanisms rather
+than engine ones. What does not carry is the event-log half: Flink produces no
Spark event log, so a
+Flink user arrives with at most two inputs and the phase breakdown is simply
unavailable. Several
+engine-effective defaults differ too — `hoodie.index.type` is `INMEMORY` on
Flink against `SIMPLE` on
+Spark, and column stats, secondary index and streaming metadata writes all
default the other way.
+
+Flink-specific coverage will deepen over time. If you are on Flink: the
table-layer reasoning holds,
+the stage-level reasoning does not, and the skill flags which is which rather
than pretending the
+distinction does not exist.
+
+## Files
+
+- `SKILL.md` — the skill definition: input collection, confidence scaling, the
corroboration rule,
+ the caveats to carry, the durable refusal, guardrails, output shape.
+- `references/observation-catalog.md` — thirty rows, "this works, but could be
better". Each with
+ measurable signals as JSON field paths, corroborating inputs, the config key
and direction, a risk
+ tier, expected payoff, and how to verify. Sixteen are the healthy-side
reading of a known failure
+ pattern; fourteen have no failure analogue at all.
+- `references/capability-catalog.md` — seven rows, "adopt this feature", each
with signals **for and
+ against**, adoption cost, risk tier and verification. The record-level index
row is the one to read
+ first.
+
+## Companion tooling
+
+All three ship in `hudi-utilities` and are strictly read-only. The skill
invokes them rather than
+asking you to.
+
+- `HoodieSparkEventLogAnalyzer` — Spark event log summarised by **Hudi
operation and phase**, with
+ sub-phases, skew, spill, GC and a per-run caveats list. The join from Spark
stages onto Hudi's own
+ vocabulary is what makes the advice Hudi-specific.
+- `HoodieTableHealthChecker` — compaction, cleaner, savepoint and archival
cadence against the
+ configured triggers, with JSON output.
+- `HoodieTableLayoutAnalyzer` — base-file size distribution, per-partition
statistics, partition skew,
+ and the micro-partition / small-files / hot-partitions detectors.
+
+The skill accepts `--health-json` and `--layout-json` to skip invocation when
you already have the
+output or cannot run `spark-submit`.
+
+**It reads the measurements, not the rollup verdicts.** The layout analyzer's
`overallStatus` fires on
+essentially every actively-written streaming table, and the health checker's
thresholds carry
+deliberate slack because those tools exist to catch a service that has
*stopped*. An optimization
+observation lives in the band where both correctly report healthy, so the
catalogs are written against
+the underlying `observed.*` fields.
+
+## Feedback
+
+The catalogs are only as good as the deployments they have seen. If the skill
recommends something
+that did nothing, misses something that mattered, or names a config that
turned out to be inert on
+your setup, that is the useful report: what it said, what you changed, and
what the measurements did
+afterwards.
diff --git a/hudi-ai-operator/skills/hudi-optimizer/SKILL.md
b/hudi-ai-operator/skills/hudi-optimizer/SKILL.md
new file mode 100644
index 000000000000..0cdb31f6866f
--- /dev/null
+++ b/hudi-ai-operator/skills/hudi-optimizer/SKILL.md
@@ -0,0 +1,339 @@
+---
+name: hudi-optimizer
+description: Finds what an Apache Hudi deployment is leaving on the table.
Reads writer properties, a Spark event log, and the table itself, corroborates
observations across them, and recommends configuration and capability changes
with risk tiers and verification steps. Invoke when a Hudi job works but is
slow, expensive, or larger on storage than expected, or when a user asks
whether a feature such as the record-level index or clustering would help their
table.
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Hudi Optimizer
+
+You are looking at an Apache Hudi deployment that **works** and saying what it
is costing the user
+that it does not need to cost.
+
+This is not failure diagnosis. If the job is failing, that is `hudi-rca`. If
the table is being
+designed, that is `hudi-architect`. You are here for the middle case: it runs,
it is correct, and it
+is slower or larger or more expensive than it should be.
+
+**You recommend. You never apply.** The guardrails at the bottom are not
negotiable and are not
+waived by a user asking nicely.
+
+## The one thing to get right
+
+**An observation is only trustworthy when at least two of the three inputs
agree.**
+
+- Event log alone: *"the write stage is slow"* — useless.
+- Event log plus the table: *"...and p50 base file is 8 MB"* — now it is small
files.
+- Plus the writer properties: *"...and `hoodie.parquet.small.file.limit` sits
below your batch
+ average, so small-file handling never engages"* — now it is actionable and
it names the config.
+
+This is not a style preference. Reviewers of the sibling table-health tooling
found that **nearly
+every false verdict came from reading one signal without its governing config
or table version.**
+Hold the rule strictly. One signal is a hypothesis; say "hypothesis".
+
+## Step 1 — collect the three inputs, and say what you have
+
+| Input | What it is | What it unlocks |
+|---|---|---|
+| **Writer properties** | `--props` file or `--hoodie-conf` values, as the
writer actually runs them | Whether a lever is even live. **237 of 939 configs
are performance levers and 93 of those are conditionally applied** — a
recommendation made without the config may be silently inert |
+| **Spark event log** | The file written with `spark.eventLog.enabled=true` |
Where the time actually went, per Hudi phase |
+| **Table path** | The table itself, read-only | File sizes, partition skew,
service cadence, timeline state |
+
+**Ask for the event log, not "Spark UI access."** The UI is ephemeral and
cannot be attached; the
+event log is a file the user can hand you containing everything the UI shows.
+
+### Confidence scales with how many you have — state it every time
+
+| Inputs | What you are allowed to do |
+|---|---|
+| **One** | **Describe.** Report what you measured and what it might mean.
Name the second input that would make it a recommendation. Do not recommend a
change |
+| **Two** | **Recommend.** The corroboration rule is satisfied. State which
two agreed and which is missing |
+| **Three** | **Recommend, with verification steps.** You can predict the
measurement that will prove the change worked |
+
+Lead with this. A user who knows you are working from two of three inputs
reads your advice
+correctly; one who does not, does not.
+
+## Step 2 — run the tools yourself
+
+Do not ask the user to run three commands and paste three outputs. **Invoke
the tools.** All three
+are read-only and ship in `hudi-utilities`.
+
+**Table services and timeline state:**
+
+```
+spark-submit --class org.apache.hudi.utilities.HoodieTableHealthChecker \
+ --master "local[2]" "$HUDI_UTILITIES_BUNDLE" \
+ --base-path <table-path> \
+ --props <writer.properties> \
+ --output JSON
+```
+
+Exit `0` every check healthy or skipped, `3` at least one unhealthy, `4` a
check itself errored, `2` bad
+arguments or unreadable; there is no exit code `1`. Run it through
`spark-submit`: the bundle does not
+shade Jackson or the Spark classes it needs, so a bare `java -cp` fails with
`NoClassDefFoundError`.
+**A `SKIPPED` check is a request for `--props`, not a pass** — it names the
property it needed.
+
+**Layout, file sizes, partition skew:**
+
+```
+spark-submit --class org.apache.hudi.utilities.HoodieTableLayoutAnalyzer \
+ --master "local[2]" "$HUDI_UTILITIES_BUNDLE" \
+ --base-path <table-path> \
+ --enable-partition-stats --analyze-table-characteristics --output JSON
+```
+
+Filter stdout before parsing: with `--output JSON` the analyzer prints two
+`# WARNING: Unable to get Instrumentation...` lines (JOL attach) ahead of the
JSON, so write to a file
+and drop lines starting with `#`, or pipe through `grep -v '^# WARNING'`. Add
+`--include-row-counts` when you need record counts for index sizing — it is
slow without a
+`column_stats` partition, because every Parquet footer is read.
+
+**The event log:**
+
+```
+spark-submit --class org.apache.hudi.utilities.HoodieSparkEventLogAnalyzer \
+ --master "local[1]" "$HUDI_UTILITIES_BUNDLE" \
+ --event-log <path> --output JSON --output-file <local-file>
+```
+
+It opens no Spark session — it reads a file — but it needs Spark's jars on the
classpath (Jackson among
+them), so a bare `java -cp` on the bundle fails with
+`NoClassDefFoundError: com/fasterxml/jackson/databind/JsonNode`. Prefer
`--output-file`: some logging
+configurations write to stdout and would corrupt the JSON.
+
+**If the user already has output, accept it.** `--health-json` and
`--layout-json` skip the
+invocation. Use them when the user has a partial environment, cannot run
`spark-submit`, or has
+already paid for the run. If you cannot run a tool and have no JSON for it,
**say which specific
+question is now unanswered** rather than silently guessing at it.
+
+## Step 3 — read the measurements, not the verdicts
+
+**This is the rule that most changes your output.**
+
+The layout analyzer's `tableCharacteristics.overallStatus` is `FLAGGED` when
*any* detector found
+anything — and it currently fires on essentially every actively-written
streaming table. The health
+checker's thresholds carry deliberate slack (a 2.0x factor on compaction, 2.0x
and a floor of 5 on
+the cleaner, 1.1x on archival) because those tools exist to catch a service
that has **stopped**, not
+one that is merely behind.
+
+**So read the numbers underneath and apply your own thresholds.**
+
+- `detectors[].effectiveConfigs` — keys prefixed `observed.` are measurements,
`effective.` are the
+ thresholds that detector used. Compare them yourself.
+-
`health.checks[].effectiveConfigs["observed.delta.commits.since.last.compaction"]`
against the
+ configured trigger, not against the checker's slack-adjusted threshold.
+- `layout.partitions[]` per-partition numbers rather than the table-level
verdict.
+
+The observation catalog is written against the underlying fields for exactly
this reason. **An
+observation fires in the band where the health checker correctly reports
`HEALTHY`** — that band is
+the whole subject of this skill.
+
+Equally: never inherit a rollup verdict as a finding. "The layout analyzer
flagged this table" is not
+an observation. "40% of partitions with five or more base files average under
50 MB, against a
+configured 120 MB target" is.
+
+## Step 4 — carry the analyzer's caveats into your reasoning
+
+Read `caveats[]` in the event-log JSON and repeat what it says. Two caveats
govern almost everything.
+
+**A phase share names where work was *forced*, not where it belongs.** Hudi
sets the Spark job
+description before the action, and Spark is lazy, so a bucket routinely
carries the cost of an
+untagged step that ran before it. Known cases, all verified in source:
+
+- `SOURCE_READ_AND_TRANSFORM` bundles source read, user transformation and
record creation.
+ `StreamSync` tags only the fetch and the emptiness check;
`HoodieStreamerUtils` carries no job
+ status at all.
+- `DEDUP_AND_INDEX_TAGGING` bundles deduplication with tagging.
`BaseWriteHelper` has exactly one
+ `setJobStatus`, and `deduplicateRecords` runs untagged immediately before it.
+- `DATA_TABLE_WRITE` can absorb write work not forced until the commit action
ran.
+
+When a stage carries `bundledSteps`, the tool is telling you the share is
large enough that the
+bundling matters. Say what the number covers.
+
+**With `hoodie.metadata.streaming.write.enabled=true` — the Spark default — a
large share of the DAG
+is unattributable.** On a measured run it was **60.5% of stage wall clock** in
`UNKNOWN`: stages that
+are not fused but simply unlabelled, because Hudi never calls `setJobStatus`
on that path.
+
+> **When `UNKNOWN` is large, say so before anything else, and do not reason
over the remainder as if
+> it were the whole picture.** A phase at "30% of wall clock" when `UNKNOWN`
is 60% means something
+> very different from the same number when `UNKNOWN` is 4%. Setting that
config to `false` yields a
+> fully attributable breakdown, and that is a legitimate thing to suggest
**for a diagnostic run**.
+
+Two more, worth stating when they come up:
+
+- **`outputBytes` and `outputRecords` are structurally zero** on every stage
of a Hudi write. Hudi
+ writes Parquet through its own file handles, bypassing Spark's instrumented
writer. **Never infer
+ "no data was written" from them.** Input metrics are fine.
+- **`-1` means "not derivable", never zero.** `skewRatio` and
`gcFractionOfRunTime` are `-1` when
+ undefined.
+
+## Step 5 — match against the catalogs
+
+**`references/observation-catalog.md`** — thirty rows of "this works, but
could be better". Each row
+names its measurable signals as JSON field paths, which inputs must
corroborate it, the exact config
+key and the direction to move it, a risk tier, and how to verify.
+
+**`references/capability-catalog.md`** — seven rows of "adopt this feature",
with signals **for and
+against**, adoption cost, and risk tier.
+
+Two habits that keep this honest:
+
+**Never report an observation whose signals you did not actually find in the
JSON.** If you cannot
+name the field and the value, you do not have the observation. The catalogs
are written in field
+paths so that this check is mechanical.
+
+**Never recommend a lever the config shows is inert.** This is the specific
failure the writer
+properties exist to prevent. Before recommending a key, check whether it is
gated:
+
+- `hoodie.parquet.small.file.limit` does nothing if the condition for
appending never holds.
+- `hoodie.metadata.index.partition.stats.enable` has **no independent effect**
— partition stats
+ require column stats and are inert without them.
+- `hoodie.bloom.index.fileid.key.sorting.enable` does nothing on its own;
bucketized checking stays
+ on unless `hoodie.bloom.index.bucketized.checking=false` is also set.
+- `hoodie.*.shuffle.parallelism` defaults to **`0`, not 200** — `0` means
"inherit the input RDD's
+ partition count", so a low task count may be reporting the source, not a
Hudi setting.
+- Setting `hoodie.bulkinsert.shuffle.parallelism` on an upsert workload does
nothing at all.
+
+## Step 6 — engine-effective defaults, not declared ones
+
+`ConfigProperty.defaultValue()` is **not** the effective default. Builders
override per engine with
+`setDefaultValue(PROP, getDefaultXxx(engineType))`. A derived-expectation
check that compares against
+a declared default is wrong about every default Spark table.
+
+| Config | Declared | **Spark** | Flink | Java |
+|---|---|---|---|---|
+| `hoodie.index.type` | — | **`SIMPLE`** | `INMEMORY` | `SIMPLE` |
+| `hoodie.metadata.index.column.stats.enable` | `false` | **`true`** | `false`
| `false` |
+| `hoodie.metadata.streaming.write.enabled` | `false` | **`true`** | `false` |
`false` |
+| `hoodie.metadata.index.secondary.enable` | `true` | `true` | **`false`** |
`true` |
+| `hoodie.metadata.enable` | declared | declared | declared | **`false`** |
+| `hoodie.write.markers.type` | — | **`TIMELINE_SERVER_BASED`** | `DIRECT` |
`DIRECT` |
+
+The clustering plan and execution strategy classes are also engine-selected,
**and additionally
+branch on whether the index is consistent-hashing bucket** — a
config-dependent default, not just an
+engine-dependent one.
+
+The practical consequence for the flagship recommendation: **a Spark user is
on `SIMPLE` by default,
+not by choice.** Word it as *"you are on the default index; here is why the
record-level index may
+suit your workload"*. Never imply they chose it.
+
+## Step 7 — refuse durable changes, structurally
+
+| Tier | Meaning | What you do |
+|---|---|---|
+| `safe` | Reversible config, no effect on storage | Recommend |
+| `operational` | Affects a table service; reversible but needs a cycle to
observe, and the intermediate state is on storage | Recommend, with the cycle
and the cost named |
+| `durable` | **Changes what is on storage. Hard or impossible to undo** —
table type, partitioning, key generator, index type, record-key encoding |
**Refuse. Hand off** |
+
+> **You do not recommend a `durable` change as an optimization.** Not as "you
could just set", not in
+> a config block, not as an aside. When the measurements point at one —
CAP-001 (index type), CAP-002
+> (bucket index and count), CAP-006 (table type), or OBS-025 — your output is
a **report**: what you
+> measured, what it implies, which counter-signals did and did not fire, and
an explicit handoff to
+> `hudi-architect` for an Architecture Decision Record.
+
+Say the refusal out loud rather than silently omitting the row. The user
should know the measurement
+points somewhere, that you found it, and why you are not the one to act on it:
+
+> *Your index tagging is 41% of wall clock and its task count tracks your
base-file count rather than
+> your batch size, which is the signature of an index whose cost scales with
the table. That points
+> at the record-level index. **Changing `hoodie.index.type` is a durable
change — it alters what is on
+> storage and cannot be reverted by resetting a config — so I will not
recommend it as a tuning step.**
+> Here are the measurements and the file-group estimate; take them to
`hudi-architect` for an ADR.*
+
+The report should be good enough that the design conversation starts from
numbers. That is the point
+of the handoff, not a way of ducking the question.
+
+## Output shape
+
+1. **What you have** — which of the three inputs are present, and the
confidence that buys. First,
+ every time.
+2. **Where the time went** — the phase breakdown, with the `UNKNOWN` share
stated up front if it is
+ material, and the attribution caveat named.
+3. **Observations** — matched catalog rows, each with: the measured values you
actually found, which
+ inputs corroborated, the change and direction, risk tier, and expected
payoff. Order by payoff,
+ not by catalog order.
+4. **Capability candidates** — with signals for **and** against. Say "don't"
when the counter-signals
+ fire; that is the output, not a failure to produce one.
+5. **Durable findings, refused and handed off** — stated plainly, with the
measurements attached.
+6. **What is missing** — which input you did not have and what it would have
settled. Specifically,
+ not as a generic ask.
+
+Lead with the confidence statement and the biggest number. Do not recap the
user's setup back to
+them.
+
+## Guardrails
+
+**Recommend only. Never apply.** Do not run a table service, do not write to
the table, do not edit
+`.hoodie/`, do not modify a properties file. You read artifacts, run the three
read-only tools, and
+explain. That is the whole surface.
+
+**Never recommend a `durable` change.** Above. Structural, not advisory.
+
+**Never recommend a lever the config shows is inert.** Check the gating before
you name the key.
+
+**Never report an observation from one input.** Say "hypothesis" and name the
input that would settle
+it.
+
+**Do not invent a config key.** Every `hoodie.*` key in the catalogs is
verified against the source
+tree. If a key is not in a catalog, say you are unsure of the exact key rather
than guessing a
+plausible one — **a misspelled Hudi config key produces no error at all**, the
default silently
+applies, and the user loses a whole tuning cycle finding out.
+
+**Do not promise a number you cannot verify.** "This will make it 3x faster"
is not something these
+inputs support. "This removes the spill on that stage; the stage currently
costs 12% of wall clock"
+is.
+
+**One change at a time where you can.** If you recommend five changes and the
run gets slower, the
+user has learned nothing. Order by payoff and say which one to try first.
+
+**Releasing a savepoint is not routine.** A savepoint protects files from
cleaning. If a stale
+savepoint is pinning the timeline (OBS-020), report it and require a human to
confirm it is no longer
+needed. Never fold it into a list of tuning steps.
+
+**Redact before quoting.** Users paste properties files containing bucket
names, hostnames and
+credentials. Do not echo them back unnecessarily.
+
+**If the job is failing, stop and hand to `hudi-rca`.** This skill assumes a
working deployment.
+Tuning a job that is failing for an unrelated reason wastes the user's time
and yours.
+
+## Engine coverage
+
+The recipes here come from Spark deployments, because that is where the
evidence came from — the
+event-log analyzer reads Spark event logs, and the measurements behind these
rows were taken on Spark
+runs.
+
+The **core patterns apply to Flink too**: file sizing, service cadence,
timeline growth, metadata-table
+cost, index economics and the capability tradeoffs are table-layer mechanisms,
not engine ones. What
+does not carry is the event-log half — Flink produces no Spark event log, so a
Flink user arrives with
+two inputs at most, and the phase breakdown is simply unavailable. Several
engine-effective defaults
+also differ, as the table in Step 6 shows.
+
+Flink-specific coverage will deepen over time. Say this plainly if a Flink
user arrives: the
+table-layer reasoning holds, the stage-level reasoning does not, and you will
flag which is which
+rather than pretending the distinction does not exist.
+
+## References
+
+- `references/observation-catalog.md` — thirty rows, "this works, but could be
better".
+- `references/capability-catalog.md` — seven rows, "adopt this feature", with
counter-signals.
+- `HoodieSparkEventLogAnalyzer.md` in `hudi-utilities` — the event-log JSON
contract, the phase and
+ operation model, and the caveats in full.
+- `HoodieTableHealthChecker.md` and `HoodieTableLayoutAnalyzer.md` — the other
two JSON contracts,
+ and the thresholds their verdicts use.
+- `hudi-rca` — when the job is failing rather than slow.
+- `hudi-architect` — where every `durable` finding goes.
diff --git
a/hudi-ai-operator/skills/hudi-optimizer/references/capability-catalog.md
b/hudi-ai-operator/skills/hudi-optimizer/references/capability-catalog.md
new file mode 100644
index 000000000000..366274ee1936
--- /dev/null
+++ b/hudi-ai-operator/skills/hudi-optimizer/references/capability-catalog.md
@@ -0,0 +1,576 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Capability catalog
+
+Seven rows that say **adopt this feature** — or, just as often, **do not**.
+
+A different kind of claim from the observation catalog. A config observation
says a lever is set
+wrongly. A capability row says a mechanism the table is not using would suit
this workload, and it
+needs the workload's *shape* rather than one run's measurements. Payoff is
sometimes 10x rather than
+10-30%, and the risk is adoption cost rather than a slower job.
+
+**This is the part documentation cannot do.** Any Hudi page can say "the
record-level index gives
+faster lookups". None of them can say *for your table, with your update
pattern, at your size* —
+that needs the three inputs, which is why they are mandatory here too.
+
+## The discipline, restated for this catalog
+
+**Every row must be able to say "don't".** A capability row with no
counter-signals is marketing. The
+counter-signals below are as measurable as the signals for, and when they fire
the right answer is to
+say so plainly and stop. A user who adopts the record-level index on a
time-ordered append-only
+table has paid metadata-table overhead for nothing, and will trust you less
afterwards.
+
+| Field | Meaning |
+|---|---|
+| **Preconditions** | Hard gates. If one fails, the row does not apply and no
amount of signal changes that |
+| **Signals FOR** | Measured fields in the three JSON outputs that argue for
adoption |
+| **Signals AGAINST** | Measured fields that argue against. Weighted equally |
+| **Payoff** | What to expect, with the mechanism that produces it |
+| **Adoption cost** | Bootstrap time, ongoing metadata-table overhead,
operational burden |
+| **Risk** | `safe` · `operational` · `durable` |
+| **Verify** | What proves it worked |
+
+### Risk tiers, and the refusal
+
+Same three tiers as the observation catalog. The difference is that **this
catalog contains `durable`
+rows**, because the honest answer to some measurements is a change to what is
on storage.
+
+> **The agent must refuse to recommend a `durable` change as an
optimization.** For CAP-001 and
+> CAP-006 the output is a *report* — the measurements, what they imply, and a
handoff to
+> `hudi-architect` for an ADR. Not a config block, not a migration script, not
"you could just set".
+> This is structural: the rows are written as handoffs and say so in their
Change field.
+
+The reason is specific rather than squeamish: an index type or table type
change is written into
+`hoodie.properties` and reflected in what is on storage. Reverting means
rewriting the table. An
+optimization pass run against a live table is the wrong place for a decision
with that blast radius,
+however good the evidence.
+
+## Index
+
+| ID | Capability | Risk | One line |
+|---|---|---|---|
+| CAP-001 | SIMPLE/BLOOM → **record-level index** | `durable` | The flagship.
Tagging cost stops scaling with table size |
+| CAP-002 | → **bucket index** | `durable` | Predictable cardinality, no index
lookup at all |
+| CAP-003 | → **clustering** | `operational` | Consolidate small files, and
give column stats something to prune on |
+| CAP-004 | → **column and partition stats** | `operational` | Query pruning —
but only with locality |
+| CAP-005 | → **secondary index** | `operational` | Lookups on a non-key
column |
+| CAP-006 | CoW → **Merge-on-Read** | `durable` | Write amplification, traded
for read-side merge |
+| CAP-007 | → **metadata table** | `operational` | Stop listing storage. The
prerequisite for most of the above |
+
+---
+
+## CAP-001 — SIMPLE or BLOOM index → record-level index
+
+**The flagship row.** Read it carefully, including the counter-signals.
+
+### Preconditions
+
+- The metadata table is enabled and healthy. **The record-level index is a
metadata-table
+ partition** — without `hoodie.metadata.enable=true` it cannot exist. See
CAP-007.
+- The table has a meaningful rate of **updates**. An append-only table does
not need an index at all;
+ see "Signals AGAINST".
+- The writer is on a release that has the index (0.14.0 for the global form).
+
+### Start from the default, not from a choice
+
+**`hoodie.index.type` defaults to `SIMPLE` on Spark** — the declared default
is overridden per engine
+in the index-config builder, so a Spark user who never set it is on `SIMPLE`.
Word every
+recommendation accordingly: *"you are on the default index; here is why the
record-level index may
+suit your workload"*. Never imply the user chose SIMPLE and chose wrong. They
did not choose.
+
+(The same per-engine override makes Flink default to `INMEMORY` and Java to
`SIMPLE`. If you are
+reading a declared default from documentation, you are reading the wrong
number.)
+
+### The mechanism, verified in source
+
+`HoodieSimpleIndex.fetchRecordLocations` computes its parallelism as
+`getParallelism(config.getSimpleIndexParallelism(), baseFiles.size())` and
reads key columns from
+each base file. **Tagging cost therefore scales with the number of base files
in the table, not with
+the size of the batch being written.** A 500-record update against a
200,000-file table does the same
+index work as a 5-million-record update.
+
+The record-level index is a key-to-location map in the metadata table. A
lookup is a point read
+against that map, so its cost scales with the **batch**. That difference —
table-size-scaling versus
+batch-scaling — is the whole argument, and it is why the benefit grows as the
table does.
+
+### Signals FOR
+
+| Signal | Field | What it tells you |
+|---|---|---|
+| The premise | `props`: `hoodie.index.type` is `SIMPLE`, `GLOBAL_SIMPLE`,
`BLOOM` or `GLOBAL_BLOOM` | Nothing to adopt otherwise |
+| **The smoking gun** | `eventlog.stages[].observed.numTasksPlanned` on
`DEDUP_AND_INDEX_TAGGING` stages ≈ `layout.totalFiles`, **not** the batch's
record count | Cost is scaling with the table |
+| Payoff size |
`eventlog.phaseSummary[phase=DEDUP_AND_INDEX_TAGGING].shareOfStageWallClock` |
How much of the run is in play |
+| Which component | `eventlog.stages[].subPhase` is `bloomIndexLookup` (or the
bucket has no RLI sub-phase) | Confirms an index lookup, not just profiling |
+| Updates are real | `table` commit metadata: `numUpdateWrites` a meaningful
share of `numInserts`, over several commits | An index earns its cost only
against updates |
+| Updates are scattered | Updated file groups spread across partitions and
ages, over N commits | Scattered updates defeat key-range pruning |
+| The table is large and growing | `layout.totalFiles`, `layout.totalBytes`
rising | SIMPLE degrades with size; RLI does not |
+| Prerequisite met | `layout.mdtEnabled` true,
`health.checks[name=compaction]` on the metadata table healthy | The index
lives there |
+
+### Signals AGAINST — weighted equally
+
+| Signal | Field | Why it argues against |
+|---|---|---|
+| **Append-only** | `numUpdateWrites` near zero across many commits | No
lookups to accelerate. The index is pure write-side cost |
+| **Time-ordered keys with recent-partition updates** | Updated file groups
concentrated in the newest partitions; keys monotonic | BLOOM's key-range
pruning is already near-optimal here. RLI adds metadata-table overhead for
nothing — **this is the single most important counter-signal** |
+| Small table | `layout.totalFiles` small; tagging share already low | The
cost RLI removes is not being paid |
+| Tagging share already small |
`phaseSummary[DEDUP_AND_INDEX_TAGGING].shareOfStageWallClock` low | There is no
headroom to recover |
+| Metadata table unhealthy | `health.checks` on the metadata table showing a
compaction backlog | Adding a partition to a struggling metadata table makes
things worse |
+| The bucket share is dedup, not index | `eventlog.stages[].bundledSteps`
names deduplication | `BaseWriteHelper` leaves `deduplicateRecords` untagged,
so dedup lands in this bucket. A large share may be dedup |
+| Cardinality is predictable and fixed | — | CAP-002 may be the better answer:
no index lookup at all |
+
+### How to size it before recommending it
+
+**Do not recommend the index without estimating what it will cost.** Hudi's
own sizing logic
+(`HoodieTableMetadataUtil.estimateFileGroupCount`) takes the record count,
multiplies by a growth
+factor, divides by the records that fit in one file group, and clamps to a
configured range:
+
+- `hoodie.metadata.record.index.growth.factor` — default `2.0`, the headroom
multiplier
+- `hoodie.metadata.record.index.max.filegroup.size` — default 1 GiB per file
group
+- `hoodie.metadata.global.record.level.index.min.filegroup.count` — default
`10`
+- `hoodie.metadata.global.record.level.index.max.filegroup.count` — default
`10000`
+
+Take the table's record count from `layout` with `--include-row-counts`, or
from commit metadata, and
+work it through. Report the result: *"your record count implies roughly N file
groups; bootstrap
+writes that index once over the whole table"*. A recommendation that carries
its own cost estimate is
+the difference between advice and a sales pitch.
+
+If the estimate lands at the `max.filegroup.count` ceiling, say so — the index
will be clamped and
+each file group will exceed the target size, which makes metadata-table
compaction heavier.
+
+### Payoff
+
+Index tagging stops scaling with table size. On the measured e2e runs,
`recordIndexLookup` isolated
+to **3.6 seconds** for the batch — and crucially, that number is a function of
the batch, so it does
+not grow as the table does. The gain is whatever `DEDUP_AND_INDEX_TAGGING`
currently costs minus a
+batch-scaled lookup, and it **compounds**: the comparison gets better every
month the table grows.
+
+The measurement that makes this row possible is recent. The record-level index
lookup runs inside the
+job Hudi describes as `Building workload profile:`, so its cost used to be
pooled with profiling and
+was not separable. `HoodieSparkEventLogAnalyzer` resolves it to `subPhase:
"recordIndexLookup"` from
+the stage name. **Without that sub-phase, this row would have no measurable
before-and-after.**
+
+### Adoption cost
+
+- **Bootstrap.** The index is built over the whole table on first enable. On a
large table this is a
+ significant one-time job — size it with the file-group estimate above.
+- **Ongoing metadata-table overhead.** Every commit writes record-index
entries. This shows up in
+ `phaseSummary[phase=METADATA_TABLE_WRITE]`, and partly in `UNKNOWN` under
the streaming-write
+ default (OBS-023).
+- **Operational.** One more metadata-table partition that must stay compacted.
The metadata table's
+ own compaction cadence (`hoodie.metadata.compact.max.delta.commits`) now
matters more.
+- **Irreversible in practice.** The index type is table configuration and the
index is on storage.
+
+### Use the global key — and never recommend partitioned RLI for pruning
+
+Two keys exist and they are **not** interchangeable:
+
+- `hoodie.metadata.global.record.level.index.enable` (alternative:
+ `hoodie.metadata.record.index.enable`) — **the global form. This is the one
to use.**
+- `hoodie.metadata.record.level.index.enable` — the **partitioned** form,
where a partition-path plus
+ record-key pair is unique across the table.
+
+**File pruning with the partitioned form is not implemented.** The code path
throws
+`IllegalArgumentException: File pruning with partitioned rli has not yet been
implemented`
+(`HoodieBackedTableMetadata`). **Never recommend the partitioned form
expecting pruning benefits.**
+This was found the direct way during the e2e work: a run configured with the
partitioned key died on
+that exception.
+
+### Risk
+
+**`durable`.** `hoodie.index.type` changes what is on storage and how records
map to file groups.
+
+> **The agent does not recommend this change.** It reports: the measurements,
the file-group
+> estimate, the counter-signals that did and did not fire, and a handoff to
`hudi-architect` for an
+> ADR. The user makes the decision with the design skill, not with the
optimizer.
+
+The report should be good enough that the design conversation starts from
numbers rather than from
+first principles. That is this row's actual job.
+
+### Verify (once adopted, by whoever adopted it)
+
+- `eventlog.stages[].subPhase == "recordIndexLookup"` appears, and its total
duration tracks **batch
+ size** rather than `layout.totalFiles`. **That relationship is the proof**,
not the absolute number.
+- `phaseSummary[phase=DEDUP_AND_INDEX_TAGGING].shareOfStageWallClock` falls.
+- `phaseSummary[phase=METADATA_TABLE_WRITE]` rises — this is the cost side,
and it should be visible.
+ If it is not, check whether `UNKNOWN` grew instead (the streaming-write
caveat).
+- Re-measure after the table has grown materially. The point of the change is
that the cost stops
+ growing; a single post-change measurement cannot show that.
+
+---
+
+## CAP-002 — → bucket index
+
+### Preconditions
+
+- Record-key cardinality is **predictable and reasonably stable**. This is a
hard gate: bucket count
+ is fixed at table creation for the simple engine and resizing is not free.
+- The workload is upsert-heavy against a known key space.
+
+### Signals FOR
+
+| Signal | Field |
+|---|---|
+| Index tagging is a large share |
`eventlog.phaseSummary[phase=DEDUP_AND_INDEX_TAGGING].shareOfStageWallClock` |
+| Key space is stable | `table` commit metadata: records per commit steady
across many commits |
+| Updates dominate | `numUpdateWrites` high against `numInserts` |
+| Multi-writer lock contention is the bottleneck | `props`:
`hoodie.write.concurrency.mode` set, with long lock waits observed |
+
+Bucket index computes the target file group from a hash of the key. **There is
no index lookup at
+all** — not a base-file scan, not a metadata-table read. That is why it can
beat the record-level
+index on write latency, and why it pairs well with non-blocking concurrency
control on MOR.
+
+### Signals AGAINST
+
+| Signal | Field | Why |
+|---|---|---|
+| **Cardinality is growing or unknown** | Records per commit trending up;
table young | A fixed bucket count that was right at 1K file groups is wrong at
500K. The failure is gradual and hard to reverse |
+| Partition sizes are very uneven | `layout.skew.cv`, `.gini`,
`.largestPartitionShare` high | Buckets are per partition, so a skewed table
gets skewed buckets |
+| Updates are scattered across partitions and the key is not the partition key
| — | The global form is more constrained; check it supports your access
pattern first |
+| Queries need point lookups on the key | — | Bucket index helps the **write**
path. The record-level index also serves reads |
+
+### Payoff
+
+Index tagging cost goes to approximately zero on the write path. Largest where
tagging currently
+dominates and the key space is genuinely stable.
+
+### Adoption cost
+
+- Bucket count must be chosen up front: `hoodie.bucket.index.num.buckets`.
Getting it wrong is
+ expensive to correct.
+- `hoodie.index.bucket.engine` selects `SIMPLE` (fixed) or
`CONSISTENT_HASHING` (resizable, with its
+ own operational surface — resizing is a table service).
+- `hoodie.bucket.index.hash.field` if hashing on something other than the
record key.
+- Consistent hashing also changes which clustering strategy class is selected
by default, so the
+ clustering configuration interacts with this choice.
+
+### Risk
+
+**`durable`.** Index type and bucket count are on storage.
+
+> **Handoff to `hudi-architect`.** Report the measurements and the cardinality
evidence. Bucket count
+> is exactly the kind of sizing decision that wants an ADR with a revisit
condition attached.
+
+### Verify
+
+`DEDUP_AND_INDEX_TAGGING` share falls sharply. Watch `layout.skew` — bucket
index can concentrate
+writes if the hash field is skewed, and that shows up as partition-level skew.
+
+---
+
+## CAP-003 — → clustering
+
+**The highest-value `operational` row in this catalog**, because it is the
only one here that is both
+reversible and frequently transformative.
+
+### Preconditions
+
+- The table has small files to consolidate, or queries that would prune with
locality.
+- There is capacity to run it — inline clustering pauses ingestion; async
needs its own resources.
+
+### Signals FOR
+
+| Signal | Field |
+|---|---|
+| Small files |
`layout.tableCharacteristics.detectors[name=small-files].status` is `MODERATE`
or `SEVERE` — **read `observed.flagged.pct`, not the status alone** |
+| File sizes under target | `layout.tableSizeStats.p50` well below `props`:
`hoodie.parquet.max.file.size` |
+| Micro-partitions |
`detectors[name=micro-partition].effectiveConfigs["observed.size.rule.triggered"]`
true |
+| Nothing scheduled | `props`: `hoodie.clustering.inline` and
`hoodie.clustering.async.enabled` both `false`; no clustering `.replacecommit`
on the timeline |
+| Column stats maintained without locality |
`hoodie.metadata.index.column.stats.enable` effectively on (the Spark default)
with `hoodie.clustering.plan.strategy.sort.columns` unset |
+| File-group count growing | `layout.totalFiles` rising while
`tableSizeStats.mean` flat |
+
+### Signals AGAINST
+
+| Signal | Field | Why |
+|---|---|---|
+| Files already at target | `layout.tableSizeStats.p50` near
`hoodie.parquet.max.file.size` | Nothing to consolidate |
+| **Heavy update traffic on the same file groups** | `numUpdateWrites` high
and concentrated | Clustering rewrites file groups that updates will
immediately rewrite again. Wasted IO, and conflict risk with concurrent writers
|
+| A large backlog with no plan cap | Table old, `layout.totalFiles` very
large, no clustering history | **Enabling clustering unbounded on a big backlog
is how HUDI-RCA-022 happens.** Cap the first plan or do not start |
+| Compaction already behind on MOR | `health.checks[name=compaction]` numbers
drifting | Fix compaction first; two services competing for the same file
groups helps neither |
+
+### Payoff
+
+File count falls, file sizes rise toward target, and — with `sort.columns` set
— column stats become
+selective so query pruning starts working. The second effect is often larger
than the first and is
+the one users do not anticipate.
+
+### Adoption cost
+
+- A rewrite of the file groups in each plan: real IO, real cluster time.
+- Clustering produces `.replacecommit` instants; the timeline and the cleaner
both see more work.
+- With concurrent writers, clustering participates in conflict resolution. On
a multi-writer table
+ this needs thinking about rather than switching on.
+
+### Configuration, with directions
+
+| Key | Direction |
+|---|---|
+| `hoodie.clustering.inline` **or** `hoodie.clustering.async.enabled` | one of
them on, not both |
+| `hoodie.clustering.inline.max.commits` /
`hoodie.clustering.async.max.commits` | set to your commit cadence |
+| `hoodie.clustering.plan.strategy.max.num.groups` | **down** from default for
the first rounds |
+| `hoodie.clustering.plan.strategy.max.bytes.per.group` | sized to what one
round can execute |
+| `hoodie.clustering.plan.strategy.small.file.limit` | the size below which a
file is a candidate |
+| `hoodie.clustering.plan.strategy.target.file.max.bytes` | the size you want
out |
+| `hoodie.clustering.plan.strategy.sort.columns` | the columns queries filter
on — this is what makes stats selective |
+| `hoodie.clustering.plan.strategy.daybased.lookback.partitions` | on a
date-partitioned table, bounds the candidate set |
+
+### Risk
+
+**`operational`.** Reversible by ceasing to schedule it, but the rewritten
files stay. Start bounded.
+
+### Verify
+
+`layout.totalFiles` falls and `tableSizeStats.p50` rises round over round. In
the event log,
+`operationSummary[operation=CLUSTERING]` appears with `EXECUTION` dominating
`PLANNING` — planning
+dominating means the plan is too big for the round, and that is the early
warning for OBS-018.
+
+---
+
+## CAP-004 — → column and partition statistics
+
+### Preconditions
+
+- The metadata table is enabled (CAP-007).
+- **Queries filter on specific columns.** Without this, the row does not apply
and no measurement
+ changes that.
+
+### Check what you already have before recommending anything
+
+**On Spark, `hoodie.metadata.index.column.stats.enable` is effectively
`true`** — the declared
+default is `false` and the Spark builder overrides it. So the common case is
not "adopt column
+stats"; it is "you already have them and they are not pruning anything". That
is OBS-013 and OBS-024,
+not this row.
+
+This row applies to a **Flink or Java-engine writer**, where the effective
default is `false`, or to
+a table where they were explicitly disabled.
+
+### Signals FOR
+
+| Signal | Field |
+|---|---|
+| Stats genuinely off | `props`: `hoodie.metadata.index.column.stats.enable`
false, and no `column_stats` partition under the metadata table |
+| Queries would prune | Query evidence — **you probably do not have it; ask** |
+| Data has locality, or clustering can give it |
`hoodie.clustering.plan.strategy.sort.columns` set, or clustering adoptable
(CAP-003) |
+| Many files per partition | `layout.fileCountPerPartition.mean` high — more
files to prune from means more to gain |
+
+### Signals AGAINST
+
+| Signal | Field | Why |
+|---|---|---|
+| **No locality and no plan to create it** | No clustering, no sort columns,
random arrival order | Every file's min/max spans the full range. The index is
maintained and prunes nothing — the most common way this disappoints |
+| Queries do not filter | — | Pure write-side cost |
+| Very wide schema with stats on every column | Schema column count high |
Stats are written per column per file. Narrow it with
`hoodie.metadata.index.column.stats.column.list` |
+| Metadata-table write cost already high |
`phaseSummary[phase=METADATA_TABLE_WRITE].shareOfStageWallClock` | You are
adding to a cost that is already visible |
+
+### Payoff
+
+Files skipped at query planning time. Potentially large on a selective filter
over sorted data;
+**exactly zero** without locality. Be explicit about that conditional — it is
the difference between
+a useful recommendation and one that quietly fails.
+
+### Adoption cost
+
+- Stats computed and written on every commit, per column, per file.
+- Readers must set `hoodie.enable.data.skipping=true` to consume them. **This
is a read-side config
+ and is frequently absent from the writer properties entirely** — say so
rather than assuming it.
+- `hoodie.metadata.index.partition.stats.enable` rides on column stats and has
**no independent
+ effect**; there is no standalone partition-stats switch.
+
+### Risk
+
+**`operational`.** The index partition is built on enable and deleted on
disable.
+
+### Verify
+
+A filtered query reads fewer files. **Without a before-and-after query plan
this is unverified** —
+say that rather than claiming success from the write side.
+
+---
+
+## CAP-005 — → secondary index
+
+### Preconditions
+
+- The metadata table is enabled, **and the record-level index is enabled** — a
secondary index maps a
+ non-key column value to record keys, so it is built on top of the primary
key index. Without it,
+ there is nothing to map to.
+- A specific non-key column is being looked up repeatedly.
+- Hudi 1.0.0 or later.
+
+### Signals FOR
+
+| Signal | Field |
+|---|---|
+| Record-level index present | `props`:
`hoodie.metadata.global.record.level.index.enable` true; the index partition
exists |
+| A known non-key lookup column | Query evidence — **ask for it; this row
cannot be derived from write-side telemetry** |
+| Queries on that column scan broadly | Query evidence |
+
+### Signals AGAINST
+
+| Signal | Field | Why |
+|---|---|---|
+| No record-level index | `props` | Precondition fails. CAP-001 comes first,
and it is `durable` |
+| Low-cardinality column | — | A near-constant column maps to almost every
record; the index prunes nothing |
+| Column stats already prune it well | — | Cheaper mechanism, already in place
|
+| Metadata-table write cost already high |
`phaseSummary[phase=METADATA_TABLE_WRITE]` | Another partition on every commit |
+
+### Payoff
+
+Point lookups on the indexed column stop scanning. Depends entirely on
cardinality and query
+frequency — both of which are query-side facts you must **ask for** rather
than infer.
+
+### Adoption cost
+
+- Another metadata-table partition, written on every commit.
+- The index is created through `CREATE INDEX` in Spark SQL per column, not by
a single config.
+ `hoodie.metadata.index.secondary.enable` is the master switch that keeps
them consistent; on Spark
+ its effective default is already `true`, so the switch is usually not the
missing piece — the
+ `CREATE INDEX` statement is.
+- Dropping it is `DROP INDEX`.
+
+### Risk
+
+**`operational`.**
+
+### Verify
+
+Query-side. `phaseSummary[phase=METADATA_TABLE_WRITE]` rising is the cost
side, visible from the
+write path; the benefit is not, so do not claim it from write-side evidence.
+
+---
+
+## CAP-006 — Copy-on-Write → Merge-on-Read
+
+### Preconditions
+
+- Write amplification on CoW is measurably costing something. This row is
**not** a default upgrade
+ path; most tables are correctly CoW.
+
+### Signals FOR
+
+| Signal | Field |
+|---|---|
+| Write phase dominates |
`eventlog.phaseSummary[phase=DATA_TABLE_WRITE].shareOfStageWallClock` high |
+| Small updates rewriting large files | `table`: `numUpdateWrites` small per
commit while bytes written is large — **the write-amplification ratio, and the
core signal** |
+| Ingestion latency is the binding constraint | — a product fact, so **ask** |
+| Updates scattered across many file groups | Updated file groups spread wide
per commit |
+
+### Signals AGAINST
+
+| Signal | Field | Why |
+|---|---|---|
+| **Append-only** | `numUpdateWrites` near zero | MOR solves write
amplification from updates. With no updates there is none, and you have added
read-side merge for nothing — **this is OBS-025 read in reverse** |
+| Read latency matters more than write latency | — | MOR trades write cost for
read-side merge |
+| No capacity to run compaction | `health.checks[name=compaction]` would be a
new obligation | **MOR without working compaction degrades without bound.**
This is the most common MOR regret |
+| Updates concentrated in few file groups | | CoW rewrite cost is already
small |
+| Non-partitioned table | `layout.numPartitions` is 1 or absent | A
non-partitioned MOR table is pathological for the listing-based rollback path
and the whole-table file-system view (HUDI-RCA-021) |
+
+### Payoff
+
+Write amplification falls — updates append to log files instead of rewriting
base files. Paid back on
+the read side, and in compaction cost.
+
+### Adoption cost
+
+- **Compaction becomes mandatory.** Cadence, target IO, and monitoring are all
new obligations.
+- Readers see merge cost on uncompacted slices; snapshot queries get slower.
+- Log files are not counted by the layout analyzer's size statistics, so your
layout visibility gets
+ slightly worse — a small but real operational cost.
+- More table-service surface overall.
+
+### Risk
+
+**`durable`.** Table type is written into `hoodie.properties` and determines
what is on storage.
+
+> **The agent does not recommend this change.** Report the write-amplification
measurement, the
+> counter-signals, and the compaction obligation, and hand off to
`hudi-architect` for an ADR. A
+> table-type change on a live table is a design decision with a migration, not
an optimization.
+
+### Verify
+
+Not applicable here. If a user has already made the change, `DATA_TABLE_WRITE`
share should fall and
+a `COMPACTION` operation should appear in `operationSummary[]` — and OBS-015
and OBS-028 become the
+rows that matter.
+
+---
+
+## CAP-007 — → metadata table
+
+Listed last because it is a **prerequisite** for CAP-001, CAP-004 and CAP-005
rather than a
+destination of its own.
+
+### Preconditions
+
+None beyond a writable table.
+
+### Check the engine default first
+
+`hoodie.metadata.enable` is effectively **`true` on Spark and Flink** and
**`false` on the Java
+engine**. On Spark this row almost never applies — the metadata table is
already there. Confirm from
+`layout.mdtEnabled` rather than from the declared default.
+
+### Signals FOR
+
+| Signal | Field |
+|---|---|
+| Not enabled | `props`: `hoodie.metadata.enable` false; `layout.mdtEnabled`
false |
+| Listing cost visible | `eventlog.stages[].jobDescription` containing
`Parallel listing paths`, with real `observed.durationMillis` |
+| Many partitions | `layout.numPartitions` high — listing cost scales with it |
+| A capability above is wanted | CAP-001, CAP-004 or CAP-005 is the actual
goal |
+
+### Signals AGAINST
+
+| Signal | Field | Why |
+|---|---|---|
+| Small or unpartitioned table | `layout.numPartitions` small,
`layout.totalFiles` small | **Metadata-table write overhead can exceed the
listing it saves.** The clearest "don't" in this catalog |
+| Very high commit rate with tiny commits | `table`: many commits, few records
each | Metadata-table write cost is per commit |
+| No capacity for another compacting service | — | The metadata table compacts
itself and that cadence must be managed |
+
+### Payoff
+
+File listing becomes a metadata-table read instead of a storage listing. Grows
with partition count
+and with object-store listing latency; negligible on a small local table.
+
+### Adoption cost
+
+- One-time bootstrap over the existing table.
+- Per-commit write cost — and on Spark, much of it is **unattributable** in
the event log because
+ `hoodie.metadata.streaming.write.enabled` defaults to `true`. Expect
`UNKNOWN` to grow, and say
+ that is expected rather than treating it as a measurement failure.
+- The metadata table has its own timeline, its own compaction, and its own
health
+ (`hoodie.metadata.compact.max.delta.commits`).
+
+### Risk
+
+**`operational`.** Bootstrapped on enable; the partitions are removed on
disable.
+
+### Verify
+
+`layout.mdtEnabled` reads true. `Parallel listing paths` stages against the
data table disappear.
+`phaseSummary[phase=METADATA_TABLE_WRITE]` appears as the cost — and if it
does not appear while
+`UNKNOWN` grows, that is the streaming-write caveat, not an error.
+
+---
+
+## Adding a row
+
+Give it the next free ID. **A row without measurable counter-signals does not
belong here** — if you
+cannot state what would make the answer "don't", you have not finished the
row. Name the exact JSON
+field for every signal. Verify every `hoodie.*` key against the source tree,
and check whether its
+effective default is overridden per engine before citing a default at all. If
the change is
+`durable`, write the Change field as a handoff and say the agent refuses to
recommend it.
diff --git
a/hudi-ai-operator/skills/hudi-optimizer/references/observation-catalog.md
b/hudi-ai-operator/skills/hudi-optimizer/references/observation-catalog.md
new file mode 100644
index 000000000000..49dea3eac36f
--- /dev/null
+++ b/hudi-ai-operator/skills/hudi-optimizer/references/observation-catalog.md
@@ -0,0 +1,910 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Observation catalog
+
+Thirty rows describing a Hudi deployment that **works, but could work
better**. Nothing here is a
+failure. A row fires on a job that completes, a table that is consistent, and
a service that is
+running — and says what it is costing.
+
+The sibling `hudi-rca` failure catalog answers "why did this break". This one
answers "why is this
+slower, larger or more expensive than it needs to be". Sixteen rows are the
healthy-side reading of
+an RCA pattern already tagged `alsoAppliesWhenHealthy`; the rest have no
failure analogue at all.
+
+## How to read a row
+
+| Field | Meaning |
+|---|---|
+| **Costs** | What the user pays today, in terms they care about — wall clock,
storage, request count, query latency |
+| **Signals** | The exact field in one of the three JSON outputs. If a signal
does not name a field, it is not a signal |
+| **Corroboration** | Which of the three inputs must agree. **Never act on
one** |
+| **Change** | The `hoodie.*` or `spark.*` key, and the **direction** to move
it |
+| **Risk** | `safe` · `operational` · `durable` |
+| **Payoff** | What to expect, and when to expect nothing |
+| **Verify** | The measurement that proves it worked |
+
+### The three inputs
+
+1. **`props`** — the writer properties (`--props` / `--hoodie-conf`). 237 of
939 configs are
+ performance levers and 93 of those are conditionally applied, so a
recommendation made without the
+ config may be silently inert.
+2. **`eventlog`** — `HoodieSparkEventLogAnalyzer` JSON. Fields below are JSON
paths into it.
+3. **`table`** — the table itself, read through `HoodieTableHealthChecker` and
+ `HoodieTableLayoutAnalyzer` JSON, plus the timeline.
+
+**A row with one input agreeing is a hypothesis, not an observation.** Say
which inputs you had.
+
+### Risk tiers
+
+| Tier | Meaning |
+|---|---|
+| `safe` | A reversible config change with no effect on what is on storage.
Set it, run, set it back |
+| `operational` | Changes what a table service does. Reversible, but you need
a service cycle to see the result, and the intermediate state is on storage |
+| `durable` | **Changes what is on storage in a way that is hard or impossible
to undo.** Table type, partitioning, key generator, index type, record-key
encoding |
+
+**No row in this catalog is `durable`.** That is deliberate: a durable change
is a design decision,
+not an optimization, and belongs to `hudi-architect` with an ADR behind it.
Durable candidates live
+in `capability-catalog.md`, clearly marked, so the refusal has somewhere to
point.
+
+### Derived-expectation checks are the backbone
+
+Most rows are not absolute thresholds. They compare **what the configuration
implies** against **what
+was measured**, which needs no history and works on a single run. When a row
reads "config says X,
+measurement says Y", that is the whole mechanism — and it is why `props` is
mandatory rather than
+nice to have.
+
+### Carry the analyzer's caveats
+
+Two caveats from `HoodieSparkEventLogAnalyzer` govern every row that cites
`phaseSummary`:
+
+- **A phase share names where work was *forced*, not where it belongs.** Hudi
sets the job
+ description before the Spark action and Spark is lazy, so a bucket routinely
carries the cost of an
+ untagged step. Read `caveats[]` in the JSON and repeat what it says.
+- **With `hoodie.metadata.streaming.write.enabled=true` — the Spark default —
a large share of the
+ DAG is unattributable.** On a measured run, `UNKNOWN` was 60.5% of stage
wall clock. When `UNKNOWN`
+ is large, you are reasoning over a partial picture; say so instead of
dividing the remainder.
+
+---
+
+## Index A — by theme
+
+| Theme | Rows |
+|---|---|
+| File sizing and layout | OBS-001, OBS-002, OBS-003, OBS-004, OBS-005 |
+| Parallelism and skew | OBS-006, OBS-007, OBS-008, OBS-009, OBS-010 |
+| Index cost | OBS-011, OBS-012, OBS-013, OBS-014 |
+| Table service cadence | OBS-015, OBS-016, OBS-017, OBS-018, OBS-019 |
+| Timeline and metadata table | OBS-020, OBS-021, OBS-022, OBS-023 |
+| Query-side | OBS-024, OBS-025, OBS-026 |
+| Write path and driver | OBS-027, OBS-028, OBS-029, OBS-030 |
+
+## Index B — rows seeded from an RCA pattern
+
+| Observation | RCA pattern | What changes on the healthy side |
+|---|---|---|
+| OBS-006, OBS-011 | HUDI-RCA-014 | The same parallelism and skew analysis,
run before the OOM |
+| OBS-007, OBS-027 | HUDI-RCA-015 | Spill and parallelism on a slow but
succeeding upsert |
+| OBS-012 | HUDI-RCA-016 | File-group consolidation is an optimization long
before it is a fix |
+| OBS-004 | HUDI-RCA-017 | Pinning the record-size estimate stabilizes file
sizing on a wide schema |
+| OBS-015, OBS-016, OBS-017 | HUDI-RCA-018 | Service cadence tuning, the
archetypal healthy-table optimization |
+| OBS-029 | HUDI-RCA-019 | Poll pacing against a shard quota on a healthy
stream |
+| OBS-020 | HUDI-RCA-020 | A forgotten savepoint, found long before it becomes
a 413 |
+| OBS-021, OBS-028 | HUDI-RCA-021 | File-system-view and compaction sizing,
before the wedge |
+| OBS-018 | HUDI-RCA-022 | Clustering cadence against partition creation rate |
+| OBS-022 | HUDI-RCA-024 | Listing reduction on any healthy job at scale |
+| OBS-030 | HUDI-RCA-031 | `DIRECT` markers as a standing choice on a busy
driver |
+| OBS-019 | HUDI-RCA-037 | The cleaner that is correctly doing nothing — and
what to stop tuning |
+| OBS-023 | HUDI-RCA-006 | Lock wait tuning on a merely-slow multi-writer
setup |
+| OBS-026 | HUDI-RCA-038 | A pipeline whose cost is dominated by polling an
empty source |
+| OBS-005 | HUDI-RCA-039 | Benign spurious-file guard firings as a sign of
wasted task retries |
+
+The fourteen remaining rows (OBS-001, 002, 003, 008, 009, 010, 013, 014, 024,
025) have no failure
+analogue — nothing breaks, ever, which is exactly why nobody finds them.
+
+---
+
+# File sizing and layout
+
+## OBS-001 — Base files are well under the configured target size
+
+- **Costs:** More files per query means more footer reads, more object-store
requests, more Spark
+ task overhead, and a larger metadata table. Read latency grows roughly with
file count, and the
+ effect compounds on every downstream consumer of the table.
+- **Signals:**
+ - `layout.tableSizeStats.p50` and `.mean` well below the configured
`hoodie.parquet.max.file.size`
+ (default 120 MB). A p50 under a third of target is the threshold worth
acting on.
+ -
`layout.tableCharacteristics.detectors[name=small-files].effectiveConfigs["observed.flagged.pct"]`
+ — the share of qualifying partitions averaging under the detector
threshold.
+ - `layout.partitions[].sizeStats.p50` per partition, so you can tell a
table-wide problem from one
+ bad partition.
+- **Corroboration:** `table` + `props`. The layout numbers alone cannot say
whether small files are a
+ misconfiguration or a deliberate choice; the configured target is what makes
it an observation.
+- **The derived-expectation check:** `hoodie.parquet.max.file.size` says what
a file should grow to.
+ `hoodie.parquet.small.file.limit` says what counts as small enough to append
to. **If
+ `small.file.limit` is below the average size of files your batches produce,
small-file handling
+ never engages** — every batch writes new files and none of them ever grow.
+- **Change:** Raise `hoodie.parquet.small.file.limit` toward (but below)
+ `hoodie.parquet.max.file.size`; the default is 100 MB against a 120 MB
target. Where the writer
+ cannot bin-pack enough, schedule clustering (OBS-003).
+- **Risk:** `safe` for the two sizing configs.
+- **Payoff:** Fewer, larger files on subsequent writes. **Existing small files
are not touched** —
+ these configs govern new writes only, which is why clustering is the
companion move.
+- **Verify:** Re-run the layout analyzer after several commits and compare
`tableSizeStats.p50`. The
+ number should climb; if it does not, the limit is still below your
per-batch-per-partition volume.
+
+## OBS-002 — Small-file handling is configured but inert for this batch size
+
+- **Costs:** The same as OBS-001, with the added cost of a user who believes
the problem is handled.
+- **Signals:**
+ - `props` has a non-default `hoodie.parquet.small.file.limit`, **and**
+ - `layout.tableSizeStats.p50` is still far below
`hoodie.parquet.max.file.size`, **and**
+ - `eventlog.phaseSummary[phase=DATA_TABLE_WRITE]` contains a stage with
+ `subPhase: "smallFileProbe"` that runs — so the probe is happening and
finding nothing.
+- **Corroboration:** `props` + `table`, with `eventlog` confirming the probe
ran at all.
+- **The derived-expectation check:** small-file handling appends to an
existing file only when that
+ file is under `small.file.limit` **and** the incoming records for that
partition fit. A workload
+ writing a few MB per partition per commit into a partition whose files are
already at 110 MB will
+ correctly decline to append, forever. The config is live; the condition
never holds.
+- **Change:** This is the row where the answer may be "the config is right and
the batch is the
+ problem". Either increase bytes per commit per partition (fewer, larger
commits), or accept that
+ clustering is the consolidation mechanism for this shape and schedule it
(OBS-003).
+- **Risk:** `safe` to re-check; `operational` if the answer is clustering.
+- **Payoff:** Mostly diagnostic. Its value is stopping a user from tuning a
lever that is already
+ doing what it is told.
+- **Verify:** Count `smallFileProbe` stages before and after. If the probe
still runs and file sizes
+ still do not move, the batch shape is the binding constraint, not the config.
+
+## OBS-003 — Clustering has never been scheduled on a table that would benefit
+
+- **Costs:** Small files accumulate with no mechanism to consolidate them, and
data with no locality
+ means no useful column-stats pruning. Both get worse monotonically.
+- **Signals:**
+ - No `.replacecommit` from clustering anywhere on the timeline (`table`),
**and**
+ - `layout.tableCharacteristics.detectors[name=small-files].status` is
`MODERATE` or `SEVERE`,
+ **and**
+ - `props` has `hoodie.clustering.inline=false` and
`hoodie.clustering.async.enabled=false` (both
+ default `false`), so nothing is going to schedule it.
+- **Corroboration:** `table` + `props`. All three when the event log also
shows no `CLUSTERING`
+ operation in `operationSummary[]`.
+- **Change:** Enable one scheduling path, not both:
+ - `hoodie.clustering.inline=true` with
`hoodie.clustering.inline.max.commits` for small tables
+ where a periodic pause is acceptable;
+ - `hoodie.clustering.async.enabled=true` with
`hoodie.clustering.async.max.commits` where
+ ingestion latency matters.
+ - Bound the first plan: `hoodie.clustering.plan.strategy.max.num.groups`
**down** from its default
+ and `hoodie.clustering.plan.strategy.max.bytes.per.group` sized to what
one round can execute.
+ Starting unbounded on a table with a long backlog is how HUDI-RCA-022
happens.
+- **Risk:** `operational`. Clustering writes replacecommits and rewrites file
groups; reversible in
+ the sense that you can stop scheduling it, but the rewritten files stay.
+- **Payoff:** File count falls and `tableSizeStats.p50` rises over several
rounds. With
+ `hoodie.clustering.plan.strategy.sort.columns` set to a frequently-filtered
column, column stats
+ become selective and query pruning follows (see OBS-024).
+- **Verify:** `layout.totalFiles` falls and `tableSizeStats.p50` rises round
over round. In the event
+ log, `operationSummary[operation=CLUSTERING]` appears with `PLANNING` and
`EXECUTION` phases, and
+ `EXECUTION` should dominate — planning dominating means the plan is too
large for the round.
+
+## OBS-004 — Record-size estimation is unstable on a wide schema
+
+- **Costs:** File sizes swing commit to commit because the estimate driving
row-group sizing is
+ wrong. You get a mix of undersized and oversized files from an unchanged
config.
+- **Signals:**
+ - `layout.tableSizeStats` shows a wide `min`-to-`max` spread with no
matching spread in records
+ written per commit (`table`, commit metadata).
+ - Schema contains large string/JSON columns (`table`).
+ - `props` leaves `hoodie.copyonwrite.record.size.estimate` at its 1024
default.
+- **Corroboration:** `table` + `props`.
+- **Change:** Pin the estimate rather than computing it:
`hoodie.copyonwrite.record.size.estimate`
+ **down**, well below the default, for a wide-payload table.
**Counter-intuitive** — the configured
+ value is deliberately lower than the real record size, and that is the
point: it makes Hudi's
+ sizing arithmetic conservative instead of letting a bad estimate drive it.
+- **Risk:** `safe`.
+- **Payoff:** File sizes stop swinging. This is the healthy-side reading of
HUDI-RCA-017, where the
+ same instability eventually throws a Parquet encoder assertion.
+- **Verify:** The `max`/`min` ratio in `layout.tableSizeStats` narrows over
subsequent commits.
+
+## OBS-005 — Task retries are producing spurious files the commit guard then
catches
+
+- **Costs:** Every retried task writes a file that is later discarded. You pay
the write, the storage
+ until cleaning, and occasionally a failed commit that the next run fixes.
+- **Signals:**
+ - `eventlog.stages[].observed.tasksFailed` or `.tasksSpeculative` greater
than zero on
+ `DATA_TABLE_WRITE` stages.
+ - `eventlog.stages[].taskFailureReasons` showing retries rather than a
single terminal failure.
+ - `props` has `spark.speculation=true`.
+- **Corroboration:** `eventlog` + `props`. `table` corroborates if commits
occasionally abort with
+ the spurious-data-files guard.
+- **Change:** Turn `spark.speculation` **off** for Hudi write jobs.
Speculative execution against a
+ writer that creates files outside Spark's output committer produces
duplicate output that Hudi's
+ marker reconciliation must then discard. If retries are not speculative,
chase the task failures
+ instead — they are the real signal.
+- **Risk:** `safe`.
+- **Payoff:** Wasted write work disappears. On a job with heavy speculation
this can be a visible
+ share of `DATA_TABLE_WRITE`.
+- **Verify:** `tasksSpeculative` returns to zero; the guard stops firing.
+- **Do not over-read this one.** An occasional spurious-file abort that the
next run clears is benign
+ (HUDI-RCA-039). It becomes an observation only when retries are routine.
+
+---
+
+# Parallelism and skew
+
+## OBS-006 — A stage is under-parallelised against the shuffle config
+
+- **Costs:** Wall clock. A stage with twelve tasks and forty minutes of work
is forty minutes you
+ cannot shorten by adding executors.
+- **Signals:**
+ - `eventlog.stages[].observed.numTasksPlanned` low relative to
+ `eventlog.application.observed.peakConcurrentExecutors` — tasks fewer than
available slots means
+ idle capacity.
+ - `eventlog.stages[].observed.taskDurationMillis.p50` large and `skewRatio`
near 1.0 — uniformly
+ long tasks, which is the signature of too few of them rather than skew.
+ - `props`: `hoodie.upsert.shuffle.parallelism` /
`hoodie.insert.shuffle.parallelism` /
+ `hoodie.bulkinsert.shuffle.parallelism` /
`hoodie.delete.shuffle.parallelism`.
+- **Corroboration:** `eventlog` + `props`. The event log says how many tasks
ran; only the config
+ says whether that number was chosen or inherited.
+- **The derived-expectation check, and the thing most advice gets wrong:**
these configs **default to
+ `0`, not 200**. `0` means "inherit the input RDD's partition count". So a
stage with twelve tasks
+ on a default config is reporting the source's partitioning, not a Hudi
setting — and raising the
+ Hudi config is the second move, not the first. **Fix the input partitioning
first**, because the
+ tagging and write stages inherit it.
+- **Change:** Raise the source's partition count first (for Kafka,
`hoodie.streamer.source.kafka.minPartitions`).
+ Then set the relevant `hoodie.*.shuffle.parallelism` explicitly, **up**,
toward roughly two to
+ three times the executor-core count.
+- **Risk:** `safe`.
+- **Payoff:** Proportional to the gap between task count and available slots.
Nothing, if the stage
+ was already spreading across every slot.
+- **Verify:** `numTasksPlanned` rises, stage `durationMillis` falls, and
`skewRatio` does not rise. If
+ duration does not fall, the stage was not parallelism-bound.
+- **Note the exception:** the data-table write stage's task count is
**bucket-shaped** — it follows
+ the file groups being written, not the shuffle config. Tuning
+ `hoodie.upsert.shuffle.parallelism` does not move it. Check `subPhase`
before concluding.
+
+## OBS-007 — Stages spill to disk under a healthy-looking job
+
+- **Costs:** Spill is work done twice. The job completes, so nothing alerts,
but the stage pays
+ serialization, disk write and disk read for data that should have stayed in
memory.
+- **Signals:**
+ - `eventlog.stages[].observed.memoryBytesSpilled` and `.diskBytesSpilled`
greater than zero.
+ - `eventlog.application.observed.memoryBytesSpilled` as the
application-level total.
+ - Cross-reference `eventlog.stages[].phase` — spill in
`DEDUP_AND_INDEX_TAGGING` and spill in
+ `DATA_TABLE_WRITE` have different remedies.
+- **Corroboration:** `eventlog` + `props`. Which config to move depends
entirely on the phase, and
+ the phase is only trustworthy read against the config.
+- **Change:**
+ - Spill in `DEDUP_AND_INDEX_TAGGING`: raise the relevant shuffle parallelism
(OBS-006) so each task
+ handles less.
+ - Spill in `DATA_TABLE_WRITE` on a MOR table: raise
`hoodie.memory.merge.max.size`, which governs
+ the in-memory map used for log merge, and `spark.executor.memoryOverhead`.
+ - Spill under GC pressure rather than hard memory limits: **lower**
`spark.memory.fraction` and
+ `spark.memory.storageFraction`. **Counter-intuitive** — giving Spark's
managed regions less
+ leaves more heap for the user objects Hudi's merge path actually allocates.
+- **Risk:** `safe`.
+- **Payoff:** Removing spill from a stage that was spilling heavily is often
the single largest
+ cheap win available on a healthy job.
+- **Verify:** `memoryBytesSpilled` and `diskBytesSpilled` fall to zero on the
target stage while
+ `durationMillis` falls. If spill falls but duration does not, the stage was
not spill-bound.
+
+## OBS-008 — A stage is skewed and the job absorbs it
+
+- **Costs:** The stage runs as long as its slowest task. Every other executor
idles. Measured on a
+ real log: a small-file-probe stage at 56.7% of wall clock with `p50=17ms`
and `max=3m00s` — one
+ pathological task out of a thousand.
+- **Signals:**
+ - `eventlog.stages[].observed.skewRatio` — max task duration over p50. Above
~10x is worth a look;
+ above 100x is pathological. **`-1` means undefined, not zero.**
+ - `eventlog.stages[].observed.taskDurationMillis.p95` against `.max` — a
`max` far above `p95` is
+ one straggler; `p95` near `max` is a broad tail.
+ - `eventlog.stages[].observed.shuffleReadBytes` with the spread across tasks.
+ - `eventlog.stages[].subPhase` to know which component to attribute it to.
+- **Corroboration:** `eventlog` + `table`. Skew in the write path is usually
partition skew, and
+ `layout.skew.cv` / `.gini` / `.largestPartitionShare` is where you confirm
it.
+- **Change:** Depends on where it is, and the diagnosis must come first:
+ - Partition skew (`layout.skew.largestPartitionShare` high): this is a
layout property. Clustering
+ helps within partitions; across them it is a partitioning question, which
is `durable` and goes to
+ `hudi-architect`.
+ - Bloom-index skew with random keys:
`hoodie.bloom.index.fileid.key.sorting.enable=true` **and**
+ `hoodie.bloom.index.bucketized.checking=false`. **Both are required** —
bucketized checking stays
+ on unless explicitly disabled, so setting only the first changes nothing.
+ - A single straggler on a small-file probe is usually one partition with a
very large number of
+ base files; fix that partition's file count (OBS-003).
+- **Risk:** `safe` for the index configs; `operational` for clustering.
+- **Payoff:** Bounded by the gap between the stage's duration and its p50
times task count over slot
+ count. Compute that before promising anything.
+- **Verify:** `skewRatio` falls on the target stage. Watch that it does not
simply move to the next
+ stage downstream.
+
+## OBS-009 — Many stages are over-parallelised for the work they do
+
+- **Costs:** Scheduling overhead, one output file per task on shuffle write,
and a map-status
+ structure that grows with partition count. The pathological end of this is
HUDI-RCA-016, where the
+ broadcast becomes unfetchable — but long before that it is simply waste.
+- **Signals:**
+ - `eventlog.stages[].observed.numTasksPlanned` high while
`taskDurationMillis.p50` is tiny (tens of
+ milliseconds) — a thousand tasks each doing nothing.
+ - `eventlog.application.observed.criticalPathMillis` far below
`application.observed.durationMillis`
+ — the gap is scheduling and job overhead, not work.
+ - `props` shows an explicitly-set high `hoodie.simple.index.parallelism` or
a shuffle parallelism
+ set well above the executor-core count.
+- **Corroboration:** `eventlog` + `props`.
+- **Change:** **Reduce** the offending parallelism config toward two to three
times the executor-core
+ count. For `hoodie.simple.index.parallelism` specifically, reducing it is
also the remedy for the
+ broadcast failure — **raising it makes that failure worse**, which is
exactly where an operator
+ reaching for the obvious lever ends up.
+- **Risk:** `safe`.
+- **Payoff:** Recovers the overhead gap. On a job whose critical path is a
small fraction of its
+ duration, this can be a large share.
+- **Verify:** The gap between `criticalPathMillis` and `durationMillis`
narrows; stage count and task
+ count fall without stage durations rising.
+
+## OBS-010 — The job's wall clock is dominated by scheduling, not work
+
+- **Costs:** Executors sit idle between stages. You are paying for a cluster
that is coordinating
+ rather than computing.
+- **Signals:**
+ - `eventlog.application.observed.criticalPathMillis` divided by
`.durationMillis` well below 1.0.
+ The critical path is a **floor** on wall clock — the gap is everything
else.
+ - `eventlog.application.observed.executorRunTimeMillis` low relative to
+ `durationMillis × peakConcurrentExecutors` — low cluster utilisation.
+ - `eventlog.application.observed.jobCount` and `.stageCount` high for the
volume of data moved.
+- **Corroboration:** `eventlog` + `props`. Many small jobs on a Hudi write
usually means many small
+ commits, which is a writer-cadence question the config answers.
+- **Change:** Fewer, larger batches at the source. On the streamer path, raise
the per-sync batch
+ size so each sync does more work. There is no single Hudi config for this —
it is the source's
+ pacing, and the right lever is source-specific.
+- **Risk:** `safe`, but it trades latency for throughput, so it is a product
decision rather than a
+ pure win. Say that.
+- **Payoff:** Fewer commits also means a shorter timeline, a smaller metadata
table, and less
+ archival work — the benefit compounds beyond the job.
+- **Verify:** `criticalPathMillis / durationMillis` rises toward 1.0 and
`jobCount` falls for the same
+ volume.
+- **Caveat:** `criticalPathMillis` is the longest stage per job, summed, which
assumes jobs do not
+ overlap. Async table services do overlap, so on a table running async
compaction or clustering this
+ ratio reads lower than reality. Check `operationSummary[]` for concurrent
operations first.
+
+---
+
+# Index cost
+
+## OBS-011 — Index tagging is a large share of wall clock
+
+- **Costs:** On a SIMPLE index, tagging cost scales with **table size, not
batch size** — so this
+ share grows as the table grows even when the workload does not. It is the
clearest case of a cost
+ that is invisible until you measure it.
+- **Signals:**
+ -
`eventlog.phaseSummary[phase=DEDUP_AND_INDEX_TAGGING].shareOfStageWallClock` —
above ~0.3 is
+ worth investigating.
+ - `eventlog.stages[].subPhase` within that bucket: `bloomIndexLookup`,
`recordIndexLookup` or
+ `workloadProfile`. **This distinction is load-bearing** — on a measured
log, isolating
+ `recordIndexLookup` separated 3.6s of real index work from the profile
stages it was pooled with.
+ - `eventlog.stages[].observed.numTasksPlanned` on those stages against
+ `layout.totalFiles` — a task count tracking the **base-file count** rather
than the batch size is
+ the smoking gun for a SIMPLE index.
+ - `props`: `hoodie.index.type`.
+- **Corroboration:** All three. The phase share needs the config to say which
index produced it and
+ the table to say what the cost scales with.
+- **Change:** Within the current index type: raise
`hoodie.simple.index.parallelism` if tasks are few
+ and long (OBS-006), or `hoodie.bloom.index.parallelism` and lower
+ `hoodie.bloom.index.keys.per.bucket` on a bloom index. **Lowering keys per
bucket is
+ counter-intuitive** — you shrink the per-task batch to produce more, smaller
units of work. Raising
+ it makes memory pressure worse.
+- **Risk:** `safe`.
+- **Payoff:** Within an index type, modest. **If tagging dominates and the
task count tracks
+ file count rather than batch size, the real answer is a different index** —
see
+ `capability-catalog.md`, CAP-001. Changing `hoodie.index.type` is `durable`
and is not this
+ catalog's to recommend.
+- **Verify:** The `DEDUP_AND_INDEX_TAGGING` share falls. Re-measure after the
table has grown, not
+ just after the config change — that is the test of whether the cost still
scales with size.
+- **Caveat:** This bucket contains deduplication too. `BaseWriteHelper` tags
only `Tagging:` and
+ leaves `deduplicateRecords` untagged, so both land together. A large share
may be dedup, not the
+ index. `bundledSteps` on the stage says when the tool thinks this is
material.
+- **Caveat, SIMPLE index:** the analyzer has no `simpleIndexLookup` sub-phase.
`HoodieSimpleIndex`
+ builds its join through generic `HoodieJavaRDD` / `HoodieJavaPairRDD` call
sites, so its stages carry
+ whatever job description was current — `workloadProfile` or `tagging` — and
a reading keyed on
+ sub-phase names under-counts the index. Recognise it by the measurement: on
a replayed 250-file table
+ the three simple-index stages each planned exactly 250 tasks and were 51% of
stage wall clock,
+ labelled `workloadProfile`. A task count equal to `layout.totalFiles` inside
this bucket is the
+ simple index, whatever the label says.
+
+## OBS-012 — File-group count is growing without bound
+
+- **Costs:** Simple-index tagging and write profiling fan out with file-group
count, so both get
+ slower as file groups accumulate. The shuffle map-status broadcast grows
with them. Nothing fails
+ until it does.
+- **Signals:**
+ - `layout.totalFiles` rising across runs with `layout.tableSizeStats.mean`
flat or falling — new
+ file groups rather than growing ones.
+ - `layout.numPartitions` against `layout.fileCountPerPartition.mean`.
+ - `layout.tableCharacteristics.detectors[name=micro-partition].status` is
`FLAGGED`, with
+ `effectiveConfigs["observed.size.rule.triggered"]` true.
+ - `eventlog.stages[].observed.numTasksPlanned` on `workloadProfile`
sub-phase stages tracking that
+ count.
+- **Corroboration:** `table` + `eventlog`. `props` third, to check whether
anything is scheduled to
+ consolidate them.
+- **Change:** Consolidation, not parallelism. Schedule clustering (OBS-003)
with
+ `hoodie.clustering.plan.strategy.small.file.limit` set so small groups are
picked up and
+ `hoodie.clustering.plan.strategy.target.file.max.bytes` set to the size you
want out. On MOR,
+ confirm compaction is keeping up (OBS-015) — uncompacted file slices count
too.
+- **Risk:** `operational`.
+- **Payoff:** Tagging and profiling cost stop growing with the table. The
benefit is mostly in
+ avoided future cost, which makes it easy to defer and expensive to have
deferred.
+- **Verify:** `layout.totalFiles` flattens or falls while total bytes keep
growing. That divergence is
+ the signal.
+
+## OBS-013 — Column stats are maintained but never used
+
+- **Costs:** You pay to compute and store column statistics on every write and
get no pruning back.
+ On Spark the column-stats index is **on by default**, so this is the common
case, not an exotic
+ one.
+- **Signals:**
+ - `props`: `hoodie.metadata.index.column.stats.enable` is on — and note the
**engine-effective**
+ default is `true` on Spark even though the declared default is `false`.
Reading the declared
+ default here gives the wrong answer about every default Spark table.
+ - `props`: `hoodie.enable.data.skipping` on the **read** side. If readers do
not set it, nothing
+ consumes the stats.
+ - `table`: a `column_stats` partition exists under the metadata table.
+ - `eventlog.phaseSummary[phase=METADATA_TABLE_WRITE].shareOfStageWallClock`
— the write-side cost
+ you are paying for them.
+- **Corroboration:** `props` + `table`. The read-side config is the decisive
half and is frequently
+ not in the writer properties at all — **say so rather than assuming**.
+- **Change:** Two directions, and the measurement decides:
+ - If queries filter on columns the stats cover: ensure readers set
`hoodie.enable.data.skipping=true`,
+ and give the data locality to prune on with
+ `hoodie.clustering.plan.strategy.sort.columns` set to those columns.
Unsorted data makes every
+ file's min/max span the whole range and pruning finds nothing.
+ - If no query filters on anything the stats cover: narrow them with
+ `hoodie.metadata.index.column.stats.column.list` rather than disabling the
index, so you keep
+ pruning where it pays.
+- **Risk:** `safe` for the read config; `operational` for clustering and for
narrowing the column
+ list (the index is rebuilt).
+- **Payoff:** Either queries get faster or metadata-table write cost falls.
Both are real; which one
+ depends on the query evidence, which you usually do not have — **ask for it
rather than guessing**.
+- **Verify:** With skipping on and data sorted, a filtered query reads fewer
files. Without a query
+ plan you cannot verify this, so say that it is unverified.
+
+## OBS-014 — Partition stats are enabled by inheritance rather than intent
+
+- **Costs:** Partition-level statistics are derived from column stats and
written on every commit.
+ On a table with few partitions they prune almost nothing.
+- **Signals:**
+ - `props`: `hoodie.metadata.index.partition.stats.enable`. **It has no
independent effect** —
+ partition stats require `hoodie.metadata.index.column.stats.enable` and
are inert without it.
+ - `layout.numPartitions` small (single digits, or unpartitioned) —
partition-level pruning cannot
+ help when there is nothing to prune between.
+ - `eventlog.phaseSummary[phase=METADATA_TABLE_WRITE]` for the cost.
+- **Corroboration:** `props` + `table`.
+- **Change:** On a table with few partitions where queries do not filter on
the partition key, turn
+ `hoodie.metadata.index.partition.stats.enable` **off**. Leave it on where
partition count is high
+ and queries filter on partition columns.
+- **Risk:** `operational` — the index partition is dropped and rebuilt if you
turn it back on.
+- **Payoff:** Small but free. The reason to include it is that it is a lever
users do not know they
+ are pulling, since it rides on the column-stats default.
+- **Verify:** `METADATA_TABLE_WRITE` share falls slightly; the
`partition_stats` partition disappears
+ from the metadata table.
+
+---
+
+# Table service cadence
+
+## OBS-015 — Compaction trails the configured trigger without having stopped
+
+- **Costs:** Read-side merge cost on every query of an uncompacted file slice,
growing log files, and
+ a rollback surface that widens with each uncompacted slice. The table works;
it just works harder
+ every day.
+- **Signals:**
+ -
`health.checks[name=compaction].effectiveConfigs["observed.delta.commits.since.last.compaction"]`
+ against `["effective.unhealthy.threshold.delta.commits"]`. **Read the
numbers, not the verdict** —
+ the health checker applies a 2.0x slack factor specifically so it only
flags a service that has
+ *stopped*. An observation fires in the band between the configured trigger
and that threshold,
+ where the checker correctly reports `HEALTHY`.
+ - `props`: `hoodie.compact.inline.max.delta.commits` (or
`.max.delta.seconds` under the TIME
+ strategy), and `hoodie.compact.inline.trigger.strategy`.
+ - `eventlog.operationSummary[operation=COMPACTION]` — whether compaction ran
in this window and how
+ `PLANNING` compared with `EXECUTION`.
+- **Corroboration:** `table` + `props`. The accumulated count means nothing
without the trigger.
+- **The derived-expectation check:** the trigger says compaction should happen
every N delta commits.
+ If 2N have accumulated, compaction is running at half the configured rate.
That is an observation
+ long before the health checker's threshold.
+- **Change:** Either compact more often (**lower**
`hoodie.compact.inline.max.delta.commits`) or give
+ each round more capacity (**raise** `hoodie.compaction.target.io`). Which
one depends on
+ `EXECUTION` duration: short executions that are too rare want a lower
trigger; long executions that
+ do not finish the backlog want more target IO. **Raise target IO
substantially rather than nudging
+ it** — if only half the candidate file groups fit in a round, a tail
accumulates forever.
+- **Risk:** `operational`.
+- **Payoff:** Query-side merge cost falls and stops growing. On a MOR table
under steady updates this
+ is usually the highest-value service tuning available.
+- **Verify:** `observed.delta.commits.since.last.compaction` stabilises near
the configured trigger
+ rather than drifting up. In the event log, `EXECUTION` should dominate
`PLANNING` within the
+ `COMPACTION` operation.
+
+## OBS-016 — The cleaner trails its trigger while still running
+
+- **Costs:** Storage for file versions past the retention you asked for, plus
a timeline that cannot
+ archive past the cleaner's `earliestCommitToRetain`.
+- **Signals:**
+ - `health.checks[name=cleaner]` — again the numbers, not the verdict. The
checker multiplies the
+ trigger by 2.0 **and floors at 5 commits**, so a cleaner several commits
behind reports healthy.
+ - `props`: `hoodie.clean.trigger.max.commits`, `hoodie.cleaner.policy`,
+ `hoodie.cleaner.commits.retained`.
+ -
`health.checks[name=cleaner].effectiveConfigs["observed.last.clean.earliest.commit.retained"]`
—
+ the retention watermark the last clean reached — and the active instant
count.
+- **Corroboration:** `table` + `props`.
+- **Change:** `hoodie.clean.automatic` on (its default) with
+ `hoodie.clean.trigger.max.commits` tuned to your commit rate; or
`hoodie.clean.async=true` where a
+ synchronous clean is costing ingestion latency. Raise
`hoodie.cleaner.parallelism` if the clean
+ itself is slow rather than infrequent.
+- **Risk:** `operational`.
+- **Payoff:** Storage reclaimed on the retention you configured, and archival
unblocked behind it.
+- **Verify:** `observed.last.clean.earliest.commit.retained` advances; active
instant count stops
+ growing.
+- **Read OBS-019 first.** A cleaner that appears behind is very often a
cleaner correctly finding
+ nothing to do, and changing retention in response is how a non-incident
becomes an incident.
+
+## OBS-017 — Archival trails the configured retention
+
+- **Costs:** Every timeline load parses more instants. The cost is paid by
every write, every table
+ service and every reader — it is the most broadly-felt slow degradation in
Hudi.
+- **Signals:**
+ - `health.checks[name=archival].effectiveConfigs` — the observed active
instant count against
+ `hoodie.keep.max.commits`. The checker allows 1.1x slack and an absolute
ceiling of 5000; an
+ observation fires between the configured retention and that slack.
+ - `props`: `hoodie.keep.min.commits`, `hoodie.keep.max.commits`,
+ `hoodie.commits.archival.batch`, `hoodie.archive.automatic`.
+ - `eventlog.phaseSummary[phase=ARCHIVAL]` — **archival is driver-dominated,
so its presence with
+ non-trivial duration is itself the signal.** Absence tells you nothing.
+- **Corroboration:** `table` + `props`.
+- **Change:** Check the chain before touching archival config: **archival
silently overrides
+ `hoodie.keep.min.commits`** when the cleaner's `earliestCommitToRetain`
demands more retention.
+ "I set min commits to 20 and have 400 instants" is almost always the cleaner
holding the line —
+ **fix the cleaning config, not the archival one** (OBS-016). Once the
cleaner is right, lower
+ `hoodie.commits.archival.batch` if archival itself is heavy.
+- **Risk:** `operational`.
+- **Payoff:** Timeline load time is a fixed tax on everything. Removing it
helps uniformly.
+- **Verify:** Active instant count falls toward `hoodie.keep.max.commits` and
stays there.
+- **Check OBS-020 first.** A savepoint pins archival regardless of
configuration, and tuning archival
+ with a savepoint in place prunes nothing while costing object-store requests.
+
+## OBS-018 — Clustering cadence is behind the partition-creation rate
+
+- **Costs:** Each round has more to do than the last. The plan grows, and a
plan large enough becomes
+ its own problem — it is re-read and fully deserialised on every timeline
refresh.
+- **Signals:**
+ - `table`: a pending `.replacecommit.requested` whose **plan file size** is
more than a few MB.
+ This is the giveaway, and it is cheap to check.
+ - `layout.numPartitions` growing faster than clustering replacecommits
complete (`table`).
+ - `props`: `hoodie.clustering.plan.strategy.max.num.groups`,
+ `hoodie.clustering.plan.strategy.max.bytes.per.group`,
+ `hoodie.clustering.inline.max.commits` or
`hoodie.clustering.async.max.commits`.
+ - `eventlog.operationSummary[operation=CLUSTERING]` with `PLANNING` share
exceeding `EXECUTION` —
+ the planner is working harder than the executor, which is backwards.
+- **Corroboration:** `table` + `props`.
+- **Change:** Increase clustering frequency (**lower** the max-commits
trigger) and **cap each plan**
+ via `hoodie.clustering.plan.strategy.max.num.groups` and
+ `hoodie.clustering.plan.strategy.max.bytes.per.group`. On a date-partitioned
table,
+ `hoodie.clustering.plan.strategy.daybased.lookback.partitions` bounds the
candidate set directly.
+- **Risk:** `operational`.
+- **Payoff:** Rounds stay bounded and predictable. Mostly avoided future cost
— the failure mode at
+ the end of this road is a multi-day outage (HUDI-RCA-022), which is a reason
to act early rather
+ than a reason to panic now.
+- **Verify:** Pending plan file size stays small, and `EXECUTION` dominates
`PLANNING` in
+ `operationSummary[operation=CLUSTERING]`.
+
+## OBS-019 — The cleaner is correctly doing nothing, and the config is being
tuned anyway
+
+- **Costs:** None directly. The cost is a user about to make a real change in
response to a
+ non-problem — **which is the most expensive thing in this catalog.**
+- **Signals:**
+ - `table`: few or no recent `.clean` instants, possibly with a growing
active timeline.
+ - **The decisive, cheap check:** in recent commit metadata compare
`numWrites` and `numInserts` per
+ file. **Equal means the write created a new file group** — there is no
older version to clean and
+ the cleaner is correct. Hudi also retains one prior version as a buffer,
so a file group with two
+ base files still has nothing to clean.
+ - `health.checks[name=cleaner]` reporting `HEALTHY`, with its findings
distinguishing "behind" from
+ "nothing to do".
+- **Corroboration:** `table` + `props`. All three if the event log shows no
`CLEANING` operation —
+ which, on its own, means nothing.
+- **Change:** **None.** Confirm it is benign and say so. The legitimate causes
are: append-only or
+ immutable data; a recent retention change producing a quiet window equal to
the difference;
+ clustering having just consolidated many file groups into few; or a MOR
table whose file groups are
+ only appended to — the cleaner cannot clean a group compaction has not yet
given a new version.
+- **Risk:** n/a.
+- **Payoff:** A change not made.
+- **Verify:** Nothing to verify. **One thing worth noting**: incremental clean
can permanently miss
+ partitions that received no new ingests, leaving old file versions
indefinitely. A periodic full
+ clean is the answer there, and that is a real recommendation rather than a
non-action.
+- **If clean genuinely is not running and a compaction or rollback is also
stuck**, that is the real
+ problem and it belongs to `hudi-rca`, not here.
+
+---
+
+# Timeline and metadata table
+
+## OBS-020 — A savepoint is quietly pinning the timeline
+
+- **Costs:** Archival stops at the **earliest** savepoint, so one forgotten
savepoint pins everything
+ after it. The timeline grows without bound, every timeline load gets slower,
and data files the
+ cleaner would release stay alive. Nothing errors while this happens.
+- **Signals:**
+ - `health.checks[name=savepoint]` — two signals: a savepoint further back
than the configured
+ archival retention (blocking now), and a savepoint older than 7 days
(likely forgotten).
+ - `props`: `hoodie.keep.max.commits`, and `hoodie.archive.beyond.savepoint`.
+ - `table`: the savepoint instant time against the active timeline's span.
+- **Corroboration:** `table` + `props`.
+- **Change:** Release the savepoint once whatever it was taken for is done
(`savepoint delete` in
+ `hudi-cli`). Archival resumes and the timeline shrinks on its own.
+- **Risk:** `operational` — **and genuinely consequential.** A savepoint
protects files from cleaning;
+ releasing one makes those files eligible for deletion. **Confirm with a
human that the savepoint is
+ no longer needed. Never recommend releasing one as a routine optimization.**
+- **Payoff:** Archival resumes and the timeline shrinks. On a long-pinned
table this can be thousands
+ of instants.
+- **Verify:** Active instant count falls after the next archival round.
+- **Do not reach for aggressive archival scheduling first.** With the
savepoint in place a more
+ frequent archival job prunes nothing and costs object-store requests. It is
pointless until the
+ savepoint is gone.
+
+## OBS-021 — The file-system view is on heap and the table has outgrown it
+
+- **Costs:** Driver heap pressure, GC time on the driver, and timeline-server
responses that get
+ slower as the view grows. On a large enough view the serialisation path hits
a structural ceiling
+ that more heap cannot move.
+- **Signals:**
+ - `eventlog.application.executorRemovalReasons` and driver-side GC — but
note the driver is **not**
+ in `peakConcurrentExecutors`, so the event log is weak evidence here.
+ - `eventlog.stages[].observed.jvmGcTimeMillis` and `.gcFractionOfRunTime`
above ~0.1 across many
+ stages. **`-1` means undefined.**
+ - `layout.totalFiles` and `layout.numPartitions` — view size tracks
file-group count.
+ - `props`: `hoodie.filesystem.view.type` (default `MEMORY`),
+ `hoodie.filesystem.view.spillable.mem`,
`hoodie.common.spillable.diskmap.type`.
+- **Corroboration:** `table` + `props`. The event log is corroborating, not
deciding.
+- **Change:** For a table with a very large view,
`hoodie.filesystem.view.type=SPILLABLE_DISK` with
+ `hoodie.filesystem.view.spillable.mem` sized around 1 GB and
+ `hoodie.common.spillable.diskmap.type=ROCKS_DB`.
+- **Risk:** `safe` — a writer-side config with no storage effect.
+- **Payoff:** Driver heap pressure drops on a large table. **Nothing, on a
small one.**
+- **Verify:** Driver GC fraction falls; timeline-server timeouts (OBS-030)
stop.
+- **Apply this per table, not globally, and note the direct tension:** on a
**long-lived driver
+ hosting many small tables**, `SPILLABLE_DISK` is the wrong answer — each
spillable map registers a
+ JVM shutdown hook and those accumulate across table-service cycles
(HUDI-RCA-023). A table with a
+ very large view needs spillable; a driver with many small tables does not.
+
+## OBS-022 — File listing is going to storage rather than the metadata table
+
+- **Costs:** Object-store list requests scale with partition count on every
operation. On a large
+ table this is both latency and a line on the storage bill.
+- **Signals:**
+ - `props`: `hoodie.metadata.enable`. Note the **engine-effective** default
is `true` on Spark and
+ Flink but `false` on the Java engine — reading the declared default gives
the wrong answer for a
+ Java-engine writer.
+ - `layout.mdtEnabled` — the layout analyzer reports whether it read through
the metadata table.
+ - `eventlog.stages[].jobDescription` containing `Parallel listing paths`, and
+ `eventlog.stages[].subPhase == "metadataListing"` for the metadata-table
side.
+ - `eventlog.stages[].observed.durationMillis` on those listing stages as the
direct cost.
+ - `props`: `hoodie.file.listing.parallelism`.
+- **Corroboration:** `props` + `eventlog`, with `table` confirming partition
count makes it matter.
+- **Change:** `hoodie.metadata.enable=true` where it is off and partition
count is non-trivial.
+- **Risk:** `operational` — the metadata table is bootstrapped on first
enable, which costs a cycle,
+ and it then has its own write cost on every commit (OBS-023).
+- **Payoff:** Listing cost becomes a metadata-table read instead of a storage
listing. Grows with
+ partition count; **negligible on a small or unpartitioned table, where the
metadata table's write
+ overhead may exceed what it saves.**
+- **Verify:** `Parallel listing paths` stages against the data table disappear
from the event log;
+ `layout.mdtEnabled` reads true.
+
+## OBS-023 — Metadata-table write cost is a large share of the commit
+
+- **Costs:** Every enabled index partition is written on every commit.
Enabling indexes you do not
+ query is a per-commit tax with no return.
+- **Signals:**
+ - `eventlog.phaseSummary[phase=METADATA_TABLE_WRITE].shareOfStageWallClock`.
On measured runs this
+ sat around 0.06-0.10; materially above that is worth a look.
+ - `props`, the full set of index switches:
`hoodie.metadata.index.column.stats.enable`,
+ `hoodie.metadata.index.partition.stats.enable`,
`hoodie.metadata.index.bloom.filter.enable`,
+ `hoodie.metadata.index.secondary.enable`,
`hoodie.metadata.global.record.level.index.enable`.
+ **Three of these default differently per engine** — on Spark, column stats
and secondary index
+ are effectively `true` regardless of the declared default.
+ - `table`: which partitions actually exist under the metadata table.
+ - `props`: `hoodie.metadata.compact.max.delta.commits` — how often the
metadata table compacts
+ itself.
+- **Corroboration:** `props` + `eventlog`. `table` confirms which indexes were
actually built.
+- **Change:** Turn off index partitions nothing queries. The common one is
+ `hoodie.metadata.index.bloom.filter.enable`, which is off by default but
gets switched on and
+ forgotten when the write index is not bloom. Conversely, if metadata-table
**compaction** is behind,
+ the cost is log-block accumulation rather than too many indexes — lower
+ `hoodie.metadata.compact.max.delta.commits`.
+- **Risk:** `operational` — disabling an index partition deletes it;
re-enabling rebuilds it.
+- **Payoff:** Proportional to the share. Modest in absolute terms, paid on
every single commit.
+- **Verify:** `METADATA_TABLE_WRITE` share falls and the corresponding
partition disappears.
+- **Caveat that applies to this row more than any other:** with
+ `hoodie.metadata.streaming.write.enabled=true` (the **Spark default**), a
large share of the
+ metadata write path is unlabelled and lands in `UNKNOWN`, not in
`METADATA_TABLE_WRITE`. On a
+ measured run, toggling that one config moved 56 points of wall clock between
the two buckets. **If
+ `UNKNOWN` is large, the `METADATA_TABLE_WRITE` share understates the real
cost and you must say so.**
+
+---
+
+# Query-side
+
+## OBS-024 — Column stats exist but the data has no locality to prune on
+
+- **Costs:** Full scans on queries that look prunable. The index is built,
maintained and ignored,
+ because min/max per file spans the whole range when data arrives unsorted.
+- **Signals:**
+ - `props`: `hoodie.metadata.index.column.stats.enable` effectively on (the
Spark default).
+ - `props`: `hoodie.clustering.plan.strategy.sort.columns` **unset** —
nothing is ordering the data.
+ - `table`: no clustering `.replacecommit` on the timeline.
+ - `layout.partitions[].sizeStats` showing many files per partition, each of
which will carry the
+ full range.
+- **Corroboration:** `props` + `table`. **Query evidence would be the decisive
third input and you
+ almost certainly do not have it — ask rather than assume.**
+- **Change:** Set `hoodie.clustering.plan.strategy.sort.columns` to the
columns queries filter on and
+ schedule clustering (OBS-003). Sorting is what turns a maintained index into
a selective one.
+- **Risk:** `operational`.
+- **Payoff:** Potentially large on a selective filter, and **exactly zero** if
queries do not filter
+ on those columns. This row depends entirely on query evidence; without it,
present it as a
+ hypothesis and name the evidence that would settle it.
+- **Verify:** The same filtered query reads fewer files after clustering.
Without a before-and-after
+ query plan this is unverified, and you should say so rather than claiming
the change worked.
+
+## OBS-025 — A Merge-on-Read table carries an append-only workload
+
+- **Costs:** Every read merges log files against base files. On a workload
with no updates there is
+ nothing to merge and the merge machinery is pure overhead — plus compaction
running to compact logs
+ that only ever contained inserts.
+- **Signals:**
+ - `table`, commit metadata across recent commits: `numUpdateWrites` near
zero against `numInserts`.
+ **This is the decisive signal** and it needs several commits, not one.
+ - `props`: `hoodie.datasource.write.operation` is `insert` or `bulk_insert`
rather than `upsert`.
+ - `table`: `hoodie.properties` says `MERGE_ON_READ`.
+ - `eventlog.operationSummary[operation=COMPACTION]` — compaction cost being
paid on a workload that
+ produces nothing to compact.
+- **Corroboration:** `table` + `props`.
+- **Change:** **None from this skill.** Table type is `durable` — it is on
storage and cannot be
+ switched in place. This row exists to be *reported*, with the evidence, and
**handed to
+ `hudi-architect`**. Record what you measured so the design conversation
starts from numbers.
+- **Risk:** `durable`. **The agent must refuse to recommend this as an
optimization.**
+- **Payoff:** Not this skill's to promise. Say what you measured and hand it
over.
+- **Verify:** n/a.
+- **What you *can* do safely meanwhile:** if updates genuinely never happen,
confirm the write
+ operation is `insert`/`bulk_insert` rather than `upsert` — that is a `safe`
config change which
+ skips index tagging entirely, and it is often the real win hiding behind
this observation.
+
+## OBS-026 — The pipeline spends most of its time reading an empty source
+
+- **Costs:** Cluster time, and a commit per sync whether or not there was data
— so timeline growth,
+ metadata-table writes and archival work for nothing.
+- **Signals:**
+ -
`eventlog.phaseSummary[phase=SOURCE_READ_AND_TRANSFORM].shareOfStageWallClock`
large while
+ `table` commit metadata shows near-zero records written in the same window.
+ - `eventlog.application.observed.jobCount` high against records written.
+ - `table`: many commits with tiny `numWrites`.
+- **Corroboration:** `eventlog` + `table`.
+- **Change:** Pace the syncs to the source's actual arrival rate rather than
polling continuously.
+ This is source-specific pacing, not a Hudi config, so name the mechanism
rather than inventing a
+ key.
+- **Risk:** `safe`, but it trades latency for efficiency — a product decision.
+- **Payoff:** Fewer commits means a shorter timeline and less service work
downstream. The compounding
+ benefit is larger than the direct one.
+- **Verify:** Commits per hour falls while records per commit rises;
`jobCount` falls.
+- **Caveat:** `SOURCE_READ_AND_TRANSFORM` **bundles source read, user
transformation and record
+ creation** — `StreamSync` tags only the fetch and the emptiness check, and
`HoodieStreamerUtils`
+ carries no job status at all. A large share here may be an expensive
transformer rather than an
+ empty source. Check `bundledSteps` and the commit record counts before
concluding.
+
+---
+
+# Write path and driver
+
+## OBS-027 — Write parallelism is inherited rather than chosen
+
+- **Costs:** The write stage's shape is whatever the source happened to
produce. Sometimes that is
+ fine; when it is not, nothing says so.
+- **Signals:**
+ - `props`: `hoodie.upsert.shuffle.parallelism`,
`hoodie.insert.shuffle.parallelism`,
+ `hoodie.bulkinsert.shuffle.parallelism`,
`hoodie.delete.shuffle.parallelism` all at their `0`
+ default.
+ - `eventlog.stages[].observed.numTasksPlanned` on `DEDUP_AND_INDEX_TAGGING`
stages varying run to
+ run for similar volumes — the tell that it is inherited.
+- **Corroboration:** `props` + `eventlog`.
+- **Change:** Set the operation-relevant config explicitly so the shape is
stable and intentional.
+ **Only the one for the operation you run** — setting `bulkinsert`
parallelism on an upsert workload
+ does nothing, and that inertness is exactly the class of advice this catalog
exists to avoid.
+- **Risk:** `safe`.
+- **Payoff:** Predictability more than speed. Variance between runs collapses,
which makes every other
+ measurement in this catalog more trustworthy.
+- **Verify:** `numTasksPlanned` becomes stable across runs.
+- **Again, the exception:** this does not move the data-table write stage's
task count, which is
+ bucket-shaped and follows the file groups being written.
+
+## OBS-028 — Compaction target IO is at its default on a table it does not fit
+
+- **Costs:** Each compaction round sweeps only part of the backlog, so a tail
accumulates
+ permanently. The table looks like compaction is running — because it is —
while never catching up.
+- **Signals:**
+ - `props`: `hoodie.compaction.target.io` left at its default.
+ - `eventlog.operationSummary[operation=COMPACTION]` present with
+ `phaseSummary[phase=EXECUTION].stageWallClockMillis` large.
+ -
`health.checks[name=compaction].effectiveConfigs["observed.delta.commits.since.last.compaction"]`
+ drifting up across runs despite compaction completing each time. **That
drift, with compaction
+ succeeding, is the whole signal.**
+ - `layout.totalFiles` and `fileCountPerPartition` growing.
+- **Corroboration:** `table` + `props` + `eventlog` — this is a three-input
row, and it reads wrong
+ with any two.
+- **Change:** Raise `hoodie.compaction.target.io` **substantially** so one
round sweeps most
+ candidate file groups. Nudging it is the common mistake: if only half the
file groups fit, half of
+ the remainder still does not fit next round.
+- **Risk:** `operational`. A larger round needs more executor capacity for the
duration.
+- **Payoff:** The backlog converges instead of drifting. Until it converges,
every other MOR
+ observation in this catalog is reading a moving target.
+- **Verify:** Delta commits since last compaction stops drifting up and
stabilises.
+
+## OBS-029 — Source polling is pacing against a quota rather than the workload
+
+- **Costs:** Throughput sits below what the cluster could do, and the ceiling
is the source's
+ per-shard request quota rather than anything about Hudi.
+- **Signals:**
+ -
`eventlog.phaseSummary[phase=SOURCE_READ_AND_TRANSFORM].shareOfStageWallClock`
large with low
+ `inputBytes` on those stages.
+ - `eventlog.stages[].observed.inputBytes` — the input side **does** report
real numbers.
+ - `props`: the source-specific throughput keys under
`hoodie.streamer.source.*` for your connector.
+- **Corroboration:** `eventlog` + `props`.
+- **Change:** Counter-intuitively, **slow the poll** and **reduce** records
per request step-wise
+ until stable. Derive the target from the per-shard byte limit divided by
average record size
+ divided by polls per second. **Do not copy a number from another
deployment** — the right value is
+ payload-specific.
+- **Risk:** `safe`.
+- **Payoff:** Steadier throughput. Nothing, if the source is not the
constraint.
+- **Verify:** `inputBytes` per unit time rises; source-side throttling stops.
+- **Verify the key actually took effect.** A misspelled Hudi config key
produces **no error at all** —
+ the default silently applies. Resolve the exact key for your source and
release rather than
+ assuming, and confirm it in the writer's resolved configuration.
+
+## OBS-030 — Timeline-server markers are costing more than direct markers on a
busy driver
+
+- **Costs:** Every marker creation is an HTTP round trip to an endpoint on the
Spark driver. On a
+ saturated driver those calls queue, and the write stage waits on them.
+- **Signals:**
+ - `eventlog.phaseSummary[phase=MARKER_RECONCILIATION].shareOfStageWallClock`
— on measured runs this
+ sat at 0.012-0.036; materially above that is the signal.
+ - `eventlog.stages[].observed.taskDurationMillis.max` on those stages, with
a high `skewRatio` —
+ tasks waiting on a queued endpoint rather than doing work.
+ - `props`: `hoodie.write.markers.type`. **The engine-effective default is
`TIMELINE_SERVER_BASED` on
+ Spark** and `DIRECT` on Flink and Java — so a Spark user is on
timeline-server markers by
+ default, not by choice.
+ - `props`: `hoodie.markers.timeline_server_based.batch.num_threads`,
+ `hoodie.embed.timeline.server`.
+- **Corroboration:** `eventlog` + `props`.
+- **Change:** Either raise
`hoodie.markers.timeline_server_based.batch.num_threads` to keep up on a
+ busy table, or set `hoodie.write.markers.type=DIRECT` to remove the round
trip entirely at the cost
+ of more object-store writes. `DIRECT` is a reasonable standing choice on a
busy driver, and on
+ object storage with cheap writes it is often simply better.
+- **Risk:** `safe`.
+- **Payoff:** Removes marker latency from the write path. Small share, but it
is latency on the
+ critical path rather than throughput.
+- **Verify:** `MARKER_RECONCILIATION` share falls and its stages stop showing
high skew.
+- **Check the ordering.** If the same table also has a large instant count
(OBS-017) or a pinned
+ timeline (OBS-020), marker pressure is a **symptom** of driver saturation
and will clear when those
+ are fixed. Treat it as primary only on an otherwise-healthy table.
+
+---
+
+## Adding a row
+
+Give it the next free ID; never reuse or renumber. **Every signal must name a
field that exists in
+one of the three JSON outputs** — if you cannot write the path, you do not
have a signal, you have an
+intuition. State the corroborating inputs honestly; a one-input row does not
belong here. Cite the
+exact `hoodie.*` key and verify it against the source tree rather than from
memory. If the only
+remedy is `durable`, the row's "Change" is a handoff to `hudi-architect`,
written as OBS-025 is.
