nsivabalan opened a new pull request, #20194: URL: https://github.com/apache/hudi/pull/20194
### Describe the issue this Pull Request addresses Part of #19262 (Hudi-aware agent skills and tools), under the Agentic Lakehouse umbrella #19256; overall plan in https://github.com/apache/hudi/issues/19262#issuecomment-5750570015. When a Hudi write fails, the user gets a stack trace that usually names a wrapper (`HoodieUpsertException`, `HoodieException`) rather than the thing that actually went wrong, and a Spark stage number with no indication of which part of Hudi it belongs to. Working out whether the cause is their data, their configuration, their cluster, or a Hudi defect takes familiarity most users reasonably do not have. This adds a diagnostic skill that reads the evidence a user can actually produce — a stack trace, their writer properties, `.hoodie/hoodie.properties`, driver logs, a Spark event log — and explains what happened, in which phase of which operation, and what to do about it. ### Summary and Changelog Three reference documents plus a skill, all under `hudi-agent-gateway/skills/hudi-rca/`. No code, no configs, no behaviour change — this is knowledge, structured so an agent can apply it. **`references/workflow-catalog.md`** (970 lines) — what Hudi actually does, per operation, derived from the source with `file:line` citations throughout. For UPSERT (CoW and MoR), INSERT/BULK_INSERT, COMPACTION, CLUSTERING, CLEANING, ARCHIVAL, ROLLBACK, metadata-table writes and the HoodieStreamer loop, it gives the ordered phases, whether each runs on the driver or executors, the Spark operation it produces, the configs that govern its parallelism, and what commonly fails there. It closes with a **"Spark stage symptom → likely Hudi phase"** lookup table, which is the practical entry point: given "stage 14, 16000 tasks, OOM", it tells you that is bloom-index probing and what to check. Writing it surfaced several places where the code does not match widely-held belief, each cited: - The shuffle-parallelism configs (`hoodie.upsert.shuffle.parallelism` and siblings) default to `0`, meaning "inherit the input partition count" — not 200, which was the pre-0.13.0 default. - Rolling back a *completed* instant ignores `hoodie.rollback.using.markers` entirely, so restores and savepoint rollbacks always take the listing path and scan every partition. - Archival raises `hoodie.keep.min.commits` on its own when cleaner retention requires it, so a growing timeline is often a cleaning-config problem rather than an archival-config one. - The write stage's task count comes from `UpsertPartitioner.totalBuckets`, so shuffle-parallelism tuning does not move it; only file-sizing configs do. - A metadata table that has never compacted blocks data-table archival outright. - Record-level index falls back to a full key scan with only a warning when `record_index` is uninitialised. **`references/failure-catalog.md`** (1576 lines) — 39 failure patterns, each with its signature (exception classes, message regexes, identifying Hudi frames), the symptom as a user sees it, the root-cause mechanism, which evidence is needed to confirm it, an ordered mitigation ladder, the governing configs, and where known the version that fixes it. Patterns are classified `input/config`, `sizing/scale`, `environment`, `defect` or `not-a-failure`, and tagged by discoverability — whether the error message tells you what to do, misleads you, or never appears at all. Two properties of the catalog are worth calling out: *Signatures match the innermost `Caused by`, not the wrapper.* On the public tracker the wrapper classes dominate by an order of magnitude over the classes that identify the fault, because that is what users paste. Classification keyed on the top exception would mostly learn nothing. *Ambiguous signatures get explicit discriminators.* `ConcurrentModificationException` is either a healthy OCC retry or a data-corrupting absence of a lock provider, with an identical stack trace — only the writer configuration separates them. `HoodieRollbackException` maps to four patterns, `OutOfMemoryError` to four, `FileNotFoundException` under `.hoodie` to four, each resolved by a named frame or path. These are indexed at the head of the catalog. Three patterns have "this is correct behaviour, do nothing" as the answer — a cleaner that appears stuck is usually right (if `numWrites == numInserts` the write created a new file group, so there is no older version to clean), a source with no new data, and spurious data files at commit that resolve themselves. An agent that confidently recommends action on those is worse than no agent. One pattern is marked explain-only: the agent identifies it and explains it, but must never recommend or script the file deletion that repairs it. That is a human's decision, behind the safe-deletion protocol. **`references/mitigation-ladders.md`** (311 lines) — 13 operational procedures that apply across patterns. The most useful is classifying a Spark OOM before reacting to it: container OOM-kill, JVM heap OOM, direct-memory exhaustion, and `FetchFailedException` from evicted executors look similar and need completely different responses — and the last one is not a memory problem at all but ephemeral local storage, which does not show up in the Spark UI. Others cover deciding whether a stuck table service is actually stuck (archival → cleaning → compaction → blocking instant, each link gating the next), and capturing evidence before restarting, since restarting is usually the most expensive mistake available. **`SKILL.md`** (254 lines) — the skill. It collects evidence in rungs and states which rung it is on, because confidence depends on it; a stack trace alone supports attribution and explanation, while mitigation-grade answers need the event log. It asks for `.hoodie/hoodie.properties` by default, since diffing immutable table properties against the current writer config resolves or narrows a large share of patterns outright. It classifies by walking the `Caused by` chain, never the wrapper. It delegates table-state questions to `HoodieTableHealthChecker` and layout questions to `HoodieTableLayoutAnalyzer` rather than guessing. When nothing matches, it explains the workflow instead: what Hudi was doing, which phase the failure sits in, and the candidate causes for that phase, so the user can investigate. That is a designed outcome rather than a failure mode — an unmatched trace with a clear account of where it happened is more useful than a confident guess. **`references/test-corpus.md`** (89 lines) — stack traces from public issues with their expected classification, as a regression set for the catalog. The initial recipes come from users running Spark, since that is where the evidence was. The core and table-layer patterns — schema, keys, timeline, table services, metadata table — apply to Flink deployments as well, and Flink-specific coverage will deepen over time. ### Impact Purely additive documentation and a skill. No Java, no configs, no storage format changes, no behaviour change. Nothing imports or executes any of it. The skill is read-only by construction: it diagnoses and recommends, and never runs a table service, deletes a file, or mutates a table. ### Risk Level none ### Documentation Update Self-contained. If reviewers would like any of the workflow catalog's corrections reflected on the website — particularly the shuffle-parallelism default, which is widely repeated as 200 — that seems worth doing separately. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
