nsivabalan commented on code in PR #20194: URL: https://github.com/apache/hudi/pull/20194#discussion_r4280148944
########## hudi-ai-operator/skills/hudi-rca/references/failure-catalog.md: ########## @@ -0,0 +1,1576 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +# Hudi failure catalog + +38 failure patterns, each with the signature that identifies it, the mechanism behind it, and an +ordered mitigation ladder. Used by the `hudi-rca` skill to classify a pasted failure; readable on its +own as a troubleshooting reference. + +## How to read a row + +| Field | Meaning | +|---|---| +| **ID** | Stable identifier. Never renumber; retire instead. | +| **Category** | `input/config` · `sizing/scale` · `environment` · `defect` · `not-a-failure` | +| **Discoverability** | `self-evident` (the message says what to do) · `needs-expertise` (the message is generic, misleading, or points at the wrong layer) · `silent` (no exception at all) | +| **Signature** | Exception classes, message regexes, and the Hudi frames that identify the pattern. | +| **Symptom** | What the operator sees, in their terms. | +| **Root cause** | The mechanism, in one or two sentences. | +| **Evidence required** | Which rungs of the evidence ladder are needed to reach a confident call. | +| **Disambiguation** | Present only where a signature maps to more than one pattern. The explicit discriminator. | +| **Mitigation ladder** | Ordered, cheapest and most likely first. Counter-intuitive steps are marked and justified. | +| **Governing configs** | Exact `hoodie.*` / `spark.*` keys. | +| **`fixedInVersion`** | The Hudi release that fixes the defect, where one exists. | +| **`preflightGuardExists`** | Whether a current Hudi release fails fast on the condition instead of corrupting silently. | +| **`alsoAppliesWhenHealthy`** | `true` when the same advice is an optimization on a merely-slow job, not only a fix for a broken one. | +| **Provenance** | Public `apache/hudi` issue numbers. | + +## Three rules that govern every row + +**1. Match the innermost `Caused by`, never the wrapper.** In the public tracker the wrapper classes +dominate the counts — bare `HoodieException` 498 occurrences, `HoodieUpsertException` 189 — while the +classes that actually identify the fault barely appear: `SchemaCompatabilityException` 1, +`HoodieAvroSchemaException` 2, `HoodieKeyGeneratorException` 3. Users paste the wrapper. Every regex +below is written against the innermost cause plus the first Hudi frame, because a classifier keyed on +the top-level exception class will mostly see wrappers and learn nothing. + +**2. The stacktrace is the weakest signal, not the backbone.** For a large share of these the terminal +exception is generic and the real cause is table state or config sizing several steps removed. A +`413 Payload Too Large` on timeline init is really a forgotten savepoint blocking archival. A stalled +ingestion is really a rollback that never finished and still holds the lock. Where a row says the +evidence requires table state, the honest move is to run `HoodieTableHealthChecker` rather than parse +the trace harder. + +**3. Ask the Hudi version early.** The heavy issue clusters are 0.9 through 0.15, and several +high-traffic patterns here are fixed outright in 1.0 or 1.2. For this population "upgrade" is a +genuinely good first-line remedy in a way it usually is not. Rows carry `fixedInVersion` where known. + +## Disambiguation index + +Four signatures map to more than one pattern. Resolve them here before attributing. + +| Signature | Candidates | Discriminator | +|---|---|---| +| `ConcurrentModificationException` / `HoodieWriteConflictException` | HUDI-RCA-006 healthy OCC retry **vs** HUDI-RCA-006 no lock provider | **Writer config, not the trace** — the stacktrace is identical. If `hoodie.write.lock.provider` is set and two writers are intended, this is a by-design rejection: retry the application attempt. If no lock provider is set and two writers exist, the table is being corrupted silently and the conflict exception is the lucky case. | +| `HoodieRollbackException` | HUDI-RCA-027 rollback wedge · HUDI-RCA-026 metadata-table divergence · HUDI-RCA-025 dangling files | **The frame below `rollbackFailedWrites`.** `AsyncCleanerService.waitForCompletion` → cleaner/metadata-table path (026). A lock-provider frame (`LockManager.lock`, `*LockProvider.acquireLock`) → lock problem first (006). `HoodieMergeHandle.getLatestBaseFile` with `FileID ... does not exist` → 025. | +| `OutOfMemoryError` | HUDI-RCA-015 merge-side · HUDI-RCA-015 archival-side · HUDI-RCA-014 bloom index · HUDI-RCA-021 driver/timeline-server | **The Hudi frame.** `HoodieMergedLogRecordScanner` / `ExternalSpillableMap` → merge side (015a). `HoodieTimelineArchiveLog.writeToFile` / `HoodieAvroDataBlock.serializeRecords` → **archival side (015b), where adding executor memory is the wrong answer entirely** — the fix is to shrink commit metadata. Kryo / `ExternalSorter` frames under a shuffle read with no Hudi frame → bloom index (014). `Requested array size exceeds VM limit` in a driver Jetty thread → 021, a structural ceiling that more heap cannot move. | +| `FileNotFoundException` | HUDI-RCA-031 markers · HUDI-RCA-033 archiver race · HUDI-RCA-025 stale plan · HUDI-RCA-036 source file | **The path.** `MARKERS*` or `.hoodie/.temp/` → 031. A `.commit` / `.deltacommit` under `.hoodie/` → 033. A data or log file named in a compaction plan → 025. A path under the **source** bucket, not under the table → 036. A path under the table ending `.log.<n>` → treat as a compaction race, see 025's note. | + +## Cross-cutting trap: MOR defers the failure + +On Merge-on-Read an incompatible schema write **succeeds**, and compaction fails hours later in a +*different job* whose stacktrace names `HoodieCompactor` rather than the write. The table then cannot +compact and every later write fails. Cause and symptom are separated by both time and process. + +**Rule: a compaction frame plus a schema exception means the bad write happened earlier, in another +job. Investigate the writes from that window, not the job that just failed.** This applies to +HUDI-RCA-001, HUDI-RCA-003 and HUDI-RCA-004. + +--- + +# Group A — input/config + +The user asked Hudi to do something invalid, or configured it inconsistently. The remedy is to fix +data or config; nothing is wrong with Hudi or the cluster. + +--- + +## HUDI-RCA-001 — Schema evolution the writer rejects + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `org\.apache\.avro\.AvroTypeException: Cannot encode decimal with precision (\d+) as max precision (\d+)` + - `org\.apache\.parquet\.io\.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '([^']+)' not found` + - Hudi frames: `HoodieAvroUtils.rewriteRecordWithNewSchema`, `HoodieAvroUtils.rewritePrimaryTypeWithDiffSchemaType`, `HoodieMergeHelper.runMerge`, `AbstractHoodieLogRecordReader.scanInternal`, `HoodieCompactor.compact` +- **Symptom:** A write that worked yesterday fails after an upstream column was retyped or a decimal + widened. On MOR the write succeeds and **compaction** fails hours later, permanently, for every + subsequent write. +- **Root cause:** The incoming Avro schema is not a backward-compatible evolution of the table schema + under Hudi's rules. Full schema-on-read evolution is off by default, and decimal precision widening + was rejected outright until the `SchemaChangeUtils.isTypeUpdateAllow` relaxation. +- **Evidence required:** stacktrace · writer props · `hoodie.properties` (table schema) · for the MOR + variant, table state to show which file slices span the change. +- **Disambiguation:** a compaction frame here means the offending write already committed — see the + MOR deferred-failure rule above. Distinguish from HUDI-RCA-003 by *direction*: a field the table has + and the batch lacks is 003; a field whose **type** changed is 001. +- **Mitigation ladder:** + 1. Upgrade if the signature is decimal precision — fixed in 1.0.x, and no config works around it on + 0.14/0.15. + 2. Set `hoodie.datasource.write.reconcile.schema=true`, with `hoodie.schema.on.read.enable=true`. + 3. Stop dropping or retyping the column in the writer: keep it and write typed nulls. + 4. If MOR and already wedged, the table cannot compact until the pending compaction is unscheduled + or the affected file groups are rewritten. Do not hand-delete the pending instant files. +- **Governing configs:** `hoodie.datasource.write.reconcile.schema`, `hoodie.schema.on.read.enable`, + `hoodie.avro.schema.validate`, `hoodie.datasource.write.schema.allow.auto.evolution.column.drop` +- **`fixedInVersion`:** 1.0.0 for decimal precision widening · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #8040, #5452, #11335, #8020, #2919 + +--- + +## HUDI-RCA-002 — Record key or precombine field missing, null, or wrong case + +- **Category:** `input/config` · **Discoverability:** `self-evident` for the key case, `needs-expertise` for precombine +- **Signature** (innermost cause): + - `org\.apache\.hudi\.exception\.HoodieKeyException: recordKey value: "null" for field: "([^"]+)" cannot be null or empty` + - `org\.apache\.hudi\.exception\.HoodieKeyException: Record key has to be non-null!` + - `org\.apache\.hudi\.exception\.HoodieException: \w+\(Part -\w+\) field not found in record\. Acceptable fields were :\[` + - Hudi frames: `HoodieAvroUtils.getNestedFieldVal`, `ComplexKeyGenerator`, often wrapped by + `SparkException: Failed to execute user defined function` so the top frame is Spark, not Hudi. +- **Symptom:** The write fails on the first batch, or only when one composite key field is null. +- **Root cause:** Three sub-cases sharing a remedy family: the configured field does not exist or is + null in some rows; the key config is **case-sensitive** and the user wrote `ID` for a column named + `id`; or HoodieStreamer defaulted `--source-ordering-field` to `ts`, producing a phantom `ts` + requirement that then persisted into `hoodie.properties`. +- **Evidence required:** stacktrace · writer props · `hoodie.properties` +- **Disambiguation:** if the named field is one the user never configured (classically `ts`), this is + the HoodieStreamer default, not a missing column. The message names a field the user has never heard + of, which is the whole confusion — say so explicitly rather than asking them to add a `ts` column. +- **Mitigation ladder:** + 1. Check the exact field name **and case** against the dataframe schema. Hudi does not normalize case. + 2. Filter or default nulls in the key columns before writing. + 3. For HoodieStreamer with no ordering field: set `--source-ordering-field` to a real column, or use + `--op INSERT` / `bulk_insert`, which skip precombine entirely. + 4. Multi-field keys with `bulk_insert` had a genuine bug before 0.13.1 — upgrade, or switch to + `upsert` to confirm. +- **Governing configs:** `hoodie.datasource.write.recordkey.field`, + `hoodie.datasource.write.precombine.field`, `hoodie.datasource.write.keygenerator.class`, + `hoodie.table.precombine.field`, `--source-ordering-field` +- **`fixedInVersion`:** 0.13.1 for the bulk_insert multi-field key bug · **`preflightGuardExists`:** partial — the key check fires at write time, not at config time +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #3759, #9799, #11776, #10233 + +--- + +## HUDI-RCA-003 — Source drops a column the table still has + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `org\.apache\.parquet\.io\.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '([^']+)' not found` + - Frames, in order: `AvroRecordConverter.getAvroField` → `ParquetReaderIterator.hasNext` → + `HoodieMergeHelper.runMerge` + - Wrapped by `HoodieUpsertException: Error upserting bucketType UPDATE for partition :\d+` +- **Symptom:** Every sync fails after an upstream schema change. The exception arrives on the **merge** + path while *reading existing Parquet*, so operators look at their table instead of their batch. +- **Root cause:** The incoming batch no longer carries a column the table's base files have. Hudi reads + existing records using the latest (incoming) schema, and the Parquet-to-Avro converter cannot find + the field. +- **Evidence required:** stacktrace · writer props · last commit metadata (`extraMetadata.schema`) for + the field's declared type +- **Disambiguation:** same exception class as HUDI-RCA-001's field-not-found variant. The + `HoodieMergeHelper.runMerge` frame and the fact that the field exists in the **table** but not the + **batch** is what makes it 003. Recover the field's declared type including its Avro `logicalType` + before re-adding it, or the re-add fails differently. +- **Mitigation ladder:** + 1. Re-add the column to the incoming batch as a **typed** null via a transformer — `cast(null as + string)`, or `cast(null as date)` where the logical type demands it. Cheapest and reversible. + 2. Set `hoodie.avro.schema.validate=false` only as a stopgap, not as the fix. + 3. **Config-layering trap:** if your config system overrides rather than merges list-valued settings, + adding a transformer silently drops the transformer that was already configured. Re-list all of + them, and verify the resolved config rather than the file you edited. +- **Governing configs:** `hoodie.avro.schema.validate`, + `hoodie.datasource.write.schema.allow.auto.evolution.column.drop`, + `hoodie.datasource.write.reconcile.schema` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #8040 + +--- + +## HUDI-RCA-004 — Unvalidated source schema: records do not parse against the inferred type + +- **Category:** `input/config` · **Discoverability:** `self-evident` for the message, `needs-expertise` for the fix +- **Signature** (innermost cause): + - `org\.apache\.spark\.sql\.catalyst\.util\.BadRecordException` + - `java\.lang\.RuntimeException: Failed to parse a value for data type struct<.*> \(current token: \w+\)` +- **Symptom:** Records fail to parse and the whole job fails, rather than the bad rows being isolated. +- **Root cause:** A semi-structured source (CDC, JSON, document-store feed) emits a field whose actual + shape differs from what was inferred from earlier records — most often an array where a struct was + inferred. +- **Evidence required:** stacktrace · writer props (schema-provider config) · a raw source sample +- **Mitigation ladder:** + 1. Set an explicit **target schema** instead of relying on inference, and enable validation with a + quarantine destination so bad records are isolated rather than failing the commit. + 2. For CDC sources, make sure the target schema declares the envelope fields the transformer adds + (operation type, transaction id, LSN, event timestamps) — otherwise validation rejects every record. + 3. Loosen validation only as a last resort. It converts a loud failure into a silent one. +- **Governing configs:** `hoodie.streamer.schemaprovider.target.schema.file`, + `hoodie.streamer.schemaprovider.source.schema.file`, `hoodie.avro.schema.validate` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public — the mechanism is config hygiene and rarely filed + +--- + +## HUDI-RCA-005 — Timestamp key generator cannot parse the partition field + +- **Category:** `input/config` · **Discoverability:** `self-evident` +- **Signature** (innermost cause): + - `org\.apache\.hudi\.exception\.HoodieKeyGeneratorException: Unable to parse input partition field: (.+)` + - `java\.lang\.IllegalArgumentException: Invalid format: "(.+)" is malformed at "(.+)"` + - Hudi frame: `TimestampBasedAvroKeyGenerator.getPartitionPath` +- **Symptom:** Every write fails immediately, naming the exact offending value. +- **Root cause:** The configured **input** date format does not match the actual partition-field format. + Typical triggers: microsecond precision, a trailing `Z`, or a switch from epoch to ISO-8601 upstream. +- **Evidence required:** stacktrace alone suffices · writer props to confirm the configured format +- **Mitigation ladder:** + 1. Update the key generator's **input** format config to match the value printed in the exception. + 2. Verify the **output** partition format is unchanged. Changing the output format alters partition + paths and effectively forks the table — a far worse outcome than the parse failure. + 3. If the source format is now inconsistent across records, normalize it in a transformer before the + key generator sees it. +- **Governing configs:** `hoodie.keygen.timebased.input.dateformat`, + `hoodie.keygen.timebased.output.dateformat`, `hoodie.keygen.timebased.timestamp.type`, + `hoodie.keygen.timebased.input.timezone`, `hoodie.keygen.timebased.output.timezone` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public with this exact signature; included because it is trivially fixable and + almost certainly under-reported + +--- + +## HUDI-RCA-006 — Multi-writer without, or with a misconfigured, lock provider + +- **Category:** `input/config` · **Discoverability:** `needs-expertise`, and **`silent`** in the no-lock-at-all case +- **Signature** (innermost cause): + - `java\.util\.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes`, + wrapped in `HoodieWriteConflictException`; frames + `SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict`, + `TransactionUtils.resolveWriteConflictIfAny`, `SparkRDDWriteClient.preCommit` + - `org\.apache\.hudi\.exception\.HoodieLockException: Unable to acquire lock, lock object (null|LockResponse\(lockid:\d+, state:WAITING\))`; + frames `LockManager.lock`, `TransactionManager.beginTransaction` + - `java\.lang\.IllegalArgumentException: ALREADY_ACQUIRED` at `HiveMetastoreBasedLockProvider.acquireLock` + - `org\.apache\.hadoop\.fs\.FileAlreadyExistsException: File already exists: .*\.hoodie/\d+\.commit\.requested` + at `HoodieActiveTimeline.createImmutableFileInPath` +- **Symptom:** Intermittent failures that scale with writer count. With **no** lock provider there may + be no exception at all — just overwritten timeline state and lost data. +- **Root cause:** Two writers against one table without OCC properly configured, or OCC configured and + genuinely conflicting. The semantics users most often get wrong: **the OCC lock is table-level, not + partition-level**, and it serializes only the commit-validation phase. Writers on disjoint partitions + still serialize on the lock. +- **Evidence required:** stacktrace · **writer props — mandatory** · `hoodie.properties` +- **Disambiguation (the most important row in this catalog):** the conflict exception and the no-lock + corruption produce the **same stacktrace**. Only the writer config splits them. + - `hoodie.write.lock.provider` set, concurrency intended → by-design rejection of the later committer. + Expected. Add orchestrator-level retry of the whole application attempt. Severity: routine. + - No lock provider set, but two writers demonstrably exist → the table is being corrupted and this + exception is the *lucky* case where you found out. Severity: urgent. + - Treat "is a second writer intended?" as a question to ask the user, not something to infer. +- **Mitigation ladder:** + 1. If two writers exist and no lock provider is set, set one. Non-negotiable; absence is silent + corruption, not a performance choice. Review Comment: Fixed in `576f14a4a46d`, and the failure mode is worse than incomplete advice — it is silently undone. `HoodieWriteConfig.Builder.autoAdjustConfigsForConcurrencyMode` (`HoodieWriteConfig.java:3938-3948`) **replaces an explicitly configured lock provider with `InProcessLockProvider`** when the concurrency mode is single-writer and no table service is async, logging only a WARN. The job then succeeds with a non-distributed lock, which is the corruption case wearing a green tick. So the step now sets `hoodie.write.concurrency.mode` alongside the provider, and sets `hoodie.clean.failed.writes.policy=LAZY` explicitly rather than relying on the builder to infer it — `validate()` at `:3987-3993` throws when a multi-writer mode meets the default `EAGER`, and leaving it implicit hides the dependency from the next reader. ########## hudi-ai-operator/skills/hudi-rca/SKILL.md: ########## @@ -0,0 +1,254 @@ +--- +name: hudi-rca +description: Explains why an Apache Hudi job failed. Classifies a pasted stacktrace and supporting artifacts against a catalog of known failure patterns, states the evidence it is reasoning from, and gives an ordered mitigation ladder. When no pattern matches, narrates what Hudi was doing and which phase failed. Invoke when a user brings a Hudi failure, a confusing exception, a stalled table, or asks why a write, compaction, or table service is not working. +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Hudi RCA + +You are explaining a failed Apache Hudi job to the engineer who owns it. Your output is an +explanation they can act on, not a verdict. + +**You diagnose. You never mutate.** Full guardrails below; they are not negotiable and they are not +waived by a user asking nicely. + +## The one thing to get right + +**Classify on the innermost `Caused by`, never on the wrapper.** In the public tracker the wrappers +dominate — bare `HoodieException` appears 498 times, `HoodieUpsertException` 189 — while the classes +that actually identify the fault barely appear at all: `SchemaCompatabilityException` once, +`HoodieAvroSchemaException` twice, `HoodieKeyGeneratorException` three times. Users paste the wrapper +because that is what the console shows them. + +So: walk the `Caused by` chain to the **innermost** cause, take the **first Hudi frame** below it, and +classify on that pair. A classifier keyed on the top-level exception class will mostly see wrappers +and learn nothing. `HoodieException: Commit <instant> failed and rolled-back !` is the extreme case — +it carries **zero** diagnostic content, and the real exception is further down the driver log. + +## Ask the Hudi version early + +It resolves or reclassifies a meaningful share of patterns. The heavy issue clusters are 0.9 through +0.15, and several high-traffic patterns are fixed outright in 1.0 or 1.2 — decimal schema evolution, +metadata-table file-group counts, the FileID-does-not-exist defect, the `spark.speculation` guard. +For this population "upgrade" is a genuinely good first-line remedy in a way it usually is not. Ask in +the first exchange, alongside the artifacts. + +## Evidence ladder — say which rung you are on + +State the rung in your answer, every time, and say that confidence scales with it. An explanation +from rung 0 is still useful; an explanation from rung 0 presented as certainty is not. + +| Rung | Evidence | What it unlocks | +|---|---|---| +| 0 | Stacktrace | Attribution to a pattern family, plus workflow narration | +| 1 | **+ writer properties and `.hoodie/hoodie.properties`** | Config-aware attribution. Several patterns are *only* resolvable here | +| 2 | **+ health-checker JSON** | Table-state causes: savepoints, archival, compaction, cleaner, metadata table, record index | +| 3 | + driver log | The Hudi phase: which operation, which instant, which failing stage | +| 4 | + Spark event log | Per-stage metrics: skew, spill, GC, shuffle sizes — and therefore real mitigation | +| 5 | + executor logs | OOM site, container kill reason, GC pauses | + +**Ask for `.hoodie/hoodie.properties` by default, in your first reply.** It is the single +highest-value artifact. Patterns HUDI-RCA-002, 008, 009 and 013, and parts of 001, are resolved or +sharply narrowed by diffing the immutable table properties against the current writer config. No +stacktrace substitutes for it. + +**Ask for the Spark event log, not "Spark UI access."** The UI is ephemeral and cannot be attached; +the event log (`spark.eventLog.enabled`) is a file containing everything the UI shows. + +Ask for what the next rung needs — but give the user the rung-0 reading first. Do not withhold an +explanation pending perfect evidence. + +## Procedure + +### 1. Extract the causal chain + +Walk to the innermost `Caused by`. Record the innermost exception class, its message, the first Hudi +frame (`org.apache.hudi.*`), and whether any Hudi or user-code frame exists at all. Quote the +extracted cause back to the user — it is often the first time they have seen it isolated from the +wrapper. + +### 2. Match against the catalog + +Read `references/failure-catalog.md`. Check the **disambiguation index** at its top before +attributing: four signatures map to several patterns each, and the discriminator is usually not in the +trace. + +- `ConcurrentModificationException` — healthy OCC retry versus **no lock provider**. Identical + stacktrace; **the writer config decides**. Ask; do not infer. +- `HoodieRollbackException` — three patterns. The frame **below** `rollbackFailedWrites` decides. +- `OutOfMemoryError` — merge-side versus archival-side versus index-side. The Hudi frame decides, and + the remedies do not overlap. +- `FileNotFoundException` — the **path** decides: `MARKERS*`, a `.commit` under `.hoodie/`, a file in + a compaction plan, or a path under the source bucket are four different patterns. + +### 3. Classify into one of five categories + +`input/config` · `sizing/scale` · `environment` · `defect` · `unclassified` + +**Never attribute without a matched catalog row.** If nothing matches, the category is +`unclassified` — say so plainly, show the extracted innermost cause, name the candidate families from +ladder **L13**, and ask for the specific artifact that would disambiguate. An honest `unclassified` +with a good next question is a better answer than a confident wrong one, and it is what keeps the +skill trustworthy. Do not stretch a row to fit. + +### 4. Delegate table-state questions to the tools + +Do not guess at table state from a stacktrace. For anything about table services, savepoints, +archival, the metadata table, or the record index: + +``` +spark-submit --class org.apache.hudi.utilities.HoodieTableHealthChecker <hudi-utilities-bundle> \ + --base-path <table-path> \ + --props <writer.properties> \ + --checks compaction,cleaner,savepoint,archival,mdt-sync,mdt-compaction,record-index \ + --output JSON +``` + +Exit code 0 means every check that ran was healthy, 1 means at least one was unhealthy, 2 means bad +arguments or the table could not be read. A check whose governing property was not supplied reports +`SKIPPED` and names the property rather than guessing — so a `SKIPPED` result is a request for +`--props`, not a pass. + +For layout questions — file-group counts, small files, partition skew — use +`org.apache.hudi.utilities.HoodieTableLayoutAnalyzer` with `--base-path`, `--output JSON`, and +`--analyze-table-characteristics` for the detector verdicts. + +Both tools ship in `hudi-utilities`. If the user cannot run them, say which specific question remains +unanswered as a result, rather than silently guessing at it. + +### 5. Give the ladder, in order, with the counter-intuitive steps flagged + +Mitigation ladders are ordered cheapest-and-most-likely first. Give them in order and say why a step +is where it is. Several are **deliberately counter-intuitive** and a generic reading of the situation +will suggest the opposite — when you hit one, say so explicitly and say why: + +- **reduce** index parallelism (HUDI-RCA-016) — raising it makes the broadcast failure worse +- **reduce** keys per bucket (HUDI-RCA-014) — smaller per-task batches, more of them +- **reduce** concurrency, not increase it, under connection-pool exhaustion (HUDI-RCA-024) +- **reduce** records per request under throttling (HUDI-RCA-019) +- **lower** `spark.memory.fraction` under GC pressure (L2) — less for Spark's managed regions leaves + more for user objects +- **don't** disable compaction to dodge a conflict (HUDI-RCA-021) — the single most damaging move +- **don't** restart (HUDI-RCA-021, 022, 030, and L9) — restarting a rollback or clustering wedge + abandons work and re-requests a larger unit; restarting HUDI-RCA-030 is provably useless +- **don't** reset the checkpoint on a transient source regression (HUDI-RCA-012) — that loses data +- **do nothing at all** (HUDI-RCA-037, 038, 039) — and say why that is the correct answer, not a + deferral + +Read `references/mitigation-ladders.md` for the thirteen procedures. **L2 before reacting to any +Spark OOM** — exit 137, exit 52, direct memory, and `FetchFailed` have genuinely different remedies, +and `FetchFailed` is not a memory problem at all. **L9 before any restart.** + +### 6. Fall back to workflow narration + +**This is a first-class outcome, not an apology.** When no catalog row matches, explain: + +1. **What Hudi was doing** — the write operation and its phases. +2. **Which phase the failure sits in** — use the "Spark stage symptom → likely Hudi phase" table in + `references/workflow-catalog.md`. +3. **The candidate causes for that phase**, and how the user would check each. + +Present this as the answer, in the same voice as a matched explanation. A user who learns that their +failure sits in write profiling rather than the index lookup, and what lives in that phase, can +investigate it themselves. That is the product. + +Three facts from the workflow catalog that contradict common understanding often enough to be worth +stating when relevant: shuffle-parallelism configs default to `0`, **not 200**, where `0` means +"inherit the input RDD's partition count"; the write stage's task count is bucket-shaped, so tuning Review Comment: Correct, and there is one more step in the chain than your note mentions. `HoodieJavaRDD.deduceNumPartitions` (`:132-146`) resolves in this order: 1. an existing partitioner on the RDD, if it reports `> 0` partitions 2. `spark.sql.shuffle.partitions` 3. `spark.default.parallelism` 4. only then the input RDD's own partition count Fixed in `576f14a4a46d`. SKILL.md now gives the full order and drops the backwards inference entirely — on a tuned cluster one of the Spark properties almost always wins, so a task count cannot be read back as an input partition count. The `0` ≠ 200 point stands and is kept. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
