nsivabalan commented on code in PR #20194: URL: https://github.com/apache/hudi/pull/20194#discussion_r4279982940
########## hudi-ai-operator/skills/hudi-rca/references/failure-catalog.md: ########## @@ -0,0 +1,1576 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +# Hudi failure catalog + +38 failure patterns, each with the signature that identifies it, the mechanism behind it, and an +ordered mitigation ladder. Used by the `hudi-rca` skill to classify a pasted failure; readable on its +own as a troubleshooting reference. + +## How to read a row + +| Field | Meaning | +|---|---| +| **ID** | Stable identifier. Never renumber; retire instead. | +| **Category** | `input/config` · `sizing/scale` · `environment` · `defect` · `not-a-failure` | +| **Discoverability** | `self-evident` (the message says what to do) · `needs-expertise` (the message is generic, misleading, or points at the wrong layer) · `silent` (no exception at all) | +| **Signature** | Exception classes, message regexes, and the Hudi frames that identify the pattern. | +| **Symptom** | What the operator sees, in their terms. | +| **Root cause** | The mechanism, in one or two sentences. | +| **Evidence required** | Which rungs of the evidence ladder are needed to reach a confident call. | +| **Disambiguation** | Present only where a signature maps to more than one pattern. The explicit discriminator. | +| **Mitigation ladder** | Ordered, cheapest and most likely first. Counter-intuitive steps are marked and justified. | +| **Governing configs** | Exact `hoodie.*` / `spark.*` keys. | +| **`fixedInVersion`** | The Hudi release that fixes the defect, where one exists. | +| **`preflightGuardExists`** | Whether a current Hudi release fails fast on the condition instead of corrupting silently. | +| **`alsoAppliesWhenHealthy`** | `true` when the same advice is an optimization on a merely-slow job, not only a fix for a broken one. | +| **Provenance** | Public `apache/hudi` issue numbers. | + +## Three rules that govern every row + +**1. Match the innermost `Caused by`, never the wrapper.** In the public tracker the wrapper classes +dominate the counts — bare `HoodieException` 498 occurrences, `HoodieUpsertException` 189 — while the +classes that actually identify the fault barely appear: `SchemaCompatabilityException` 1, +`HoodieAvroSchemaException` 2, `HoodieKeyGeneratorException` 3. Users paste the wrapper. Every regex +below is written against the innermost cause plus the first Hudi frame, because a classifier keyed on +the top-level exception class will mostly see wrappers and learn nothing. + +**2. The stacktrace is the weakest signal, not the backbone.** For a large share of these the terminal +exception is generic and the real cause is table state or config sizing several steps removed. A +`413 Payload Too Large` on timeline init is really a forgotten savepoint blocking archival. A stalled +ingestion is really a rollback that never finished and still holds the lock. Where a row says the +evidence requires table state, the honest move is to run `HoodieTableHealthChecker` rather than parse +the trace harder. + +**3. Ask the Hudi version early.** The heavy issue clusters are 0.9 through 0.15, and several +high-traffic patterns here are fixed outright in 1.0 or 1.2. For this population "upgrade" is a +genuinely good first-line remedy in a way it usually is not. Rows carry `fixedInVersion` where known. + +## Disambiguation index + +Four signatures map to more than one pattern. Resolve them here before attributing. + +| Signature | Candidates | Discriminator | +|---|---|---| +| `ConcurrentModificationException` / `HoodieWriteConflictException` | HUDI-RCA-006 healthy OCC retry **vs** HUDI-RCA-006 no lock provider | **Writer config, not the trace** — the stacktrace is identical. If `hoodie.write.lock.provider` is set and two writers are intended, this is a by-design rejection: retry the application attempt. If no lock provider is set and two writers exist, the table is being corrupted silently and the conflict exception is the lucky case. | +| `HoodieRollbackException` | HUDI-RCA-027 rollback wedge · HUDI-RCA-026 metadata-table divergence · HUDI-RCA-025 dangling files | **The frame below `rollbackFailedWrites`.** `AsyncCleanerService.waitForCompletion` → cleaner/metadata-table path (026). A lock-provider frame (`LockManager.lock`, `*LockProvider.acquireLock`) → lock problem first (006). `HoodieMergeHandle.getLatestBaseFile` with `FileID ... does not exist` → 025. | +| `OutOfMemoryError` | HUDI-RCA-015 merge-side · HUDI-RCA-015 archival-side · HUDI-RCA-014 bloom index · HUDI-RCA-021 driver/timeline-server | **The Hudi frame.** `HoodieMergedLogRecordScanner` / `ExternalSpillableMap` → merge side (015a). `HoodieTimelineArchiveLog.writeToFile` / `HoodieAvroDataBlock.serializeRecords` → **archival side (015b), where adding executor memory is the wrong answer entirely** — the fix is to shrink commit metadata. Kryo / `ExternalSorter` frames under a shuffle read with no Hudi frame → bloom index (014). `Requested array size exceeds VM limit` in a driver Jetty thread → 021, a structural ceiling that more heap cannot move. | +| `FileNotFoundException` | HUDI-RCA-031 markers · HUDI-RCA-033 archiver race · HUDI-RCA-025 stale plan · HUDI-RCA-036 source file | **The path.** `MARKERS*` or `.hoodie/.temp/` → 031. A `.commit` / `.deltacommit` under `.hoodie/` → 033. A data or log file named in a compaction plan → 025. A path under the **source** bucket, not under the table → 036. A path under the table ending `.log.<n>` → treat as a compaction race, see 025's note. | + +## Cross-cutting trap: MOR defers the failure + +On Merge-on-Read an incompatible schema write **succeeds**, and compaction fails hours later in a +*different job* whose stacktrace names `HoodieCompactor` rather than the write. The table then cannot +compact and every later write fails. Cause and symptom are separated by both time and process. + +**Rule: a compaction frame plus a schema exception means the bad write happened earlier, in another +job. Investigate the writes from that window, not the job that just failed.** This applies to +HUDI-RCA-001, HUDI-RCA-003 and HUDI-RCA-004. + +--- + +# Group A — input/config + +The user asked Hudi to do something invalid, or configured it inconsistently. The remedy is to fix +data or config; nothing is wrong with Hudi or the cluster. + +--- + +## HUDI-RCA-001 — Schema evolution the writer rejects + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `org\.apache\.avro\.AvroTypeException: Cannot encode decimal with precision (\d+) as max precision (\d+)` + - `org\.apache\.parquet\.io\.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '([^']+)' not found` + - Hudi frames: `HoodieAvroUtils.rewriteRecordWithNewSchema`, `HoodieAvroUtils.rewritePrimaryTypeWithDiffSchemaType`, `HoodieMergeHelper.runMerge`, `AbstractHoodieLogRecordReader.scanInternal`, `HoodieCompactor.compact` +- **Symptom:** A write that worked yesterday fails after an upstream column was retyped or a decimal + widened. On MOR the write succeeds and **compaction** fails hours later, permanently, for every + subsequent write. +- **Root cause:** The incoming Avro schema is not a backward-compatible evolution of the table schema + under Hudi's rules. Full schema-on-read evolution is off by default, and decimal precision widening + was rejected outright until the `SchemaChangeUtils.isTypeUpdateAllow` relaxation. +- **Evidence required:** stacktrace · writer props · `hoodie.properties` (table schema) · for the MOR + variant, table state to show which file slices span the change. +- **Disambiguation:** a compaction frame here means the offending write already committed — see the + MOR deferred-failure rule above. Distinguish from HUDI-RCA-003 by *direction*: a field the table has + and the batch lacks is 003; a field whose **type** changed is 001. +- **Mitigation ladder:** + 1. Upgrade if the signature is decimal precision — fixed in 1.0.x, and no config works around it on + 0.14/0.15. + 2. Set `hoodie.datasource.write.reconcile.schema=true`, with `hoodie.schema.on.read.enable=true`. + 3. Stop dropping or retyping the column in the writer: keep it and write typed nulls. + 4. If MOR and already wedged, the table cannot compact until the pending compaction is unscheduled + or the affected file groups are rewritten. Do not hand-delete the pending instant files. +- **Governing configs:** `hoodie.datasource.write.reconcile.schema`, `hoodie.schema.on.read.enable`, + `hoodie.avro.schema.validate`, `hoodie.datasource.write.schema.allow.auto.evolution.column.drop` +- **`fixedInVersion`:** 1.0.0 for decimal precision widening · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #8040, #5452, #11335, #8020, #2919 + +--- + +## HUDI-RCA-002 — Record key or precombine field missing, null, or wrong case + +- **Category:** `input/config` · **Discoverability:** `self-evident` for the key case, `needs-expertise` for precombine +- **Signature** (innermost cause): + - `org\.apache\.hudi\.exception\.HoodieKeyException: recordKey value: "null" for field: "([^"]+)" cannot be null or empty` + - `org\.apache\.hudi\.exception\.HoodieKeyException: Record key has to be non-null!` + - `org\.apache\.hudi\.exception\.HoodieException: \w+\(Part -\w+\) field not found in record\. Acceptable fields were :\[` + - Hudi frames: `HoodieAvroUtils.getNestedFieldVal`, `ComplexKeyGenerator`, often wrapped by + `SparkException: Failed to execute user defined function` so the top frame is Spark, not Hudi. +- **Symptom:** The write fails on the first batch, or only when one composite key field is null. +- **Root cause:** Three sub-cases sharing a remedy family: the configured field does not exist or is + null in some rows; the key config is **case-sensitive** and the user wrote `ID` for a column named + `id`; or HoodieStreamer defaulted `--source-ordering-field` to `ts`, producing a phantom `ts` + requirement that then persisted into `hoodie.properties`. +- **Evidence required:** stacktrace · writer props · `hoodie.properties` +- **Disambiguation:** if the named field is one the user never configured (classically `ts`), this is + the HoodieStreamer default, not a missing column. The message names a field the user has never heard + of, which is the whole confusion — say so explicitly rather than asking them to add a `ts` column. +- **Mitigation ladder:** + 1. Check the exact field name **and case** against the dataframe schema. Hudi does not normalize case. + 2. Filter or default nulls in the key columns before writing. + 3. For HoodieStreamer with no ordering field: set `--source-ordering-field` to a real column, or use + `--op INSERT` / `bulk_insert`, which skip precombine entirely. + 4. Multi-field keys with `bulk_insert` had a genuine bug before 0.13.1 — upgrade, or switch to + `upsert` to confirm. +- **Governing configs:** `hoodie.datasource.write.recordkey.field`, + `hoodie.datasource.write.precombine.field`, `hoodie.datasource.write.keygenerator.class`, + `hoodie.table.precombine.field`, `--source-ordering-field` +- **`fixedInVersion`:** 0.13.1 for the bulk_insert multi-field key bug · **`preflightGuardExists`:** partial — the key check fires at write time, not at config time +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #3759, #9799, #11776, #10233 + +--- + +## HUDI-RCA-003 — Source drops a column the table still has + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `org\.apache\.parquet\.io\.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '([^']+)' not found` + - Frames, in order: `AvroRecordConverter.getAvroField` → `ParquetReaderIterator.hasNext` → + `HoodieMergeHelper.runMerge` + - Wrapped by `HoodieUpsertException: Error upserting bucketType UPDATE for partition :\d+` Review Comment: Correct, and the pattern is wrong about where the failure happens, not just which config to reach for. Dropping a column fails on the **driver**, before any Parquet is read: `HoodieSchemaCompatibility.checkSchemaCompatible` runs `findMissingFields` as Step 1 and throws `MissingSchemaFieldException` (`hudi-common/.../HoodieSchemaCompatibility.java:117`, and again at `:159` for the recursive path). The Parquet trace in the catalog is a pre-0.14 shape. Your second point lands harder than it looks. The catalog's step 2 points at `hoodie.avro.schema.validate`, but that gates **Step 2** of the same method — the write never reaches it, because Step 1 has already thrown. And it defaults to `false`, so the advice is inert twice over. Reworking the pattern to match on `MissingSchemaFieldException`, recommend `hoodie.write.set.null.for.missing.columns=true` (`HoodieCommonConfig.java:78`, since 0.14.1), and keep the Parquet trace as a labelled pre-0.14 variant rather than the primary signature. Worth noting for the record: `MissingSchemaFieldException extends SchemaCompatibilityException`, so matching the parent still catches it — but the specific class is the precise signal and is what the driver actually reports. ########## hudi-ai-operator/skills/hudi-rca/references/failure-catalog.md: ########## @@ -0,0 +1,1576 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +# Hudi failure catalog + +38 failure patterns, each with the signature that identifies it, the mechanism behind it, and an +ordered mitigation ladder. Used by the `hudi-rca` skill to classify a pasted failure; readable on its +own as a troubleshooting reference. + +## How to read a row + +| Field | Meaning | +|---|---| +| **ID** | Stable identifier. Never renumber; retire instead. | +| **Category** | `input/config` · `sizing/scale` · `environment` · `defect` · `not-a-failure` | +| **Discoverability** | `self-evident` (the message says what to do) · `needs-expertise` (the message is generic, misleading, or points at the wrong layer) · `silent` (no exception at all) | +| **Signature** | Exception classes, message regexes, and the Hudi frames that identify the pattern. | +| **Symptom** | What the operator sees, in their terms. | +| **Root cause** | The mechanism, in one or two sentences. | +| **Evidence required** | Which rungs of the evidence ladder are needed to reach a confident call. | +| **Disambiguation** | Present only where a signature maps to more than one pattern. The explicit discriminator. | +| **Mitigation ladder** | Ordered, cheapest and most likely first. Counter-intuitive steps are marked and justified. | +| **Governing configs** | Exact `hoodie.*` / `spark.*` keys. | +| **`fixedInVersion`** | The Hudi release that fixes the defect, where one exists. | +| **`preflightGuardExists`** | Whether a current Hudi release fails fast on the condition instead of corrupting silently. | +| **`alsoAppliesWhenHealthy`** | `true` when the same advice is an optimization on a merely-slow job, not only a fix for a broken one. | +| **Provenance** | Public `apache/hudi` issue numbers. | + +## Three rules that govern every row + +**1. Match the innermost `Caused by`, never the wrapper.** In the public tracker the wrapper classes +dominate the counts — bare `HoodieException` 498 occurrences, `HoodieUpsertException` 189 — while the +classes that actually identify the fault barely appear: `SchemaCompatabilityException` 1, +`HoodieAvroSchemaException` 2, `HoodieKeyGeneratorException` 3. Users paste the wrapper. Every regex +below is written against the innermost cause plus the first Hudi frame, because a classifier keyed on +the top-level exception class will mostly see wrappers and learn nothing. + +**2. The stacktrace is the weakest signal, not the backbone.** For a large share of these the terminal +exception is generic and the real cause is table state or config sizing several steps removed. A +`413 Payload Too Large` on timeline init is really a forgotten savepoint blocking archival. A stalled +ingestion is really a rollback that never finished and still holds the lock. Where a row says the +evidence requires table state, the honest move is to run `HoodieTableHealthChecker` rather than parse +the trace harder. + +**3. Ask the Hudi version early.** The heavy issue clusters are 0.9 through 0.15, and several +high-traffic patterns here are fixed outright in 1.0 or 1.2. For this population "upgrade" is a +genuinely good first-line remedy in a way it usually is not. Rows carry `fixedInVersion` where known. + +## Disambiguation index + +Four signatures map to more than one pattern. Resolve them here before attributing. + +| Signature | Candidates | Discriminator | +|---|---|---| +| `ConcurrentModificationException` / `HoodieWriteConflictException` | HUDI-RCA-006 healthy OCC retry **vs** HUDI-RCA-006 no lock provider | **Writer config, not the trace** — the stacktrace is identical. If `hoodie.write.lock.provider` is set and two writers are intended, this is a by-design rejection: retry the application attempt. If no lock provider is set and two writers exist, the table is being corrupted silently and the conflict exception is the lucky case. | +| `HoodieRollbackException` | HUDI-RCA-027 rollback wedge · HUDI-RCA-026 metadata-table divergence · HUDI-RCA-025 dangling files | **The frame below `rollbackFailedWrites`.** `AsyncCleanerService.waitForCompletion` → cleaner/metadata-table path (026). A lock-provider frame (`LockManager.lock`, `*LockProvider.acquireLock`) → lock problem first (006). `HoodieMergeHandle.getLatestBaseFile` with `FileID ... does not exist` → 025. | +| `OutOfMemoryError` | HUDI-RCA-015 merge-side · HUDI-RCA-015 archival-side · HUDI-RCA-014 bloom index · HUDI-RCA-021 driver/timeline-server | **The Hudi frame.** `HoodieMergedLogRecordScanner` / `ExternalSpillableMap` → merge side (015a). `HoodieTimelineArchiveLog.writeToFile` / `HoodieAvroDataBlock.serializeRecords` → **archival side (015b), where adding executor memory is the wrong answer entirely** — the fix is to shrink commit metadata. Kryo / `ExternalSorter` frames under a shuffle read with no Hudi frame → bloom index (014). `Requested array size exceeds VM limit` in a driver Jetty thread → 021, a structural ceiling that more heap cannot move. | +| `FileNotFoundException` | HUDI-RCA-031 markers · HUDI-RCA-033 archiver race · HUDI-RCA-025 stale plan · HUDI-RCA-036 source file | **The path.** `MARKERS*` or `.hoodie/.temp/` → 031. A `.commit` / `.deltacommit` under `.hoodie/` → 033. A data or log file named in a compaction plan → 025. A path under the **source** bucket, not under the table → 036. A path under the table ending `.log.<n>` → treat as a compaction race, see 025's note. | + +## Cross-cutting trap: MOR defers the failure + +On Merge-on-Read an incompatible schema write **succeeds**, and compaction fails hours later in a +*different job* whose stacktrace names `HoodieCompactor` rather than the write. The table then cannot +compact and every later write fails. Cause and symptom are separated by both time and process. + +**Rule: a compaction frame plus a schema exception means the bad write happened earlier, in another +job. Investigate the writes from that window, not the job that just failed.** This applies to +HUDI-RCA-001, HUDI-RCA-003 and HUDI-RCA-004. + +--- + +# Group A — input/config + +The user asked Hudi to do something invalid, or configured it inconsistently. The remedy is to fix +data or config; nothing is wrong with Hudi or the cluster. + +--- + +## HUDI-RCA-001 — Schema evolution the writer rejects + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `org\.apache\.avro\.AvroTypeException: Cannot encode decimal with precision (\d+) as max precision (\d+)` + - `org\.apache\.parquet\.io\.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '([^']+)' not found` + - Hudi frames: `HoodieAvroUtils.rewriteRecordWithNewSchema`, `HoodieAvroUtils.rewritePrimaryTypeWithDiffSchemaType`, `HoodieMergeHelper.runMerge`, `AbstractHoodieLogRecordReader.scanInternal`, `HoodieCompactor.compact` +- **Symptom:** A write that worked yesterday fails after an upstream column was retyped or a decimal + widened. On MOR the write succeeds and **compaction** fails hours later, permanently, for every + subsequent write. +- **Root cause:** The incoming Avro schema is not a backward-compatible evolution of the table schema + under Hudi's rules. Full schema-on-read evolution is off by default, and decimal precision widening + was rejected outright until the `SchemaChangeUtils.isTypeUpdateAllow` relaxation. +- **Evidence required:** stacktrace · writer props · `hoodie.properties` (table schema) · for the MOR + variant, table state to show which file slices span the change. +- **Disambiguation:** a compaction frame here means the offending write already committed — see the + MOR deferred-failure rule above. Distinguish from HUDI-RCA-003 by *direction*: a field the table has + and the batch lacks is 003; a field whose **type** changed is 001. +- **Mitigation ladder:** + 1. Upgrade if the signature is decimal precision — fixed in 1.0.x, and no config works around it on + 0.14/0.15. + 2. Set `hoodie.datasource.write.reconcile.schema=true`, with `hoodie.schema.on.read.enable=true`. + 3. Stop dropping or retyping the column in the writer: keep it and write typed nulls. + 4. If MOR and already wedged, the table cannot compact until the pending compaction is unscheduled + or the affected file groups are rewritten. Do not hand-delete the pending instant files. +- **Governing configs:** `hoodie.datasource.write.reconcile.schema`, `hoodie.schema.on.read.enable`, + `hoodie.avro.schema.validate`, `hoodie.datasource.write.schema.allow.auto.evolution.column.drop` +- **`fixedInVersion`:** 1.0.0 for decimal precision widening · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #8040, #5452, #11335, #8020, #2919 + +--- + +## HUDI-RCA-002 — Record key or precombine field missing, null, or wrong case + +- **Category:** `input/config` · **Discoverability:** `self-evident` for the key case, `needs-expertise` for precombine +- **Signature** (innermost cause): + - `org\.apache\.hudi\.exception\.HoodieKeyException: recordKey value: "null" for field: "([^"]+)" cannot be null or empty` + - `org\.apache\.hudi\.exception\.HoodieKeyException: Record key has to be non-null!` + - `org\.apache\.hudi\.exception\.HoodieException: \w+\(Part -\w+\) field not found in record\. Acceptable fields were :\[` + - Hudi frames: `HoodieAvroUtils.getNestedFieldVal`, `ComplexKeyGenerator`, often wrapped by + `SparkException: Failed to execute user defined function` so the top frame is Spark, not Hudi. +- **Symptom:** The write fails on the first batch, or only when one composite key field is null. +- **Root cause:** Three sub-cases sharing a remedy family: the configured field does not exist or is + null in some rows; the key config is **case-sensitive** and the user wrote `ID` for a column named + `id`; or HoodieStreamer defaulted `--source-ordering-field` to `ts`, producing a phantom `ts` + requirement that then persisted into `hoodie.properties`. +- **Evidence required:** stacktrace · writer props · `hoodie.properties` +- **Disambiguation:** if the named field is one the user never configured (classically `ts`), this is + the HoodieStreamer default, not a missing column. The message names a field the user has never heard + of, which is the whole confusion — say so explicitly rather than asking them to add a `ts` column. +- **Mitigation ladder:** + 1. Check the exact field name **and case** against the dataframe schema. Hudi does not normalize case. + 2. Filter or default nulls in the key columns before writing. + 3. For HoodieStreamer with no ordering field: set `--source-ordering-field` to a real column, or use + `--op INSERT` / `bulk_insert`, which skip precombine entirely. + 4. Multi-field keys with `bulk_insert` had a genuine bug before 0.13.1 — upgrade, or switch to + `upsert` to confirm. +- **Governing configs:** `hoodie.datasource.write.recordkey.field`, + `hoodie.datasource.write.precombine.field`, `hoodie.datasource.write.keygenerator.class`, + `hoodie.table.precombine.field`, `--source-ordering-field` +- **`fixedInVersion`:** 0.13.1 for the bulk_insert multi-field key bug · **`preflightGuardExists`:** partial — the key check fires at write time, not at config time +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #3759, #9799, #11776, #10233 + +--- + +## HUDI-RCA-003 — Source drops a column the table still has + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `org\.apache\.parquet\.io\.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '([^']+)' not found` + - Frames, in order: `AvroRecordConverter.getAvroField` → `ParquetReaderIterator.hasNext` → + `HoodieMergeHelper.runMerge` + - Wrapped by `HoodieUpsertException: Error upserting bucketType UPDATE for partition :\d+` +- **Symptom:** Every sync fails after an upstream schema change. The exception arrives on the **merge** + path while *reading existing Parquet*, so operators look at their table instead of their batch. +- **Root cause:** The incoming batch no longer carries a column the table's base files have. Hudi reads + existing records using the latest (incoming) schema, and the Parquet-to-Avro converter cannot find + the field. +- **Evidence required:** stacktrace · writer props · last commit metadata (`extraMetadata.schema`) for + the field's declared type +- **Disambiguation:** same exception class as HUDI-RCA-001's field-not-found variant. The + `HoodieMergeHelper.runMerge` frame and the fact that the field exists in the **table** but not the + **batch** is what makes it 003. Recover the field's declared type including its Avro `logicalType` + before re-adding it, or the re-add fails differently. +- **Mitigation ladder:** + 1. Re-add the column to the incoming batch as a **typed** null via a transformer — `cast(null as + string)`, or `cast(null as date)` where the logical type demands it. Cheapest and reversible. + 2. Set `hoodie.avro.schema.validate=false` only as a stopgap, not as the fix. + 3. **Config-layering trap:** if your config system overrides rather than merges list-valued settings, + adding a transformer silently drops the transformer that was already configured. Re-list all of + them, and verify the resolved config rather than the file you edited. +- **Governing configs:** `hoodie.avro.schema.validate`, + `hoodie.datasource.write.schema.allow.auto.evolution.column.drop`, + `hoodie.datasource.write.reconcile.schema` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #8040 + +--- + +## HUDI-RCA-004 — Unvalidated source schema: records do not parse against the inferred type + +- **Category:** `input/config` · **Discoverability:** `self-evident` for the message, `needs-expertise` for the fix +- **Signature** (innermost cause): + - `org\.apache\.spark\.sql\.catalyst\.util\.BadRecordException` + - `java\.lang\.RuntimeException: Failed to parse a value for data type struct<.*> \(current token: \w+\)` +- **Symptom:** Records fail to parse and the whole job fails, rather than the bad rows being isolated. +- **Root cause:** A semi-structured source (CDC, JSON, document-store feed) emits a field whose actual + shape differs from what was inferred from earlier records — most often an array where a struct was + inferred. +- **Evidence required:** stacktrace · writer props (schema-provider config) · a raw source sample +- **Mitigation ladder:** + 1. Set an explicit **target schema** instead of relying on inference, and enable validation with a + quarantine destination so bad records are isolated rather than failing the commit. + 2. For CDC sources, make sure the target schema declares the envelope fields the transformer adds + (operation type, transaction id, LSN, event timestamps) — otherwise validation rejects every record. + 3. Loosen validation only as a last resort. It converts a loud failure into a silent one. +- **Governing configs:** `hoodie.streamer.schemaprovider.target.schema.file`, + `hoodie.streamer.schemaprovider.source.schema.file`, `hoodie.avro.schema.validate` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public — the mechanism is config hygiene and rarely filed + +--- + +## HUDI-RCA-005 — Timestamp key generator cannot parse the partition field + +- **Category:** `input/config` · **Discoverability:** `self-evident` +- **Signature** (innermost cause): + - `org\.apache\.hudi\.exception\.HoodieKeyGeneratorException: Unable to parse input partition field: (.+)` + - `java\.lang\.IllegalArgumentException: Invalid format: "(.+)" is malformed at "(.+)"` + - Hudi frame: `TimestampBasedAvroKeyGenerator.getPartitionPath` +- **Symptom:** Every write fails immediately, naming the exact offending value. +- **Root cause:** The configured **input** date format does not match the actual partition-field format. + Typical triggers: microsecond precision, a trailing `Z`, or a switch from epoch to ISO-8601 upstream. +- **Evidence required:** stacktrace alone suffices · writer props to confirm the configured format +- **Mitigation ladder:** + 1. Update the key generator's **input** format config to match the value printed in the exception. + 2. Verify the **output** partition format is unchanged. Changing the output format alters partition + paths and effectively forks the table — a far worse outcome than the parse failure. + 3. If the source format is now inconsistent across records, normalize it in a transformer before the + key generator sees it. +- **Governing configs:** `hoodie.keygen.timebased.input.dateformat`, + `hoodie.keygen.timebased.output.dateformat`, `hoodie.keygen.timebased.timestamp.type`, + `hoodie.keygen.timebased.input.timezone`, `hoodie.keygen.timebased.output.timezone` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public with this exact signature; included because it is trivially fixable and + almost certainly under-reported + +--- + +## HUDI-RCA-006 — Multi-writer without, or with a misconfigured, lock provider + +- **Category:** `input/config` · **Discoverability:** `needs-expertise`, and **`silent`** in the no-lock-at-all case +- **Signature** (innermost cause): + - `java\.util\.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes`, + wrapped in `HoodieWriteConflictException`; frames + `SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict`, + `TransactionUtils.resolveWriteConflictIfAny`, `SparkRDDWriteClient.preCommit` + - `org\.apache\.hudi\.exception\.HoodieLockException: Unable to acquire lock, lock object (null|LockResponse\(lockid:\d+, state:WAITING\))`; + frames `LockManager.lock`, `TransactionManager.beginTransaction` + - `java\.lang\.IllegalArgumentException: ALREADY_ACQUIRED` at `HiveMetastoreBasedLockProvider.acquireLock` + - `org\.apache\.hadoop\.fs\.FileAlreadyExistsException: File already exists: .*\.hoodie/\d+\.commit\.requested` + at `HoodieActiveTimeline.createImmutableFileInPath` +- **Symptom:** Intermittent failures that scale with writer count. With **no** lock provider there may + be no exception at all — just overwritten timeline state and lost data. +- **Root cause:** Two writers against one table without OCC properly configured, or OCC configured and + genuinely conflicting. The semantics users most often get wrong: **the OCC lock is table-level, not + partition-level**, and it serializes only the commit-validation phase. Writers on disjoint partitions + still serialize on the lock. +- **Evidence required:** stacktrace · **writer props — mandatory** · `hoodie.properties` +- **Disambiguation (the most important row in this catalog):** the conflict exception and the no-lock + corruption produce the **same stacktrace**. Only the writer config splits them. + - `hoodie.write.lock.provider` set, concurrency intended → by-design rejection of the later committer. + Expected. Add orchestrator-level retry of the whole application attempt. Severity: routine. + - No lock provider set, but two writers demonstrably exist → the table is being corrupted and this + exception is the *lucky* case where you found out. Severity: urgent. + - Treat "is a second writer intended?" as a question to ask the user, not something to infer. +- **Mitigation ladder:** + 1. If two writers exist and no lock provider is set, set one. Non-negotiable; absence is silent + corruption, not a performance choice. + 2. **Lower** `hoodie.write.lock.wait_time_ms` to around 5000. Counter-intuitive: a long wait converts + a fast, retryable rejection into a long stall that looks like a hang, and the later writer has to + retry regardless. + 3. For DynamoDB locks, set the endpoint URL explicitly and use on-demand billing; stale lock rows + occasionally need manual removal. + 4. Add retry of the whole application attempt at the orchestrator. The later committer is rejected by + design and must retry. + 5. Where lock serialization is itself the bottleneck on MOR, consider 1.x non-blocking concurrency + control, which pairs best with a bucket index. +- **Governing configs:** `hoodie.write.concurrency.mode`, `hoodie.write.lock.provider`, + `hoodie.write.lock.wait_time_ms`, `hoodie.write.lock.num_retries`, `hoodie.write.lock.dynamodb.*`, + `hoodie.write.lock.zookeeper.*`, `hoodie.cleaner.policy.failed.writes` +- **`fixedInVersion`:** n/a — NBCC added in 1.x as an alternative mode, not a fix +- **`preflightGuardExists`:** no — Hudi does not detect "two writers, no lock" at write time +- **`alsoAppliesWhenHealthy`:** true — lock wait tuning is an optimization on a merely-slow multi-writer setup +- **Provenance:** #7653, #3731, #4456, #6226, #9213 + +--- + +## HUDI-RCA-007 — Hive or Glue catalog sync failure: table written but invisible + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `java\.lang\.RuntimeException: Unable to instantiate org\.apache\.hadoop\.hive\.ql\.metadata\.SessionHiveMetaStoreClient`; + frames `HMSDDLExecutor.<init>`, `HiveSyncTool.initClient` + - `NoSuchObjectException: .* table not found` + - Wrapped in `HoodieHiveSyncException: Got runtime exception when hive syncing` or + `Could not sync using the meta sync class` + - Log giveaway: `Trying to connect to metastore with URI thrift://localhost:9083` when the user + believes they configured Glue +- **Symptom:** Data lands on storage correctly but the query engine cannot see it, or the write fails + only in the post-commit sync step. +- **Root cause:** Sync mode and catalog wiring mismatch, Hive metastore version incompatibility, or + table-name case sensitivity. The `localhost:9083` line is the tell that the intended catalog config + never took effect. +- **Evidence required:** stacktrace · writer props · driver log (for the metastore URI line) +- **Mitigation ladder:** + 1. Lowercase the table and database names everywhere. An uppercase table name causes a rename failure. + 2. Set `hoodie.datasource.hive_sync.mode` explicitly — `hms` for Glue and HMS. + 3. For Glue, set the AWS Glue sync tool class explicitly rather than relying on inference. + 4. Verify the Hive metastore version is 2.3 or newer for Spark 3 bundles. + 5. Confirm `hoodie.datasource.hive_sync.partition_extractor_class` matches the actual partitioning — + the non-partitioned extractor for unpartitioned tables, the multi-part extractor for multi-level. +- **Governing configs:** `hoodie.datasource.hive_sync.enable`, `hoodie.datasource.hive_sync.mode`, + `hoodie.datasource.hive_sync.database`, `hoodie.datasource.hive_sync.table`, + `hoodie.datasource.hive_sync.partition_extractor_class`, `hoodie.datasource.hive_sync.partition_fields`, + `hoodie.meta.sync.client.tool.class`, `hoodie.datasource.write.hive_style_partitioning` +- **`fixedInVersion`:** 0.11.1 for the Glue client-factory regression · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #5484, #8368, #2409, #954 + +--- + +## HUDI-RCA-008 — Config changed between runs against an existing table + +- **Category:** `input/config` · **Discoverability:** `self-evident` when the guard fires, **`silent`** when it does not +- **Signature:** + - `HoodieException: Config conflict \(key, current value, existing value\): ` — when Hudi's guard + catches it + - When the guard does not catch it: **no exception**. Duplicates, no-op deletes, or an unqueryable + table. +- **Symptom:** A pipeline that ran for months starts failing, or silently producing duplicates, after + someone edited the writer config or upgraded Hudi. +- **Root cause:** Several table-level properties are written into `hoodie.properties` on first write and + are effectively **immutable**: record key field, partition path field, key generator class, table + type, precombine field, hive-style partitioning. Changing them later either trips the conflict guard + or silently reinterprets existing records. +- **Evidence required:** `hoodie.properties` · writer props. **This pattern is definitionally a diff + between the two** — no stacktrace substitutes for it, and this is why the skill requests + `hoodie.properties` by default. +- **Disambiguation:** overlaps HUDI-RCA-013 (silent duplicates). If the properties diff shows a changed + key generator or record-key field, the duplicates are 008's mechanism and 013's symptom. Report 008 — + it names the cause. +- **Mitigation ladder:** + 1. Read `.hoodie/hoodie.properties` and diff it against the current writer config. This is step one of + nearly any Hudi triage, not just this pattern. + 2. Revert the changed property to match the table. + 3. If the change is genuinely required, rewrite the table into a new path with `bulk_insert` rather + than fighting the guard. Editing `hoodie.properties` by hand is not a migration. + 4. For upgrade-induced key-encoding changes, pin the old version or rewrite. A single-field record key + under `ComplexKeyGenerator` was encoded differently from 0.14.1 onward, which silently created + duplicates and silently no-op'd deletes. +- **Governing configs:** `hoodie.table.recordkey.fields`, `hoodie.table.partition.fields`, + `hoodie.table.keygenerator.class`, `hoodie.table.type`, `hoodie.table.precombine.field`, + `hoodie.table.version`, `hoodie.datasource.write.hive_style_partitioning` +- **`fixedInVersion`:** no — the 0.14.1 key-encoding change shipped without a migration path +- **`preflightGuardExists`:** partial — the conflict guard covers some keys, not all +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #10508, #10587, #10233 + +--- + +## HUDI-RCA-009 — Wrong query type or engine wiring makes data look missing or stale + +- **Category:** `input/config` · **Discoverability:** **`silent`** +- **Signature:** **No exception.** Recognised by symptom, not by trace: an `INSERT OVERWRITE` wrote + files but `SELECT` returns old data; only a `_ro` table exists where the user expected `_rt`; + timestamps render as implausible far-future dates in the query engine. +- **Symptom:** Data is demonstrably on storage but the query returns stale, empty, or garbled results. +- **Root cause:** Query-side wiring, not write-side. Reading the read-optimized (`_ro`) view of a MOR + table and seeing pre-compaction state; missing `spark.sql.hive.convertMetastoreParquet=false`, so + Spark uses its own Parquet reader and bypasses Hudi's file-slice resolution; expecting two Hive tables + for a Copy-on-Write table, which only ever has one; or a timestamp type the catalog declares + differently from how it was written. +- **Evidence required:** `hoodie.properties` (table type) · the query and the engine · writer props +- **Proactive detection:** compare `COUNT(*)` on the snapshot view against the `_ro` view. A gap that + closes after compaction is this pattern, by design, not data loss. +- **Mitigation ladder:** + 1. Set `spark.sql.hive.convertMetastoreParquet=false`. Note that whitespace around `=` in a SQL `set` + has been enough to make this silently not apply — verify it took effect. + 2. Query the snapshot view, not `_ro`; set `hoodie.datasource.query.type=snapshot`. + 3. For MOR, confirm compaction has actually run if you need read-optimized freshness. + 4. For timestamp rendering, enable `hoodie.datasource.hive_sync.support_timestamp`. +- **Governing configs:** `hoodie.datasource.query.type`, + `hoodie.datasource.hive_sync.support_timestamp`, `spark.sql.hive.convertMetastoreParquet`, + `hoodie.datasource.write.drop.partition.columns` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #4154, #2409, #2123, #2338 + +--- + +## HUDI-RCA-010 — HoodieStreamer source or schema-provider misconfiguration + +- **Category:** `input/config` · **Discoverability:** `needs-expertise` +- **Signature:** + - `org\.apache\.hudi\.exception\.HoodieException: Commit \d+ failed and rolled-back !` at + `DeltaSync.writeToSink` / `StreamSync.writeToSink` — **this message carries no diagnostic content + whatsoever** + - `java\.lang\.NullPointerException` inside `AvroDeserializer` with no Hudi frame + - Structurally wrong data: a whole CDC envelope ingested instead of the unwrapped row +- **Symptom:** The Streamer fails with a generic "commit failed and rolled-back", or ingests + structurally wrong data. +- **Root cause:** Schema provider mismatched to source. A provider returning a null target schema while + a transformer is configured; a generic Avro source used where a CDC-specific source and payload class + are required to unwrap the envelope; Confluent wire-format Avro needing registry-aware deserialization. +- **Evidence required:** stacktrace · **Streamer CLI args** · driver log — the real exception is in the + driver log *below* the "failed and rolled-back" line, not in the thrown one +- **Disambiguation:** `Commit ... failed and rolled-back` on its own is a **content-free wrapper**. Do + not attribute from it. Ask for the driver log and classify from the exception beneath it. If the user + can only supply this line, say so and request the driver log — that is the correct outcome, not a guess. +- **Mitigation ladder:** + 1. Read the driver log below the "failed and rolled-back" line and classify the real exception. + 2. For CDC sources, use the database-specific source class with its matching payload class, not a + generic Avro source. + 3. For Confluent Avro, let the Kafka Avro source fetch from the registry rather than hand-rolling a + schema provider. + 4. If using a transformer, do not return null from the target schema provider. Set a real schema, or + omit the provider so Hudi infers from the dataframe. +- **Governing configs:** `--source-class`, `--schemaprovider-class`, `--payload-class`, + `--transformer-class`, `--source-ordering-field`, `hoodie.streamer.schemaprovider.registry.url`, + `hoodie.streamer.source.kafka.topic` +- **`fixedInVersion`:** no · **`preflightGuardExists`:** no — and the uninformative message is itself + worth an upstream report +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #2515, #2149, #2589, #8519, #10233, #5540 + +--- + +## HUDI-RCA-011 — Source initialization fails: credentials or schema registry + +- **Category:** `input/config` · **Discoverability:** `self-evident` for credentials, `needs-expertise` for the registry case +- **Signature:** Failures during writer startup, **before any batch is read**: + - `org\.apache\.kafka\.common\.errors\.\w+Exception`, SSL or keystore load failures, + connection-refused against brokers + - `org\.apache\.hudi\.utilities\.exception\.HoodieSchemaProviderException` +- **Symptom:** The job never ingests anything. No data lands, so there is no partial state to reason + about — which is itself the diagnostic. +- **Root cause:** Source credentials or TLS material are wrong or unreadable, or the source requires a + schema registry that was never configured. A common origin: a stream configuration **cloned** from an + older one created before registry support existed, carrying an empty registry reference that only + fails at writer init. +- **Evidence required:** stacktrace · writer props **as resolved, not as written** · the age of the + failure signal +- **Disambiguation:** distinguish an **init** failure from a **sync** failure. An init-failure counter + that never decrements can reflect a long-past startup failure on a driver that has since recovered. + Confirm the failure is recent before investigating it. +- **Mitigation ladder:** + 1. Fix the credential or registry configuration and restart. + 2. **Do not create a stream by cloning another** when the source type has gained required fields since + the original was created. Recreate from scratch — the clone silently carries empty required fields. + 3. Validate required source fields at creation time rather than discovering them at writer init. +- **Governing configs:** `hoodie.streamer.schemaprovider.registry.url`, + `hoodie.streamer.source.kafka.topic`, Kafka client security properties passed through + `hoodie.streamer.kafka.*` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public — onboarding-time failures are rarely filed + +--- + +## HUDI-RCA-012 — Checkpoint ahead of the source: offsets aged out or topic recreated + +- **Category:** `input/config` (persistent case) / `environment` (transient case) · **Discoverability:** **`silent`** in its most common form +- **Signature:** + - `org\.apache\.kafka\.clients\.consumer\.OffsetOutOfRangeException: Fetch position \S+ is out of range for partition` + - `Some data may have been lost because they are not available in Kafka any more` + - Or **no exception at all** — a healthy-looking job ingesting nothing +- **Symptom:** The stream looks perfectly healthy and ingests nothing. +- **Root cause — two causes with opposite correct responses:** + - **Persistent:** the topic was deleted and recreated, or retention aged past the committed + checkpoint. The checkpoint is permanently beyond the high watermark and will never recover. + - **Transient:** the broker's latest offset briefly moved backward (unclean leader election, replica + truncation, rebalance). Self-heals on the next produce. +- **Evidence required:** writer checkpoint from commit metadata · source topic offsets and metadata · + stacktrace where present +- **Disambiguation:** compare the committed checkpoint against current end offsets, then **watch the + drift for one interval**. Transient regressions close within minutes; persistent ones do not move at + all. A topic creation timestamp later than your last successful commit means recreation. Retention + settings alone do not delete a topic — if the topic is gone, something deleted it. +- **Mitigation ladder:** + 1. **Transient: do nothing.** Resetting the checkpoint here loses data for no reason. This is the + counter-intuitive step and the most common mistake on this pattern. + 2. **Persistent:** reset the checkpoint to the earliest available offset or a deliberately chosen + position, accepting that the gap is unrecoverable. Write down what was skipped. + 3. Confirm with the source owner before resetting anything. + 4. Prevent: set retention comfortably above your worst-case ingestion outage, and never delete a topic + a live stream is checkpointed against. +- **Governing configs:** `hoodie.streamer.source.kafka.topic`, + `auto.offset.reset` (the native Kafka consumer property, passed through — not a `hoodie.*` key), + `hoodie.streamer.checkpoint.provider.path` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public with this signature + +--- + +## HUDI-RCA-013 — Wrong operation or index choice produces silent duplicates + +- **Category:** `input/config` · **Discoverability:** **`silent`** — the highest-value row in the catalog +- **Signature:** **No exception.** Found only by running + `SELECT key, COUNT(*) FROM t GROUP BY key HAVING COUNT(*) > 1`, or noticed as unexplained table growth. +- **Symptom:** A primary key appears more than once. Often accompanied by table size growing far faster + than ingested volume. +- **Root cause:** A ladder of distinct mechanisms the user cannot tell apart without help: + `hoodie.datasource.write.operation` set to `insert` or `bulk_insert`, which **do not deduplicate + against existing data by design**; a non-global index with a record that moved partitions, so the key + legitimately exists twice; a global index on MOR where read-optimized queries show duplicates until + compaction runs; querying through Hive without `convertMetastoreParquet=false`; or a key-generator + encoding change across versions. +- **Evidence required:** `hoodie.properties` · writer props · the dedup query result · table type +- **Proactive detection:** `SELECT COUNT(1), COUNT(DISTINCT <record_key>)` on the **snapshot** view. + Run it after any index or key-generator change. +- **Disambiguation:** before calling this a defect, establish which mechanism applies. Cross-partition + duplicates with a non-global index are **working as configured**. Duplicates on a MOR `_ro` view that + disappear after compaction are **transient by design**. Only duplicates on the snapshot view with + `upsert` and a correct index are a real fault — and then check HUDI-RCA-008 for a key-encoding change. +- **Mitigation ladder:** + 1. Confirm `hoodie.datasource.write.operation=upsert`. This alone explains most reports. + 2. Determine whether the duplicates are cross-partition. If so, move to a global index and set + `hoodie.bloom.index.update.partition.path=true`. + 3. Re-run the dedup check against the **snapshot** view, not `_ro`, with + `convertMetastoreParquet=false`. + 4. If MOR, force a compaction and re-check — read-optimized duplicates may be transient by design. + 5. If the table crossed 0.14.1 with a single-field record key under `ComplexKeyGenerator`, compare + `_hoodie_record_key` encoding before and after: this is the known encoding regression, and the + remedy is to pin the old version or rewrite the table. +- **Governing configs:** `hoodie.datasource.write.operation`, `hoodie.index.type`, + `hoodie.bloom.index.update.partition.path`, `hoodie.datasource.write.keygenerator.class`, + `hoodie.combine.before.insert`, `hoodie.datasource.query.type` +- **`fixedInVersion`:** no · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #2338, #2255, #5777, #10508 + +--- + +# Group B — sizing/scale + +The config was right at 1,000 file groups and wrong at 500,000. Nobody's bug: the remedy is to re-tune +and add a guardrail. This is the category where a generic LLM most often gives the exactly wrong +advice, because several ladders here run *downward*. + +--- + +## HUDI-RCA-014 — Bloom index OOM from low parallelism or key skew + +- **Category:** `sizing/scale` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `java\.lang\.OutOfMemoryError: Java heap space` + - Frames are Kryo and shuffle internals with **nothing naming Hudi's index**: + `com\.esotericsoftware\.kryo\.io\.Input\.readString`, `Tuple2Serializer.read`, + `KryoDeserializationStream.readObject`, `ExternalSorter.insertAll`, `BlockStoreShuffleReader.read` +- **Symptom:** A heap OOM in a shuffle-read stage during index lookup. Nothing in the trace says "index". +- **Root cause:** The index tagging stage either has too few partitions for the key volume, or the keys + are skewed so a few tasks receive disproportionate work. +- **Evidence required:** stacktrace · writer props · **Spark event log** for stage task counts and the + p75-versus-max shuffle-read spread +- **Disambiguation:** distinguish from HUDI-RCA-015 by the frames. Kryo and `ExternalSorter` under a + shuffle read with no Hudi frame is the index stage (014); `HoodieMergedLogRecordScanner` or + `ExternalSpillableMap` is the merge side (015). Diagnose skew versus low parallelism before acting: a + wide gap between p75 and max task input is skew; a small task count with uniformly large tasks is low + parallelism. They have different fixes. +- **Mitigation ladder:** + 1. **Low parallelism:** raise input partitioning at the source first, since the index stage inherits + it. For Kafka that is `hoodie.streamer.source.kafka.minPartitions`. Only then set + `hoodie.bloom.index.parallelism` explicitly. + 2. **Lower `hoodie.bloom.index.keys.per.bucket`.** Counter-intuitive — you are shrinking the per-task + batch to produce more, smaller units of work. Raising it makes the OOM worse. + 3. **Skew with random keys:** set `hoodie.bloom.index.fileid.key.sorting.enable=true` **and** + `hoodie.bloom.index.bucketized.checking=false`. Both are required: bucketized checking stays on + unless explicitly disabled, so setting only the first changes nothing. + 4. If the OOM frame is `ExternalSorter`, lower + `spark.shuffle.spill.numElementsForceSpillThreshold` (e.g. to 100000) to force earlier spilling. + 5. Only then change index type or add executor memory. +- **Governing configs:** `hoodie.bloom.index.parallelism`, `hoodie.bloom.index.keys.per.bucket`, + `hoodie.bloom.index.fileid.key.sorting.enable`, `hoodie.bloom.index.bucketized.checking`, + `hoodie.index.type`, `spark.shuffle.spill.numElementsForceSpillThreshold` +- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** **true** — the same parallelism and skew analysis is the first move on an + index stage that merely runs slowly +- **Provenance:** #1491, #11960, #12116 + +--- + +## HUDI-RCA-015 — Executor OOM during upsert or bulk_insert at scale + +- **Category:** `sizing/scale` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `java\.lang\.OutOfMemoryError: (Java heap space|GC overhead limit exceeded|Requested array size exceeds VM limit)` + - `ExecutorLostFailure`, `Container killed by YARN`, `exit code 52` + - **(a) merge side** — Hudi frames `ExternalSpillableMap`, `HoodieMergedLogRecordScanner`, + `BoundedInMemoryExecutor` + - **(b) archival side** — frames `ByteArrayOutputStream.hugeCapacity` → + `HoodieAvroDataBlock.serializeRecords` → `HoodieTimelineArchiveLog.writeToFile` → + `HoodieTimelineArchiveLog.archiveIfRequired`; or + `java\.lang\.NullPointerException: null of string of map of union in field extraMetadata of org\.apache\.hudi\.avro\.model\.HoodieCommitMetadata` + at the same frame +- **Symptom:** The job runs fine at small volume, then OOMs persistently as the key space or log-file + volume grows — often **independent of input batch size**, which is the tell that it is *state* that + grew, not input. In the archival variant the frames are **in archival, nowhere near the user's + upsert**, which operators reasonably but wrongly read as an upsert sizing problem. +- **Root cause:** **(a)** The in-memory map used for MOR log merge exceeds what the executor heap allows, + because merge spill thresholds are sized wrongly relative to heap, or because genuine key-space scale + has outgrown the index choice. **(b)** Commit metadata files grew enormous — most often because cleaner + retention was set so aggressively that each commit records a huge number of file operations — and + archival then loads them in batches and exhausts the heap. +- **Evidence required:** stacktrace · writer props · **Spark event log** · executor logs +- **Disambiguation:** before tuning anything, classify the OOM — see ladder **L2** in + `mitigation-ladders.md`. Exit 137 with a healthy Java heap is native memory, not heap. A + `FetchFailedException` is **not memory at all**; it is ephemeral local disk, and it does not show up + in the Spark UI. Reacting to a FetchFailed by adding heap wastes a tuning cycle. Then split (a) from + (b) on the Hudi frame: a `HoodieTimelineArchiveLog` frame means the archival variant, where **adding + executor memory does not help at all** and the fix is to shrink commit metadata. +- **Mitigation ladder:** + 1. Classify the OOM first (L2), then split merge-side from archival-side on the frame. The remedies do + not overlap. + 2. **(a) merge side:** raise `hoodie.memory.merge.max.size` and `spark.executor.memoryOverhead` — + cheapest, and correct for a genuine heap-side merge OOM. + 3. Raise `hoodie.upsert.shuffle.parallelism` / `hoodie.bulkinsert.shuffle.parallelism`. **Note these + default to `0`, not 200** — `0` means "inherit the input RDD's partition count". Any advice built + on "it is 200 unless you change it" is wrong on every current release. + 4. **Lower `spark.memory.fraction` and `spark.memory.storageFraction`** if the symptom is GC pressure + rather than hard OOM. Counter-intuitive: giving Spark's managed regions *less* leaves more heap for + user objects. + 5. **(b) archival side:** lower `hoodie.commits.archival.batch` so fewer commit files are deserialized + at once, then fix whatever made commit metadata huge — usually an over-aggressive + `hoodie.cleaner.commits.retained`. Counter-intuitive: retaining *more* commits can produce *smaller* + per-commit metadata, because each clean then touches fewer files. Driver heap buys time, not a fix. + 6. At genuine scale, switch index to record-level or bucket index, enable the metadata table, and + partition the table. +- **Governing configs:** `hoodie.memory.merge.max.size`, `hoodie.memory.compaction.max.size`, + `hoodie.upsert.shuffle.parallelism`, `hoodie.bulkinsert.shuffle.parallelism`, `hoodie.index.type`, + `hoodie.metadata.enable`, `hoodie.commits.archival.batch`, `hoodie.cleaner.commits.retained`, + `spark.executor.memoryOverhead`, `spark.memory.fraction`, `spark.memory.storageFraction` +- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** **true** — the same parallelism and spill analysis applies to a slow but + succeeding upsert +- **Provenance:** #1491, #11960, #12116, #8332, #2408, #2515 + +--- + +## HUDI-RCA-016 — Write-profiling shuffle produces an unfetchable broadcast + +- **Category:** `sizing/scale` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `org\.apache\.spark\.shuffle\.MetadataFetchFailedException: Unable to deserialize broadcasted map statuses for shuffle` + - `\[INTERNAL_ERROR_BROADCAST\] Failed to get broadcast_\d+_piece\d+ of broadcast_\d+` + - Hudi frame: `BaseSparkCommitActionExecutor.buildProfile`, failing in a `countByKey` stage; wrapped + in `HoodieUpsertException: Failed to upsert` +- **Symptom:** A Spark broadcast-fetch error with nothing naming Hudi. The instinctive fix — more + parallelism — makes it worse. +- **Root cause:** Write profiling and simple-index tagging fan out with **file-group count**. The + shuffle's map-status broadcast grows with the number of shuffle partitions until Spark cannot + reliably fetch and deserialize its pieces. +- **Evidence required:** stacktrace · writer props · file-group count (`HoodieTableLayoutAnalyzer`) · + Spark event log +- **Mitigation ladder:** + 1. **Reduce `hoodie.simple.index.parallelism`.** Counter-intuitive and the whole point of this row: + fewer shuffle partitions means a smaller map-status structure to broadcast. **Raising parallelism + makes this failure worse**, which is exactly what an operator reaching for the obvious lever will do. + 2. Address file-group growth itself — tune clustering and compaction so file groups consolidate + rather than accumulate without bound. + 3. Consider an index whose tagging cost does not scale with file-group count (record-level or bucket). +- **Governing configs:** `hoodie.simple.index.parallelism`, `hoodie.index.type`, + `hoodie.clustering.inline`, `hoodie.compact.inline` +- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** **true** — file-group consolidation is an optimization long before it is a fix +- **Provenance:** none public with this exact broadcast signature + +--- + +## HUDI-RCA-017 — Record-size estimate blowup on wide payloads + +- **Category:** `sizing/scale` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause): + - `java\.lang\.IllegalArgumentException: You cannot call toBytes\(\) more than once without calling reset\(\)` + - Frames: `RunLengthBitPackingHybridEncoder.toBytes` → `ColumnWriteStoreBase.sizeCheck` → + `HoodieBaseParquetWriter.write` → `HoodieCreateHandle.doWrite` + - Often preceded by `ERROR HoodieCreateHandle : Error writing record HoodieRecord{key=...}` +- **Symptom:** A Parquet encoder assertion with nothing suggesting that the records are too wide. +- **Root cause:** Hudi's dynamic record-size estimation drives Parquet row-group sizing. Very wide + values (large string or JSON columns) break the estimate badly enough that the encoder is asked to + finalize a page twice. +- **Evidence required:** stacktrace · table schema · writer props (batch size) +- **Disambiguation:** driven by **value width, not table size** — a small table with one fat JSON column + hits this. If the job also stalls at "Building Workload Profile" first, that corroborates. +- **Mitigation ladder:** + 1. Halve the per-sync batch size and observe. + 2. If that does not clear it, **pin the estimate instead of computing it**: + `hoodie.copyonwrite.record.size.estimate` set well below the 1024 default (e.g. 200). This bypasses + dynamic estimation entirely — counter-intuitive, because the configured value is a *lower* number + than the real record size, and that is the point. + 3. Longer term, keep genuinely large blobs out of wide columns, or reduce row-group size at the source. +- **Governing configs:** `hoodie.copyonwrite.record.size.estimate`, + `hoodie.parquet.small.file.limit`, `hoodie.parquet.max.file.size` +- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** true — pinning the estimate also stabilizes file sizing on a healthy wide-schema table +- **Provenance:** none public with this signature + +--- + +## HUDI-RCA-018 — Table-service backlog: compaction, cleaning, or archival falling behind + +- **Category:** `sizing/scale` · **Discoverability:** **`silent`** early, `needs-expertise` once terminal +- **Signature:** Starts silent — growing log files, growing `.hoodie`, lengthening write times — then + surfaces as: + - `INFO No Instants to archive` + - `Not archiving as there is no compaction yet on the metadata table` + - OOM inside `HoodieTimelineArchiveLog.archive` (see HUDI-RCA-015, archival variant) +- **Symptom:** Writes get progressively slower, `.hoodie` accumulates thousands of instants, object-store + list costs climb, and eventually something fails hard. +- **Root cause:** A pending or stuck instant blocks archival; the timeline cannot archive past an + incomplete table-service instant, and the metadata table's timeline cannot archive until the data + table's does. +- **Evidence required:** **health-checker output** — this is the pattern the checker exists for · + `hoodie.properties` · writer props +- **Disambiguation:** diagnose the chain in order using ladder **L4**: archival → clean → compaction → + blocking inflight instant. Each link gates the next, and fixing the wrong link accomplishes nothing. + Note that **archival silently overrides `hoodie.keep.min.commits`** when the cleaner's + `earliestCommitToRetain` demands more retention. "I set min commits to 20 but have 400 instants" is + usually this — **fix the cleaning config, not the archival one.** +- **Mitigation ladder:** + 1. Run `HoodieTableHealthChecker --checks archival,cleaner,compaction,mdt-compaction --output JSON` + and read the chain rather than guessing which service is behind. + 2. Finish or roll back pending table-service instants on the data table first; metadata-table archival + unblocks behind it. + 3. Fix the **cleaner** configuration if `earliestCommitToRetain` is what pins the timeline. Tuning + archival while the cleaner holds the line does nothing. + 4. For MOR, ensure compaction is actually scheduled and running — inline with + `hoodie.compact.inline.max.delta.commits`, or a dedicated offline compactor. + 5. **Do not hand-delete `.requested` or `.inflight` files.** Use the CLI's unschedule path. Deleting + timeline state by hand is how a recoverable backlog becomes an unrecoverable table. +- **Governing configs:** `hoodie.keep.min.commits`, `hoodie.keep.max.commits`, + `hoodie.commits.archival.batch`, `hoodie.cleaner.commits.retained`, `hoodie.clean.automatic`, + `hoodie.archive.automatic`, `hoodie.compact.inline`, `hoodie.compact.inline.max.delta.commits` +- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** yes — `HoodieTableHealthChecker` detects this + class before it becomes terminal +- **`alsoAppliesWhenHealthy`:** **true** — service cadence tuning is the archetypal healthy-table optimization +- **Provenance:** #9478, #2515, #2020 + +--- + +## HUDI-RCA-019 — Streaming source throttled by the provider + +- **Category:** `environment` / `sizing/scale` · **Discoverability:** `needs-expertise` — the provider + exception is clear, but the correct tuning direction is the opposite of the instinct +- **Signature** (innermost cause): + - A provider throughput-exceeded exception from the source read path — on shard-based streams, + `ProvisionedThroughputExceededException` + - `org\.apache\.hudi\.exception\.HoodieIOException: Failed to list shards for \S+` during shard discovery + - Often lands in an "unknown" failure bucket because the exception is provider-specific rather than Hudi's +- **Symptom:** Ingestion throughput collapses or fails intermittently against a shard-based streaming + source. Adding capacity appears not to help. +- **Root cause:** The job is requesting records faster than the per-shard quota allows. Two things + routinely hide this: **closed shards still appear in listings but contribute zero read capacity**, so a + scale-up that lands on closed shards changes nothing; and every component reading those shards consumes + the same per-shard request quota, including background lag and monitoring probes, not just the + ingestion loop. +- **Evidence required:** stacktrace · writer props **as resolved, not as written** · shard and throughput + metrics (count of **open** shards, actual average record size) +- **Disambiguation:** before tuning, measure actual average record size and the count of **open** shards. + A throttling exception on a stream that was just scaled up usually means the new shards are closed. +- **Mitigation ladder:** + 1. Slow the poll: raise the get-records interval (e.g. to 1000 ms; provider defaults are often far + lower). Counter-intuitive, because the instinct under throttling is to retry harder. + 2. **Reduce** max records per request step-wise (e.g. 10000 → 5000 → 2000) until stable. Derive the + target from per-shard byte limit ÷ average record size ÷ polls per second. **Do not copy a number + from another deployment** — the right value is payload-specific. + 3. Add open shards, verifying they are actually open and not closed-but-listed. + 4. Reduce the frequency of auxiliary probes competing for the same shard quota. + 5. **Verify the key actually took effect** — see ladder **L12**. A misspelled Hudi config key produces + no error at all; the library default simply applies, and the intended interval is silently ignored. +- **Governing configs:** the source-specific throughput keys under `hoodie.streamer.source.*` for your + connector (records per request, poll interval) — resolve the exact key for your source and release + with the config catalog rather than assuming, and see L12 on verifying it took effect +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** **true** — poll pacing against shard quota is an ordinary throughput + optimization on a healthy stream +- **Provenance:** none public with this signature + +--- + +## HUDI-RCA-020 — Timeline bloat from a savepoint that blocks archival + +- **Category:** `input/config` (the stray savepoint) with a `defect` component (the archival gap) · + **Discoverability:** **`silent`** until terminal +- **Signature** (innermost cause): + - `org\.apache\.hudi\.exception\.HoodieRemoteException: Failed to initialise timeline remotely in server, status code: 413` + - `Payload Too Large` + - Frame: `RemoteHoodieTableFileSystemView.initialiseTimelineInRemoteView` + - Archival log line: `Setting Earliest Commit to Not Archive same as Earliest commit to retain` +- **Symptom:** The timeline grows invisibly for weeks, then ingestion breaks outright with a 413 on + timeline initialization. +- **Root cause:** **Archival stops at the earliest savepoint, not the latest.** One forgotten savepoint + pins the entire timeline after it. The active timeline grows without bound until the serialized + timeline exceeds what the embedded timeline server will accept. +- **Evidence required:** stacktrace · **health-checker output** (`--checks savepoint,archival`) · + `.hoodie` instant count +- **Disambiguation:** a 413 on timeline init is almost never a network or payload-size problem. Check + for a savepoint **first** — it is the single most commonly missed check on this signature. +- **Mitigation ladder:** + 1. Run `HoodieTableHealthChecker --checks savepoint,archival --output JSON`. The savepoint check + exists precisely for this. + 2. **Immediate unblock:** disable remote timeline initialization / the embedded timeline server so the + oversized request is never made. Minutes, and buys room to do the real fix. + 3. **Remove the stray savepoint.** This is the actual fix; archival resumes and the timeline shrinks on + its own. + 4. **Do not reach for aggressive archival scheduling first.** Counter-intuitive but load-bearing: with + the savepoint still in place, a more frequent archival job prunes nothing and costs object-store + requests. It is pointless until the savepoint is gone. + 5. Re-enable remote timeline init once the instant count is back to a few thousand. + 6. Tune archival so it is not trying to prune the whole backlog in one round. +- **Governing configs:** `hoodie.embed.timeline.server`, + `hoodie.filesystem.remote.backup.view.enable`, `hoodie.keep.min.commits`, `hoodie.keep.max.commits`, + `hoodie.archive.beyond.savepoint` +- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** yes — the `savepoint` health check +- **`alsoAppliesWhenHealthy`:** **true** — a forgotten savepoint is worth finding long before it is terminal +- **Provenance:** #9478 + +--- + +## HUDI-RCA-021 — Non-partitioned MOR table: rollback wedge and whole-table listings + +- **Category:** `sizing/scale` (a design shape, not a tuning value) · **Discoverability:** `needs-expertise`, severely +- **Signature:** A cluster of symptoms rather than one exception: + - Tasks blocked off-CPU in `HoodieAppendHandle.doAppend` → `HoodieLogFile.rollOver` → + `FSUtils.computeNextLogVersion` → `getAllLogFiles` → `listStatus` + - A `.rollback.inflight` plan file of **megabytes** that never completes and **grows** across attempts + - Driver timeline server: `OutOfMemoryError: Java heap space` and + `OutOfMemoryError: Requested array size exceeds VM limit` in Jetty threads under + `objectMapper.writeValueAsString` + - Downstream consequences: executor `SocketTimeoutException: Read timed out`, marker-create failures + (HUDI-RCA-031), writers queued behind the stuck rollback reporting lock acquisition failures +- **Symptom:** A large non-partitioned MOR table that progressively wedges, firing several unrelated-looking + alerts at once. +- **Root cause:** Two mechanisms compounding. MOR failed-write rollback uses the **listing** strategy, + enumerating every file in the partition — and on a non-partitioned table the "partition" is the whole + table, so rollback cost scales with total file-group count rather than with the failed instant. + Separately, the timeline server serializes the entire file-system view into a single JSON string, + which on a whole-table view hits the JVM's roughly 2 GB single-array ceiling. That is a **structural** + limit, not a heap-size one. +- **Evidence required:** stacktrace · `hoodie.properties` (table type, partitioning) · layout analysis + (file-group count) · `.rollback.inflight` plan sizes · driver log · Spark event log +- **Disambiguation:** note that **rolling back a completed instant ignores `hoodie.rollback.using.markers` + entirely** — the config is ANDed with "instant is not completed", so restores and savepoint rollbacks + always take the listing path regardless of how that config is set. This is the mechanism behind "my + rollback took longer than the write", and it is why setting the marker config does not help here. +- **Mitigation ladder:** + 1. **Do not restart the driver while a rollback is in flight.** Each restart abandons it and re-requests + a larger one. The plan file growing across attempts is the proof. + 2. **Do not disable compaction to dodge a write conflict.** It stops the conflict and lets log files + pile up per file group without bound, making both the per-append listing and every future rollback + slower. This is the single most damaging available move. + 3. Move the file-system view off heap: `hoodie.filesystem.view.type=SPILLABLE_DISK`, with + `hoodie.filesystem.view.spillable.mem` sized around 1 GB and + `hoodie.common.spillable.diskmap.type=ROCKS_DB`. + 4. Raise `hoodie.markers.timeline_server_based.batch.num_threads` so marker traffic is not the next + bottleneck. + 5. Size compaction for the workload: `hoodie.compaction.target.io` must be large enough that one round + sweeps most candidate file groups. If only half fit per round, a tail accumulates forever. Raise it + substantially rather than nudging it. + 6. Give the driver more heap only after 3 through 5 — for the JSON serialization path this is a ceiling + problem, not a heap problem. + 7. Prefer partitioning the table. A non-partitioned MOR table is pathological for both the listing-based + rollback path and the whole-table file-system-view path. +- **Governing configs:** `hoodie.filesystem.view.type`, `hoodie.filesystem.view.spillable.mem`, + `hoodie.common.spillable.diskmap.type`, `hoodie.markers.timeline_server_based.batch.num_threads`, + `hoodie.compaction.target.io`, `hoodie.rollback.using.markers` +- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** partial — layout analysis surfaces the shape +- **`alsoAppliesWhenHealthy`:** **true** — the view and compaction sizing are optimizations well before the wedge +- **Provenance:** none public with this combination + +--- + +## HUDI-RCA-022 — Unbounded incremental clustering plan wedges everything behind it + +- **Category:** `sizing/scale` with a `defect` component · **Discoverability:** `needs-expertise` — several + alerts fire, none of which names clustering +- **Signature:** + - A `.replacecommit.requested` whose **plan file is tens to hundreds of MB**, `.inflight` for many hours + - Driver `java\.lang\.OutOfMemoryError: GC overhead limit exceeded`, with a heap dump showing many + duplicated copies of the same large plan byte array across worker threads + - Metadata-table compaction stops scheduling entirely; MDT log blocks accumulate on the `files` partition + - Marker CREATE timeouts, connection-pool shutdown, container OOM-kills where the Java heap looks healthy +- **Symptom:** A multi-day outage on a heavily-partitioned append-only table, announced by half a dozen + symptoms that each look like a different problem. +- **Root cause:** The incremental clustering planner caps group count and total bytes but **not input + file-slice count**. On a heavily-partitioned append-only table where clustering has fallen behind, it + accumulates pending partitions and tries to catch up all at once. The plan is then re-read and fully + deserialized **on every timeline refresh**, just to answer "is this a clustering instant?", so each + worker thread pins its own copy. +- **Evidence required:** table state (`.replacecommit.requested` age and plan size, decoded plan) · + heap dump · Spark event log +- **Disambiguation:** the giveaway is plan **file size**. Over a few MB is already a warning. If ingestion + writes roughly one file group per commit and the pending plan proposes hundreds of thousands of file + slices, the plan is wildly disproportionate and is the cause, not a victim. +- **Mitigation ladder:** + 1. **Capture heap dumps from the driver and a few executors before restarting anything.** Restarting + destroys the only evidence and does not clear the wedge. + 2. **Disable clustering entirely** and let ingestion and metadata-table compaction drain. This is the + only reliable break. Counter-intuitive: throttling clustering while the giant plan is still pending + does not help, because the cost is in deserializing the existing plan, not in making new ones. + 3. Re-enable with a reduced per-round group count and a cap on input file groups per plan where your + release supports it. + 4. If executors are container-OOM-killed with a healthy Java heap, suspect **native** memory from + compression codecs: raise `spark.executor.memoryOverheadFactor`. + 5. Keep clustering cadence ahead of partition creation so a backlog never forms. +- **Governing configs:** `hoodie.clustering.inline`, `hoodie.clustering.async.enabled`, + `hoodie.clustering.plan.strategy.max.num.groups`, + `hoodie.clustering.plan.strategy.max.bytes.per.group`, `spark.executor.memoryOverheadFactor` +- **`fixedInVersion`:** n/a — the input-slice-count cap is the missing guard +- **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** **true** — clustering cadence versus partition creation rate is a healthy-table check +- **Provenance:** none public with this mechanism + +--- + +## HUDI-RCA-023 — Long-lived driver leaks spillable file-system-view resources + +- **Category:** `defect` surfacing as `sizing/scale` · **Discoverability:** **`silent`** +- **Signature (a heap dump, not a stacktrace):** + - Leak suspect: `java\.lang\.ApplicationShutdownHooks` retaining the bulk of heap via tens of thousands + of shutdown-hook threads + - `BitCaskDiskMap` and `HoodieFileGroupId` dominating the object histogram + - Externally: a busy-thread dump where nearly all top-CPU threads are GC threads +- **Symptom:** No exception. The driver stays "Ready", ingests in bursts, and degrades. Several different + alerts fire, none naming the cause. +- **Root cause:** With `SPILLABLE_DISK` file-system views, each spillable map registers a JVM shutdown + hook. Table-service cycles rebuild the view on every full refresh, and replaced maps are abandoned + without being closed, so the hooks are never deregistered and pin the maps for the life of the JVM. + Leak volume scales with tables multiplied by table-service cycles. +- **Evidence required:** **heap dump — a stacktrace is useless here** · writer props (view type) · + driver uptime +- **Disambiguation:** note the direct tension with HUDI-RCA-021, which prescribes `SPILLABLE_DISK` for a + different reason. Apply the view type **per table**, not globally: a table with a very large view needs + spillable; a long-lived driver hosting many small tables does not. +- **Mitigation ladder:** + 1. Set `hoodie.filesystem.view.type=MEMORY` for the dominant table — this is Hudi's own default and + removes the leaking path. + 2. Restart the driver to clear accumulated hooks — **after** capturing dumps, not before. + 3. Shorten driver lifetime or reduce tables per driver as a stopgap. + 4. Upgrade to a release where the table object is closed per cycle. +- **Governing configs:** `hoodie.filesystem.view.type`, `hoodie.filesystem.view.spillable.mem`, + `hoodie.common.spillable.diskmap.type` +- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** none public — requires heap-dump forensics on a long-running multi-table driver + +--- + +## HUDI-RCA-024 — Object-store connection pool exhausted + +- **Category:** `sizing/scale` · **Discoverability:** `needs-expertise` +- **Signature** (innermost cause — **both parts required**): + - An SDK client exception, **and** the text `Timeout waiting for connection from pool` + - Wrapped in `org\.apache\.hudi\.exception\.HoodieIOException: getFileStatus on ` + - Frame: the filesystem connector's exception-translation utility +- **Symptom:** Intermittent to persistent IO failures under high file-listing or write concurrency, + reading like a storage outage. +- **Root cause:** The filesystem client's HTTP connection pool is smaller than the concurrency the job + drives. More executors and more threads make it worse. +- **Evidence required:** stacktrace · writer props and Hadoop config · executor count +- **Disambiguation:** the match requires **both** the SDK client exception **and** the pool-timeout text. + Other SDK errors from the same connector are a different problem entirely — do not match on the SDK + exception class alone. +- **Mitigation ladder:** + 1. Double the connection-pool maximum (`fs.s3a.connection.maximum` on the S3A connector) and restart. + Repeat once if needed. + 2. Above roughly 2000, **stop doubling and think.** Per-pod connection counts compound across executors, + and you can exhaust the host or the endpoint instead of fixing anything. + 3. **Reduce job-side concurrency** — fewer executors, lower listing parallelism. Counter-intuitive, but + it attacks the demand side rather than the supply side, and the demand side is the one that scales. + 4. Reduce the number of file-listing operations outright: enable the metadata table, partition better. +- **Governing configs:** `fs.s3a.connection.maximum` (or the equivalent for your connector), + `hoodie.metadata.enable`, `hoodie.file.listing.parallelism` +- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no +- **`alsoAppliesWhenHealthy`:** **true** — listing reduction is an optimization on any healthy job at scale +- **Provenance:** none public with this exact match rule + +--- + +# Group C — defects and table state + +Hudi reached a state it should not have. Remedies are to pin or upgrade a version, rebuild derived +state, or — in one case only — run a supervised repair. **Attribution matters here:** these are not the +user's mistake, and saying so plainly is part of the job. + +--- + +## HUDI-RCA-025 — Dangling data files from an incomplete rollback — EXPLAIN ONLY + +> **⚠️ EXPLAIN-ONLY ROW — HARD CONSTRAINT FOR THE AGENT AND FOR FUTURE EDITORS ⚠️** +> +> The agent **identifies** this pattern, **names it as a known defect class rather than the user's +> mistake**, points at `HoodieTableHealthChecker`, and **states that repair requires the safe-deletion +> protocol (ladder L3) with a human driving it.** +> +> **The agent must never recommend, script, enumerate, or sequence the file deletion.** Not as a +> suggestion, not as an example, not "for reference", not even when the user asks directly. The correct +> response to "just tell me which files to delete" is to restate that the deletion premise needs two +> independent verifications and a savepoint, and to hand over L3 for a human to execute. +> +> This row is the template for any future pattern whose only remedy is destructive. If you are adding +> such a pattern, copy this block. + +- **Category:** `defect` · **Discoverability:** `needs-expertise` — and actively misleading +- **Signature** (innermost cause): + - `java\.util\.NoSuchElementException: FileID \S+ of partition path \S+ does not exist` + - Frames: `HoodieMergeHandle.getLatestBaseFile` → `BaseSparkCommitActionExecutor.getUpdateHandle` + - In the driver log beforehand: `BaseRollbackActionExecutor : Rollback of Commits [<instant>] is complete` +- **Symptom:** Ingestion fails on **every** attempt after a prior write failure, deterministically, on + the same file ID across restarts. +- **Root cause:** A commit failed and was rolled back, but its data files and markers were never + physically deleted. A later operation resolves a file slice to an untracked file that has no timeline + entry. **The message says a file "does not exist", but the actual condition is an *extra*, untracked + file** — which is why it reads like user error or data loss when it is neither. +- **Evidence required:** stacktrace · **health-checker output** · table state (partition listing, + `.hoodie`, `.hoodie/.temp/`) · driver log for the rollback-complete line +- **Disambiguation:** shares `FileNotFoundException`-family symptoms with several rows — see the + disambiguation index. A missing path under the **source** bucket is HUDI-RCA-036. A path under the + table ending `.log.<n>` indicates a deltacommit that raced an unfinished compaction, which is a + different mechanism with a different (also destructive, also human-driven) remedy. Confirm all of: + the file exists on storage, its instant is **not** in the active timeline, and its instant **is** + named by a later `.rollback`. +- **How this is explained to the user:** + 1. Name it: this is a known timeline-consistency defect class, not a configuration mistake. The error + message is misleading by design-accident. + 2. Point at `HoodieTableHealthChecker` to characterize the table's current state. + 3. State that repair means removing untracked files, that this is destructive and irreversible, and + that it must run inside the **safe-deletion protocol (L3)** — stakeholder notice, writer paused, + savepoint taken, **two independent verifications of the deletion premise**, backup copy, then delete. + 4. State that a **human must drive it**, and that if any verification is ambiguous the correct action + is to stop: deleting a non-orphan corrupts the table silently. + 5. Where the release supports it, note that `rollback` or `restore` to the last clean commit is an + alternative to hand-deletion — it rewrites the timeline, so it is also expert-only and also L3. + 6. Upgrade: the deltacommit-removed-while-requested-remained trigger is fixed in later releases, so + establishing the version is worthwhile before anything else. +- **Governing configs:** `hoodie.cleaner.policy.failed.writes`, `hoodie.rollback.using.markers`, + `hoodie.filesystem.view.incr.timeline.sync.enable` (experimental; if it is on, that alone can produce + plans naming wrong filenames — turn it off) +- **`fixedInVersion`:** 1.0.x for the requested/inflight-orphan trigger · **`preflightGuardExists`:** partial +- **`alsoAppliesWhenHealthy`:** false +- **Provenance:** #11202, #2020, #19290 + +--- + +## HUDI-RCA-026 — Metadata table disagrees with the filesystem + +- **Category:** `defect` (data correctness) · **Discoverability:** **`silent`** — reads can return wrong or + missing files with no write failure at all +- **Signature:** Validator output from `HoodieMetadataTableValidator`: + - `Validation of latest file slices for partition \S+ failed` + - `File system-based listing: \d+ & MDT-based listing: \d+` + - `Validation of record index content failed` + - May instead surface as `HoodieMetadataException: Failed to retrieve files in partition`, or as + `Invalid number of file groups for partition:column_stats` +- **Symptom:** Queries return wrong or missing files, or an offline operation cannot list partitions. + Often no write failure at all. +- **Root cause:** The metadata table diverged from the data table. Known triggers: enabling column-stats + or bloom-filter indexes mid-life on an existing table; duplicate concurrent writers caused by + `spark.speculation=true`, which could trigger a record-index purge; a deltacommit deleted while its + requested/inflight state remained. +- **Evidence required:** validator output · **both timelines, active and archived** · writer props + (`hoodie.cleaner.policy.failed.writes`, `hoodie.write.concurrency.mode`, + `hoodie.write.lock.provider` — these gate which rollback path can fire) +- **Disambiguation — classify the direction first, per table, before touching anything:** + + | Direction | Meaning | Response | + |---|---|---| + | **MDT behind** | Storage has slices the metadata table lacks | Rebuild the metadata table | + | **MDT ahead** | The metadata table lists slices storage no longer has | **A rebuild does not fix this** — it re-derives from a data table whose storage still holds orphans, so the divergence returns | + | **Per-slice** | Same count, but a file group resolves to competing base files differing only in write token | HUDI-RCA-028 | + + One validator run can cover several tables diverging in **opposite** directions from the same defect. + Interpreting the timeline evidence: a `.clean` naming the file means the cleaner removed it logically + while the object survived. A data-side `.rollback` whose rolled-back instant **is** the divergent base + instant, with no filename hit, means the commit was rolled back and the objects are orphan task-attempt + files — **the metadata table is correct, do not rebuild.** A `.replacecommit` means clustering replaced + the slice, and replaced files leave only when a later clean deletes them. +- **Mitigation ladder:** + 1. **Archive the evidence first.** Both branches overwrite the diverged state; once you rebuild, the + evidence is gone. + 2. Confirm the signal is real before investigating at all — see ladder **L6**. Several tables failing + the same check within minutes usually means the *check* changed, not the data. + 3. **Branch A, metadata table is wrong:** rebuild. Set `hoodie.metadata.enable=false`, let one write + proceed, re-enable so the `files` partition rebuilds from scratch, then re-run the validator. Review Comment: Agreed, and this wording appears on #20154 as well — you flagged it there too. Fixing both. `hoodie.metadata.auto.delete.partitions` defaults to `true`, so "set `hoodie.metadata.enable=false`, let one write proceed, re-enable" deletes the entire metadata table via `HoodieTable.maybeDeleteMetadataTable` — record index, column stats, bloom filters and secondary indexes, not just the files partition. And it is silent about it. Replacing the step with explicit `metadata delete` then `metadata create` in hudi-cli, which at least says what it is doing, behind a human gate, with the dropped indexes named and the requirement to stop other writers stated. ########## hudi-ai-operator/skills/hudi-rca/references/mitigation-ladders.md: ########## @@ -0,0 +1,311 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +# Mitigation ladders + +Thirteen operational procedures, each ordered. They are not tied to one failure signature — they are +the discipline that makes the difference between diagnosing a Hudi failure and making it worse. +`failure-catalog.md` rows reference them by number. + +Several of these say **do nothing**, **do not restart**, or **turn the number down**. Those steps are +not hedges. They are the steps most often got wrong, and the reason is usually that the obvious lever +points the other way. + +**The single most expensive mistake available is restarting before capturing evidence.** That is L9, +and it earns its place by frequency. + +--- + +## L1 — Locate the real exception in a multi-table writer + +When one driver writes many tables, the trace you find first is often not the trace you need. + +1. Search driver logs for `ERROR`, then search **within those results** for the table name. Keep + scrolling — one polling cycle can contain several tables' traces interleaved. +2. **Save the full trace somewhere durable the moment you find it.** Re-finding it is expensive, and + log rotation does not wait. +3. If you cannot attribute the failure to a table, temporarily move that table to a **single-table** + writer. One table per job makes the trace unambiguous. Move it back afterwards. +4. If the first pass (ERROR only) returns nothing, widen to INFO and WARN. Some stalls produce **no + ERROR line at all** — see L8 and HUDI-RCA-038. +5. Only then start root-causing. + +--- + +## L2 — Classify a Spark OOM before reacting + +An OOM is not one failure mode. These six have genuinely different remedies, and reaching for heap +first is right in only one of them. + +1. **Container OOM-kill (exit 137), Java heap fine** → native or off-heap memory, not heap. Raise + `spark.executor.memoryOverheadFactor` (commonly to 0.35, up to around 0.5). Suspect compression + codecs and shuffle transfer. +2. **JVM heap OOM (exit 52)** → a heap dump is usually written automatically. **Get it.** This is a + code or data-shape problem, not a sizing problem: memory buys time, not a fix. +3. **`OutOfDirectMemoryError` / `Direct buffer memory`** → raise overhead and direct-buffer memory. As + an emergency measure only, stop Spark preferring direct buffers for shuffle transfer, at a + throughput cost. **Do not leave that on.** +4. **Evicted mid-stage with `FetchFailedException` on shuffle files** → **this is not memory at all.** + It is **ephemeral local storage**. Evicted executors delete their shuffle files, surviving tasks + fail to fetch, Spark retries the whole stage, writes more shuffle, and evicts more executors. Raise + the executors' local-disk request and limit. **This does not show up in the Spark UI** — check pod + and node events instead. Treating a FetchFailed as a memory problem is a wasted tuning cycle, and + it is the most commonly misread entry in this list. +5. **Driver OOM requesting a large array** → the JVM cannot allocate a single contiguous array above + roughly 2 GB. This is a **structural ceiling**; more heap will not help. See HUDI-RCA-021. +6. **A few partitions consistently OOM while others are fine** → skew, or large rows. Check per-record + size in the stage detail. Above roughly 1 MB average, lower + `spark.shuffle.spill.numElementsForceSpillThreshold`. +7. **GC pressure rather than hard OOM** → **reduce** `spark.memory.fraction` and + `spark.memory.storageFraction`. Counter-intuitive: giving Spark's managed regions *less* leaves + more heap for user objects. Confirm G1GC and enable GC logging before tuning further. +8. Check for a pathological CPU-to-memory ratio. Powerful CPUs paired with small memory cause this + repeatedly, and no Hudi config will fix it. + +--- + +## L3 — Safe-deletion protocol + +**Never delete a file from a Hudi table outside this envelope.** Nine steps, in order, no exceptions. + +1. Inform stakeholders. +2. **Pause the writer.** +3. **Take a savepoint.** +4. Verify the deletion premise with **two independent checks** — for example, the file is untracked in + commit metadata **and** its instant is named by a completed rollback. One check is not enough. +5. **Copy the files you intend to delete to a backup prefix.** +6. Delete. +7. Validate: re-run the failing operation, or the validator. +8. Resume the writer. +9. After a few successful commits, pause, **delete the savepoint**, resume. + +**If any verification is ambiguous, stop.** Deleting a non-orphan corrupts the table silently — there +is no error, and you will find out later from query results. The ambiguity is the signal, not an +obstacle to work around. + +A human drives this procedure. An agent may explain it, point at the evidence that would satisfy +step 4, and say which files it believes are implicated — but **an agent must not perform, script, or +sequence the deletion.** See the explain-only block on HUDI-RCA-025. Review Comment: You are right that the rule is applied inconsistently, which is worse than not having it. `mitigation-ladders.md:102` lets the agent "say which files it believes are implicated" while SKILL.md L204 and RCA-025 say never to list files. The ladder wins by being the more specific instruction at the point of use, so the looser wording is the one to remove. RCA-020 step 3 is the same class of problem: "remove the stray savepoint" is a durable, unrecoverable action sitting without a gate, when the reversible option exists. Though not `hoodie.archive.beyond.savepoint=true` — that one is not fully supported and archival will not move past the first savepoint regardless, so recommending it would trade one bad instruction for another. The gate plus an explicit statement of what deleting a savepoint costs is the right shape. Will go through every ladder for the same pattern rather than fixing the two you found. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
