Raghuramchadalawada opened a new pull request, #20336: URL: https://github.com/apache/hudi/pull/20336
### Describe the issue this Pull Request addresses Closes #18915 Continuation of #18914, which I opened from my previous employer's GitHub account and no longer have access to. Same commits, rebased onto current `master`. Please see #18914 for earlier review discussion. ### Summary and Changelog Custom JAR transformers that do column projection (e.g. `ColumnFilter` with `mode=include`) drop `_corrupt_record` because they are unaware of the error-table contract. `ErrorTableAwareChainedTransformer` called `validate()` after every transformer, so any such transformer failed the pipeline with `HoodieValidationException`. - Before each transformer runs, stash the `_corrupt_record` column (cached and materialized so the stash isn't recomputed from a possibly non-deterministic lineage). - If the transformer drops the column, restore the original values onto its output via a positional RDD zip (`ErrorTableUtils.restoreCorruptRecordColumn`). The zip is evaluated eagerly; if it fails (row count or partitioning changed), log a WARN and fall back to re-adding the column as null. - Remove the now-unused `ErrorTableUtils.validate()` and make `isErrorTableCorruptRecordColumnPresent()` public. - Tests in `TestErrorTableAwareChainedTransformer`: `_corrupt_record` values survive a transformer that drops the column, a column-projecting transformer keeps the column (null values), and a transformer that keeps the column is unaffected. ### Impact No public API change. Behavior change: transformers that drop `_corrupt_record` no longer fail the pipeline. Its values are carried through when the transformer only projects columns; otherwise the column is re-added as null. Known limitation: a transformer that reorders rows without changing the count (e.g. `orderBy`) zips successfully but with misaligned values. This is documented in the Javadoc. ### Risk Level low. The change only affects the case that previously threw `HoodieValidationException`; transformers that keep `_corrupt_record` follow the same path as before, except for the extra cached single-column stash per transformer. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
