deepakpanda93 commented on issue #9977: URL: https://github.com/apache/hudi/issues/9977#issuecomment-4845571051
This is fixed in Hudi 0.13.0+ (and all 1.x). Root cause: with >100 columns, Spark disables whole-stage codegen (spark.sql.codegen.maxFields, default 100). On that path the MOR / HoodieSparkUtils.createRdd iterator was non-idempotent and got consumed more than once, silently dropping records during clustering — which is why exactly half the rows of the touched base file went missing. (As @ad1happy2go found, raising spark.sql.codegen.maxFields above your column count avoids it.) Fix: [PR #7334](https://github.com/apache/hudi/pull/7334) ("Rebased MOR iterators onto a CachingIterator to be idempotent"), related issue [#8416](https://github.com/apache/hudi/issues/8416). This landed in 0.13.0 and is in all later releases including 1.x. You hit it on 0.12.2-amzn, which predates the fix — and it was not backported to the 0.12.x line, hence the need for the spark.sql.codegen.maxFields workaround there. Upgrading to a recent 1.x release resolves it without that workaround. Closing as fixed in 0.13.0+. If you ever see row loss after clustering on 1.x, please reopen with a repro. Thanks @mzheng-plaid for the excellent investigation and reproducible test case! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
