deepakpanda93 commented on issue #8259: URL: https://github.com/apache/hudi/issues/8259#issuecomment-4844793208
This is fixed in 0.14.1+ / 1.x. Root cause: clustering with the row writer + hive-style partitioning wrote the partition column at the end of the base-file schema, while the table schema kept it in the middle. The COW upsert merge then read that base file positionally against the table schema, so it tried to decode the partition int column as binary — hence UnsupportedOperationException: PlainLongDictionary.decodeToBinary / ParquetDecodingException. Fix: the merge path no longer reads positionally. HoodieMergeHelper now reads the base file with its own schema and rewrites each record to the writer schema by field name (rewriteRecordWithNewSchema), so a clustered file with a reordered partition column is handled correctly. This is reinforced by the engine-agnostic FileGroup reader (default in 1.0+, also name-based). Related row-writer/partition-column fixes landed in 0.14.1 (HUDI-7040 [#10272](https://github.com/apache/hudi/pull/10272), HUDI-6926 [#9840](https://github.com/apache/hudi/pull/9840)). The reproductions here were on 0.12.x–0.14.0, which predate those fixes (on older releases the workaround was hoodie.datasource.write.row.writer.enable=false). Closing as fixed in 1.x. If you can still reproduce table corruption after clustering on a recent 1.x release, please reopen with a minimal repro and we'll investigate. Thanks @kazdy for the very thorough analysis and repro repo! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
