deepakpanda93 commented on issue #9918: URL: https://github.com/apache/hudi/issues/9918#issuecomment-4845452037
This is resolved — the root cause is Spark speculative execution, and Hudi now guards against it as of 1.2.0. As @watermelon12138 correctly diagnosed, the broken/zero-length parquet (is not a Parquet file (too small length: 0) / bad magic number) comes from speculative execution duplicating the compaction/write task: when the speculative attempt is killed mid-write, it leaves a corrupt file alongside the good one (same file-group id + instant), and the reader then trips on the corrupt one. This is tracked in [#9615](https://github.com/apache/hudi/issues/9615) / [#16220](https://github.com/apache/hudi/issues/16220). Fix: [#18045](https://github.com/apache/hudi/pull/18045) adds a guardrail that prevents the Spark write client from running when spark.speculation=true, so Hudi fails fast instead of silently producing broken files — in release-1.2.0. (A related guard to fail on duplicate data files during marker reconciliation landed in 1.0.2, HUDI-8967 [#12792](https://github.com/apache/hudi/pull/12792).) For now / older releases: set spark.speculation=false for your Hudi write jobs — that eliminates the issue. Any already-corrupted files will need a one-time manual cleanup. Closing as resolved (duplicate of [#9615](https://github.com/apache/hudi/issues/9615) / [#16220](https://github.com/apache/hudi/issues/16220), fixed by [#18045](https://github.com/apache/hudi/pull/18045) in 1.2.0). Thanks @watermelon12138 for the precise root-cause analysis! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
