Davis-Zhang-Onehouse opened a new pull request, #20709: URL: https://github.com/apache/hudi/pull/20709
### Describe the issue this Pull Request addresses issue: #20708 Part of #20683, RFC-118 (#20684). Stacked on #20707: review only the last commit. Until the PRs below it merge, this PR also shows their commits. Rollback of out-of-line blob files by markers, copy failure classification, the inline ceiling and Parquet settings for large binary columns. A failed blob round no longer leaves unreferenced copies: each out-of-line file gets a marker of the round's instant, and the round is moved to INFLIGHT before the copy so a rollback deletes them. The copy retry and failure classes are removed again by follow-up sub-issue 37 of the plan; marker rollback stays for the embedding table's stored inputs. ### Summary and Changelog - New `hoodie.write.out.of.line.markers.enable` (default true) in `HoodieWriteConfig`: each out-of-line file under `.hoodie/.blob/` gets a timeline-server CREATE marker of the write instant, and a rollback of a failed write deletes it. When on, the writer requires timeline-server markers and marker-based rollback. `HoodieTable` leaves managed blob paths out of the data-file write stats. - `StreamSync` / `HoodieStreamerConfig`: the round's instant can go INFLIGHT before the transformer runs, so files a transformer writes follow the marker protocol and are rolled back with a failed round. Off unless a source contract asks for it. - Blob copy: retry settings, copy failure classes for source and row-building failures, GCS access-denial handling and handle release. - Inline ceiling for blob rows and Parquet settings for large binary columns: `HoodieBaseParquetWriter` honours per-column `parquet.column.statistics.enabled#<col>` (Parquet 1.14+, refused on older Parquet) and `parquet.enable.dictionary#<col>`, explicit page-size-check cadence (skipped with a warning before Parquet 1.12.3), and an internal size check on every record. `HoodieInternalRowFileWriterFactory` applies the configured Parquet config injector on the row-writer path, as the record writer already does. - 33 test classes, including failed-copy cleanup, the early-instant contract and Parquet page bounds. Reader, projection and listener changes for BLOB reads are in #20688, not here. No code was copied from other projects. ### Impact One new advanced write config (`hoodie.write.out.of.line.markers.enable`, default true) that only affects writers creating out-of-line files. `HoodieBaseParquetWriter` gains opt-in per-column keys; writers that do not set them keep their current behaviour. Row-writer bulk inserts and clustering now honour a configured Parquet config injector. ### Risk Level high: touches `StreamSync`, `HoodieTable`, `HoodieWriteConfig` and the shared Parquet writers. The new behaviour is opt-in or limited to out-of-line files; default paths are unchanged. Verified on JDK 11 / Spark 3.5 / Scala 2.12 with 309 tests across hudi-hadoop-common, hudi-spark and hudi-utilities (none skipped), module compilation, checkstyle and RAT. A failed-copy cleanup test fails without the marker rollback and passes with it. ### Documentation Update The new write config and the blob settings carry their config documentation. RFC-118 describes rollback; a website page follows once the chain lands. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
