cshuo opened a new pull request, #19576: URL: https://github.com/apache/hudi/pull/19576
### Describe the issue this Pull Request addresses Closes #19437. Closes #19438. Spark LSM-layout tables need the standard HoodieRecord/RDD write operations, MOR compaction, bulk insert, and Dataset Row writer overwrite paths to preserve the table's sorted-run invariant. The bulk-insert paths bypass the standard commit partitioning path, so they require layout-aware partitioners and explicit rejection of configurations whose record-key ordering cannot be verified. ### Summary and Changelog - Cover insert, upsert, delete, prepared operations, partition/table overwrite, delete-partition, and MOR compaction behavior for LSM-layout COW and MOR tables. - Add separate LSM bulk-insert partitioners for HoodieRecord and Dataset Row paths, preserving the supported distribution semantics while ordering records by UTF-8 partition path and record key. - Route LSM construction through the existing table-aware partitioner factory entry points and reject non-sorting modes or user-defined partitioners whose ordering cannot be guaranteed. - Require populated metadata fields for direct LSM Dataset Row partitioning and keep the HoodieRecord fallback available through the DataSource path. - Add client, DataSource, partitioner, and SQL coverage that verifies snapshot results, replace metadata, table layout, physical base-file ordering, supported sort modes, and failure behavior. ### Impact LSM-layout Spark tables can use the covered COW and MOR write operations, including bulk insert and Row writer overwrite paths, without weakening the sorted-file invariant. There are no new public APIs or configuration options. LSM bulk insert now rejects non-sorting modes and custom partitioners because their output ordering cannot be verified. Supported modes retain their existing distribution behavior and incur the sorting or shuffle cost already implied by the selected mode. ### Risk Level Medium. The change affects Spark write partitioning and distributed ordering for LSM tables. Risk is limited to the LSM layout and is mitigated by targeted COW/MOR client and DataSource coverage, SQL writer coverage, and partition-level ordering assertions. Validation run includes 13/13 `TestLSMBulkInsertPartitioner` cases, 6/6 `TestLSMDataSource` cases, 3/3 `TestInsertWithLSMLayout` cases, module compilation/checkstyle/scalastyle during the targeted Maven runs, and `git diff --check`. ### Documentation Update None. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
