yihua opened a new issue, #856: URL: https://github.com/apache/hudi-rs/issues/856
## Summary hudi-rs has two enums for deciding which version of a record key wins: - `RecordMergeStrategyValue` ([`merge/mod.rs`](https://github.com/apache/hudi-rs/blob/ed3653b42fab43a6747b1031c5d5e0bfde4fac67/crates/core/src/merge/mod.rs#L28-L47)), public, with values `AppendOnly` and `OverwriteWithLatest`. It is read from `HudiTableConfig::RecordMergeStrategy` (`hoodie.table.record.merge.strategy`) and drives file group reader version 1 through `merge::record_merger::RecordMerger`. Added in #249 and #260 (0.3.0). - `MergeMode` ([`reader_v2/reader_context.rs`](https://github.com/apache/hudi-rs/blob/ed3653b42fab43a6747b1031c5d5e0bfde4fac67/crates/core/src/file_group/reader_v2/reader_context.rs#L425-L445)), `pub(crate)`, with values `CommitTimeOrdering`, `EventTimeOrdering` and `Custom`. It is resolved from `hoodie.record.merge.mode` by [`resolve_merge_mode`](https://github.com/apache/hudi-rs/blob/ed3653b42fab43a6747b1031c5d5e0bfde4fac67/crates/core/src/file_group/reader_v2/resolver.rs#L382) and drives reader version 2. Added in #660. The second matches the merge modes Hudi Java writes. The first does not, in three ways. 1. **The config key is hudi-rs only.** Java writes `hoodie.record.merge.mode` and `hoodie.record.merge.strategy.id`, never `hoodie.table.record.merge.strategy`. For any table Java wrote, the value is derived in [`parse_value_or_default`](https://github.com/apache/hudi-rs/blob/ed3653b42fab43a6747b1031c5d5e0bfde4fac67/crates/core/src/config/table.rs#L322-L358) from `hoodie.populate.meta.fields` and the ordering fields, and the table's own merge mode is ignored. 2. **`OverwriteWithLatest` means the opposite of its Java namesake.** The v1 merger [sorts by record key, ordering field, then `_hoodie_commit_seqno`](https://github.com/apache/hudi-rs/blob/ed3653b42fab43a6747b1031c5d5e0bfde4fac67/crates/core/src/merge/record_merger.rs#L103-L110), which is event-time ordering. In Java, `OverwriteWithLatestAvroPayload` maps to commit-time ordering, as the `MergeMode::CommitTimeOrdering` doc notes. 3. **`AppendOnly` has no merge mode counterpart, and v1 and v2 disagree on it.** For a table with meta fields but no ordering field, v1 picks `AppendOnly` and concatenates without deduplicating. v2 infers `COMMIT_TIME_ORDERING` and deduplicates by key. The two readers can return different rows for that table shape. This comes from reading the code; it is not reproduced yet. ## Proposal - Move `MergeMode` into `crate::merge`, make it public, and have the v1 `RecordMerger` resolve its mode through the same `resolve_merge_mode` as v2. - Model the `hoodie.populate.meta.fields=false` case as an explicit no-merge path keyed on that config, rather than as a merge mode. - Deprecate `HudiTableConfig::RecordMergeStrategy` and `RecordMergeStrategyValue` for one release, then remove them. - Add a regression test for item 3 that pins v1 to the v2 (Java) behavior. ## Impact - **Storage:** none. Nothing on `main` writes `hoodie.table.record.merge.strategy` to `hoodie.properties`; the key is only read as an option or derived. The native write path in #666 persists the Java keys (`hoodie.record.merge.mode`, `hoodie.record.merge.strategy.id`), not this one. - **Public API:** breaking for `hudi-core` (`RecordMergeStrategyValue`, `HudiTableConfig::RecordMergeStrategy`). Neither is referenced from `python/`, `cpp/`, `crates/datafusion` or `crates/jvm-ffi`. - **Write path:** #666 uses `RecordMergeStrategyValue` to decide whether an append is allowed, so whichever of #666 and this change lands second updates those call sites. - **Read results:** item 3 changes what reader version 1 returns for tables without an ordering field. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
