slachiewicz opened a new pull request, #977: URL: https://github.com/apache/hudi-rs/pull/977
## Description Part of #549. Table version 9 persists `CustomKeyGenerator`'s per-field key types in `hoodie.table.partition.fields` (`region:SIMPLE,ts_str:TIMESTAMP`), and the reader passed `region:SIMPLE` on as a field name, which no schema contains. `PartitionFields` now keeps only the part before the annotation. Key generator inference, which recognizes the custom key generator by that annotation, reads the raw config value instead of the parsed list. Two COW fixtures come with this, deliberately a pair: one written by Hudi 0.15.0 (table version 6), which records `region,ts_str` and persists no per-field key types at all, and one by Hudi 1.2.0 on Spark 4.1 (version 9), which records the annotated spelling. A reader cannot recover the types on version 6, so the obvious fallback for an untyped field is unsafe; that is why filters on the time-derived field are not rewritten through `CustomKeyGenerator` here. The tests pin the `region=eu/ts_str=2024-03-01` path shape and that plain partition-column predicates prune on both versions. Split out of #724 at review. Verified: `cargo test -p hudi-core --lib -- table::partition keygen expr config` and `cargo test -p hudi-core --test table_read_tests -- custom_keygen` pass, and `cargo fmt --all -- --check` and `cargo clippy -p hudi-core --all-targets -- -D warnings` are clean locally. ## How are the changes test-covered - [ ] N/A - [x] Automated tests (unit and/or integration tests) - [ ] Manual tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
