slachiewicz opened a new pull request, #977:
URL: https://github.com/apache/hudi-rs/pull/977

   ## Description
   
   Part of #549.
   
   Table version 9 persists `CustomKeyGenerator`'s per-field key types in
   `hoodie.table.partition.fields` (`region:SIMPLE,ts_str:TIMESTAMP`), and the 
reader passed
   `region:SIMPLE` on as a field name, which no schema contains. 
`PartitionFields` now keeps
   only the part before the annotation. Key generator inference, which 
recognizes the custom
   key generator by that annotation, reads the raw config value instead of the 
parsed list.
   
   Two COW fixtures come with this, deliberately a pair: one written by Hudi 
0.15.0 (table
   version 6), which records `region,ts_str` and persists no per-field key 
types at all, and
   one by Hudi 1.2.0 on Spark 4.1 (version 9), which records the annotated 
spelling. A reader
   cannot recover the types on version 6, so the obvious fallback for an 
untyped field is
   unsafe; that is why filters on the time-derived field are not rewritten 
through
   `CustomKeyGenerator` here. The tests pin the `region=eu/ts_str=2024-03-01` 
path shape and
   that plain partition-column predicates prune on both versions.
   
   Split out of #724 at review.
   
   Verified: `cargo test -p hudi-core --lib -- table::partition keygen expr 
config` and
   `cargo test -p hudi-core --test table_read_tests -- custom_keygen` pass, and
   `cargo fmt --all -- --check` and `cargo clippy -p hudi-core --all-targets -- 
-D warnings` are
   clean locally.
   
   ## How are the changes test-covered
   
   - [ ] N/A
   - [x] Automated tests (unit and/or integration tests)
   - [ ] Manual tests
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to