raghuvanshraj opened a new pull request, #12303:
URL: https://github.com/apache/arrow-rs/pull/12303

   # Which issue does this PR close?
   
   - Closes #12302.
   
   # Rationale for this change
   
   One Variant value can be written many ways: the metadata dictionary may hold 
unused or unsorted names, an integer may be wider than it needs to be, a 
decimal may carry trailing zeros, `-0.0` and `0.0` differ, and so on. Anything 
that hashes or compares the raw `metadata` + `value` bytes then treats equal 
values as different. In datafusion-variant that shows up as `GROUP BY` 
producing one group per row and self joins matching nothing 
(datafusion-contrib/datafusion-variant#183).
   
   This adds a way to rebuild a value so that equal values share one encoding.
   
   # What changes are included in this PR?
   
   `Variant::canonicalize(&self) -> Result<(Vec<u8>, Vec<u8>), ArrowError>` in 
a new `canonicalize.rs` module. The rules follow the canonical form Spark is 
adopting in apache/spark#59669:
   
   - metadata holds only the field names the value references, sorted by 
unsigned UTF-8 bytes, smallest offset width
   - object fields are written in key order; list elements keep their order
   - integers use the smallest of INT8/16/32/64 that fits
   - decimals drop trailing zeros; an integral decimal that fits an `i64` 
becomes an integer, otherwise the smallest DECIMAL4/8/16 is used
   - FLOAT and DOUBLE keep their type; `-0.0` becomes `0.0` and every NaN 
becomes `f32::NAN` / `f64::NAN`
   - strings use the short-string form when they fit
   - null, booleans, dates, times, timestamps, binary and UUID already have one 
encoding and are copied as-is
   
   An object with a duplicate field name is rejected with an error. The 
implementation collects the referenced names into a `BTreeSet`, seeds a 
`VariantBuilder` with them, and re-appends the value through 
`VariantBuilderExt`, so the existing builders do the encoding.
   
   One difference from Spark: the builder sets `sorted_strings` in the metadata 
header when the dictionary is sorted, so canonical output here always has that 
bit set, while Spark's canonical header leaves it unset. Equal values still get 
equal bytes within arrow-rs. Happy to clear the bit if matching Spark's bytes 
exactly matters.
   
   # Are these changes tested?
   
   Yes, 13 unit tests in the new module plus a doc example: integer narrowing, 
decimal zero stripping and promotion including values wider than `i64`, float 
zero and NaN handling, short vs long strings, unchanged types round-trip to 
identical bytes, unused and unsorted dictionary names, nested names sorted by 
bytes (`B` < `a` < `é`), the #183 case of the same string under different 
dictionaries, objects with fields in different order, null object fields kept, 
list order kept, idempotency, and duplicate field name rejection.
   
   # Are there any significant user-facing changes?
   
   New public method `Variant::canonicalize`. No existing API changes.
   
   # AI usage disclosure
   
   Written with an AI coding assistant for the implementation and test 
scaffolding; the rules, the Spark comparison and the review of every test were 
done by hand.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to