raghuvanshraj opened a new pull request, #12303: URL: https://github.com/apache/arrow-rs/pull/12303
# Which issue does this PR close? - Closes #12302. # Rationale for this change One Variant value can be written many ways: the metadata dictionary may hold unused or unsorted names, an integer may be wider than it needs to be, a decimal may carry trailing zeros, `-0.0` and `0.0` differ, and so on. Anything that hashes or compares the raw `metadata` + `value` bytes then treats equal values as different. In datafusion-variant that shows up as `GROUP BY` producing one group per row and self joins matching nothing (datafusion-contrib/datafusion-variant#183). This adds a way to rebuild a value so that equal values share one encoding. # What changes are included in this PR? `Variant::canonicalize(&self) -> Result<(Vec<u8>, Vec<u8>), ArrowError>` in a new `canonicalize.rs` module. The rules follow the canonical form Spark is adopting in apache/spark#59669: - metadata holds only the field names the value references, sorted by unsigned UTF-8 bytes, smallest offset width - object fields are written in key order; list elements keep their order - integers use the smallest of INT8/16/32/64 that fits - decimals drop trailing zeros; an integral decimal that fits an `i64` becomes an integer, otherwise the smallest DECIMAL4/8/16 is used - FLOAT and DOUBLE keep their type; `-0.0` becomes `0.0` and every NaN becomes `f32::NAN` / `f64::NAN` - strings use the short-string form when they fit - null, booleans, dates, times, timestamps, binary and UUID already have one encoding and are copied as-is An object with a duplicate field name is rejected with an error. The implementation collects the referenced names into a `BTreeSet`, seeds a `VariantBuilder` with them, and re-appends the value through `VariantBuilderExt`, so the existing builders do the encoding. One difference from Spark: the builder sets `sorted_strings` in the metadata header when the dictionary is sorted, so canonical output here always has that bit set, while Spark's canonical header leaves it unset. Equal values still get equal bytes within arrow-rs. Happy to clear the bit if matching Spark's bytes exactly matters. # Are these changes tested? Yes, 13 unit tests in the new module plus a doc example: integer narrowing, decimal zero stripping and promotion including values wider than `i64`, float zero and NaN handling, short vs long strings, unchanged types round-trip to identical bytes, unused and unsorted dictionary names, nested names sorted by bytes (`B` < `a` < `é`), the #183 case of the same string under different dictionaries, objects with fields in different order, null object fields kept, list order kept, idempotency, and duplicate field name rejection. # Are there any significant user-facing changes? New public method `Variant::canonicalize`. No existing API changes. # AI usage disclosure Written with an AI coding assistant for the implementation and test scaffolding; the rules, the Spark comparison and the review of every test were done by hand. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
