SavicStefan commented on code in PR #59669:
URL: https://github.com/apache/spark/pull/59669#discussion_r4424809926
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:
##########
@@ -205,6 +205,79 @@ public static Variant pickAtPaths(Variant v, PickNode
tree) {
return builder.result();
}
+ // Return v in canonical form: only referenced dictionary keys, unique and
sorted by unsigned
+ // UTF-8 bytes; object fields stored in key order; minimal integer, decimal,
and offset widths;
+ // strings of at most 63 UTF-8 bytes use short-string encoding; decimal
trailing zeros are
+ // stripped, and integral decimals become longs only when they fit.
Normalize floating-point
+ // -0.0 to 0.0 and NaNs within each type. Preserve string and key bytes,
including invalid UTF-8.
+ // Return v itself if it is already canonical.
+ public static Variant canonicalize(Variant v) {
+ if (isCanonical(v)) {
Review Comment:
`Variant` is created per evaluation, its what Spark uses to read inside the
value, and its discarded, so a flag is lost immediately.
`VariantVal` lives longer, but every time it goes through an `UnsafeRow`
(shuffle, sort, spill, cache, and the aggregation grouping key in
`HashAggregateExec`) and its rebuilt from the bytes in
`VariantVal.readFromUnsafeRow` so the flag will reset and we will then need to
rebuild it again.
Keeping it would mean changing the row format.
I think `isCanonical` is the right approach here, because like
`doCanonicalize` produces a new VariantVal with canonical byte, and those bytes
are preserved even when they go through an `UnsafeRow` (shuffle, sort, spill,
cache), so any later `CanonicalizeVariant` can recognize them with a single
`isCanonical` scan and skip the rebuild, while a flag on the object would be
lost at those boundaries.
`isCanonical` is a read-only scan, it allocates just one `boolean[]` for
key usage and it doesnt collect or sort keys, doesnt write new value/metadata
buffers and it stops at the first non-canonical byte.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
