[
https://issues.apache.org/jira/browse/HIVE-30171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30171:
----------------------------------
Description:
=, IN, GROUP BY, DISTINCT and joins compare raw bytes. Task 0 showed on base
that v = parse_json('{"a":1,"b":"x"}') matches no rows even when the stored
value is identical.
The review of HIVE-30170 (#6881) found more places that compare or hash a
variant byte-wise. Each must follow the
semantic rule, or be kept byte-wise on purpose:
- KeyWrapperFactory.ListKeyWrapper (row-mode hash GROUP BY, DISTINCT,
count(DISTINCT)): hashes the raw key
objects. Iceberg's VariantData has no equals/hashCode, so map-side
aggregation never merges equal Iceberg variants
and switches itself off after 100k rows. Results stay correct; the reducer
groups by BinarySortable bytes.
- GenericUDAFApproximateDistinct: hashes the Java-serialized list of the
metadata and value arrays.
- HiveVariantComparator (merge join): its own byte-wise order, identical today
to ObjectInspectorUtils.compare.
Merge join must agree with the BinarySortable key order, so decide whether it
stays byte-wise.
- Array set functions use Object.equals on the raw element: array_distinct,
array_intersect, array_except,
array_union, array_remove. The answer depends on the source of the array
(standard values match since
HIVE-30170; Iceberg, text and LazyBinary values never match). array_contains
and array_position already go through
ObjectInspectorUtils.compare.
- array_max / array_min accept array<variant> and then throw ClassCastException
(Comparator.naturalOrder). Their
initialize never checks the element type, so struct elements reach the same
call; a separate JIRA, independent
of VARIANT.
was:=, IN, GROUP BY, DISTINCT and joins compare raw bytes. Task 0 showed on
base that v = parse_json('{"a":1,"b":"x"}') matches no rows even when the
stored value is identical.
> Semantic equality, hashing and grouping for VARIANT
> ---------------------------------------------------
>
> Key: HIVE-30171
> URL: https://issues.apache.org/jira/browse/HIVE-30171
> Project: Hive
> Issue Type: Sub-task
> Reporter: Denys Kuzmenko
> Priority: Major
>
> =, IN, GROUP BY, DISTINCT and joins compare raw bytes. Task 0 showed on base
> that v = parse_json('{"a":1,"b":"x"}') matches no rows even when the stored
> value is identical.
> The review of HIVE-30170 (#6881) found more places that compare or hash a
> variant byte-wise. Each must follow the
> semantic rule, or be kept byte-wise on purpose:
> - KeyWrapperFactory.ListKeyWrapper (row-mode hash GROUP BY, DISTINCT,
> count(DISTINCT)): hashes the raw key
> objects. Iceberg's VariantData has no equals/hashCode, so map-side
> aggregation never merges equal Iceberg variants
> and switches itself off after 100k rows. Results stay correct; the reducer
> groups by BinarySortable bytes.
> - GenericUDAFApproximateDistinct: hashes the Java-serialized list of the
> metadata and value arrays.
> - HiveVariantComparator (merge join): its own byte-wise order, identical
> today to ObjectInspectorUtils.compare.
> Merge join must agree with the BinarySortable key order, so decide whether
> it stays byte-wise.
> - Array set functions use Object.equals on the raw element: array_distinct,
> array_intersect, array_except,
> array_union, array_remove. The answer depends on the source of the array
> (standard values match since
> HIVE-30170; Iceberg, text and LazyBinary values never match).
> array_contains and array_position already go through
> ObjectInspectorUtils.compare.
> - array_max / array_min accept array<variant> and then throw
> ClassCastException (Comparator.naturalOrder). Their
> initialize never checks the element type, so struct elements reach the same
> call; a separate JIRA, independent
> of VARIANT.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
