Hello Peter Rozsa, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24990
to look at the new patch set (#3).
Change subject: IMPALA-15460: Read VARIANT object fields stored out of name
order
......................................................................
IMPALA-15460: Read VARIANT object fields stored out of name order
The Variant spec orders an object's field ids and offsets by field
name, but the values may be stored in any order, and Spark stores them
in document order. VariantValue::FieldFromLayout() took offset[i + 1]
as the end of field i and rejected the field if that offset was
smaller. variant_get() then returned NULL silently, variant_to_json()
returned NULL, and SELECT of the column failed with "Failed to read
object field".
This patch sizes each field and array element from its own header with
the new VariantValue::ValueSize(), like parquet-java's valueSize().
The last offset of an object or array bounds its values.
A value on the path that cannot be read is now reported as corrupt
(an error from variant_get(), a warning from try_variant_get())
instead of a silent NULL. Lookups return FOUND, NOT_FOUND or CORRUPT.
GetFieldByName() now searches the object's own field ids by name,
linearly below 32 fields and by binary search above, like parquet-java.
The old dictionary lookup took the first match, so it missed the field
when the dictionary contained the name more than once. Names with code
points from U+E000 fall back to a linear scan, because Spark < 4.3
(SPARK-58949) and parquet-java (GH-3735) sort field ids by UTF-16 code
units. FindFieldId() is removed. The JSON serializer now fails on
unknown primitive type ids.
Testing:
- Added BE tests, including bytes written by Spark.
- Added an Iceberg V3 table written by Spark and an EE test.
Assisted-by: Claude Opus 5.5 <[email protected]>
Change-Id: Ief71d02788a4c830619e6ceaf45ef8eed366411f
---
M be/src/exprs/variant-functions-ir.cc
M be/src/exprs/variant-functions-test.cc
M be/src/runtime/variant-value.cc
M be/src/runtime/variant-value.h
M be/src/util/variant-util-test.cc
M testdata/data/README
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_spark_variant/data/00000-0-cc54c0ad-f5ea-47bb-ad8f-425b498f0d56-0-00001.parquet
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_spark_variant/metadata/4dc789bc-9470-41d7-9828-948ca99d85d7-m0.avro
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_spark_variant/metadata/snap-8513010963727552604-1-4dc789bc-9470-41d7-9828-948ca99d85d7.avro
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_spark_variant/metadata/v1.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_spark_variant/metadata/v2.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_spark_variant/metadata/version-hint.text
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-variant-spark.test
M tests/query_test/test_iceberg.py
14 files changed, 1,594 insertions(+), 207 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24990/3
--
To view, visit http://gerrit.cloudera.org:8080/24990
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ief71d02788a4c830619e6ceaf45ef8eed366411f
Gerrit-Change-Number: 24990
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
