Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24990
Change subject: IMPALA-15460: Read VARIANT object fields stored out of name order ...................................................................... IMPALA-15460: Read VARIANT object fields stored out of name order The Variant spec orders an object's field ids and offsets by field name, but the values may be stored in any order, and Spark stores them in document order. VariantValue::FieldFromLayout() took offset[i + 1] as the end of field i and rejected the field if that offset was smaller. variant_get() then returned NULL silently, variant_to_json() returned NULL, and SELECT of the column failed with "Failed to read object field". This patch sizes each field and array element from its own header with the new VariantValue::ValueSize(), like parquet-java's valueSize(). The last offset of an object or array bounds its values. A value on the path that cannot be read is now reported as corrupt (an error from variant_get(), a warning from try_variant_get()) instead of a silent NULL. Lookups return FOUND, NOT_FOUND or CORRUPT. GetFieldByName() now matches the name against the object's own field ids. The old dictionary lookup took the first match, so it missed the field when the dictionary contained the name more than once. FindFieldId() is removed. The JSON serializer now fails on unknown primitive type ids. Testing: - Added BE tests, including bytes written by Spark. - Added an Iceberg V3 table written by Spark and an EE test. Assisted-by: Claude Opus 5.5 <[email protected]> Change-Id: Ief71d02788a4c830619e6ceaf45ef8eed366411f --- M be/src/exprs/variant-functions-ir.cc M be/src/exprs/variant-functions-test.cc M be/src/runtime/variant-value.cc M be/src/runtime/variant-value.h M be/src/util/variant-util-test.cc M testdata/data/README A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_spark_variant/data/00000-0-cc54c0ad-f5ea-47bb-ad8f-425b498f0d56-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_spark_variant/metadata/4dc789bc-9470-41d7-9828-948ca99d85d7-m0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_spark_variant/metadata/snap-8513010963727552604-1-4dc789bc-9470-41d7-9828-948ca99d85d7.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_spark_variant/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_spark_variant/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_spark_variant/metadata/version-hint.text A testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-variant-spark.test M tests/query_test/test_iceberg.py 14 files changed, 1,478 insertions(+), 204 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24990/1 -- To view, visit http://gerrit.cloudera.org:8080/24990 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ief71d02788a4c830619e6ceaf45ef8eed366411f Gerrit-Change-Number: 24990 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
