Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24521 )
Change subject: IMPALA-15052: Add read support for unshredded VARIANT values ...................................................................... IMPALA-15052: Add read support for unshredded VARIANT values Builds on the backend VARIANT decoding from IMPALA-15067 to enable end-to-end reading of unshredded VARIANT columns from Iceberg tables stored as Parquet. VARIANT is read-only: it can be scanned and rendered as JSON, but it cannot be created or written, and it can only appear in queries as a pass-through column reference. Type system: - TYPE_VARIANT added across thrift (TPrimitiveType/TTypeNodeType), the BE (PrimitiveType/ColumnType, 24-byte slot = two StringValues) and the FE (VariantType extending Type with struct-like metadata/value fields) - An unshredded VARIANT is modeled as a two-child group: "metadata" and "value", both BINARY. Read path (Parquet): - VariantColumnReader (extends StructColumnReader) reads the variant group's metadata + value children. The physical layout matches a two-field struct, so ParquetSchemaResolver resolves VARIANT like STRUCT. - A malformed variant group whose Parquet child count does not match the catalog (i.e. not exactly metadata + value) fails with a runtime error. - The reader is excluded from late materialization (it inherits the struct SkipRows() restriction) so a filtered scan never calls SkipRows() on it. - FE descriptors: the Analyzer builds a children tuple with metadata/value slots for each VARIANT column. Result display: - VARIANT columns are serialized to their JSON representation in query output (hs2-util, query-result-set), via the backend VariantSlotToJson helper. BINARY fields inside a variant are base64-encoded in the JSON (as for complex types; matches Trino). - A VARIANT value that fails to decode (corruption) fails the query, instead of being silently substituted with a wrong value. - To clients (HS2 result metadata, GetColumns, beeswax) a VARIANT column is reported as STRING/VARCHAR, like the other complex types. NOTE: This may vary, as client interfaces may add native support for VARIANT types in the future. DDL and analysis (VARIANT is read-only): - VARIANT is a non-reserved keyword: it is recognized by the parser as a type, but existing identifiers named "variant" keep working. It is rejected as a user-specified column type or CAST target (CREATE/ALTER TABLE, CAST). Unsupported operations by design - ORDER BY, GROUP BY, SELECT DISTINCT, UNION/INTERSECT/EXCEPT, CASE/DECODE analytic PARTITION BY/ORDER BY, aggregate DISTINCT Currently not supported operations - IS NULL, COMPUTE STATS, INSERT, CASE Supported operations - UNION ALL of a variant column is supported. - Only a top-level VARIANT column can be queried. A VARIANT nested inside another complex type (ARRAY<VARIANT>, MAP<..,VARIANT>, STRUCT<..VARIANT..>, and deeper combinations) is rejected. Such a table still loads; only queries that reference the nested VARIANT fail Iceberg / HMS: - IcebergSchemaConverter maps the Iceberg VARIANT type id to TYPE_VARIANT. - HMS has no 'variant' column type, so Type.toHiveMetastoreType() stores VARIANT as its underlying struct<metadata:binary,value:binary> (matching Iceberg's Hive catalog and Trino). Impala still reads the real VARIANT type from the Iceberg metadata; the HMS FieldSchema is only used for Hive interop. - DESCRIBE and DESCRIBE FORMATTED show the user-facing 'variant' type, not the HMS struct (DescribeResultFactory builds the formatted output from the Impala type). Codegen: - Codegen is disabled for queries that scan a VARIANT column (currently query-wide; see TODO in HdfsScanNode), and Tuple::CodegenMaterializeExprs falls back to the interpreted path for VARIANT slots. The interpreted infrastructure treats VARIANT like STRUCT to avoid crashes. Forward compatibility: - No user-facing variant_get()/variant_to_json() builtin is registered yet. These will be added once VARIANT is a first-class expression type (a VariantVal ABI that lets functions accept and return VARIANT). Testing: - New e2e test (iceberg-v3-variant.test) covering JSON rendering of primitives/strings/arrays/objects, filtered scans, TOP-N, UNION ALL, spilling joins and a spilling sort, slot/tuple unification, and the negative analysis cases above, against a Trino-written Iceberg V3 table (trino_variant). - The same test also covers the nested-VARIANT rejections against a second Trino-written table (trino_nested_variant) that carries VARIANT nested in ARRAY/MAP/STRUCT (and ARRAY<STRUCT<VARIANT>>), plus a top-level VARIANT column to confirm that case still works. It also checks DESCRIBE and DESCRIBE FORMATTED rendering of the VARIANT and nested-VARIANT columns (describing such a table is always allowed). - trino_variant is bulk-loaded during data loading, so tests read the shared functional_parquet.trino_variant table. - JdbcTest covers the JDBC client-facing presentation (VARCHAR, TYPE_NAME 'VARIANT', JSON values). - ParserTest confirms VARIANT parses as a type and remains usable as an identifier (non-reserved keyword). Change-Id: Ie2f8a7c9b1d4e5f6a0c3b8d7e9f1a2b4c6d8e0f1 Assisted-by: Claude Opus 4.6 <[email protected]> Assisted-by: Claude Opus 4.8 (1M context) <[email protected]> Reviewed-on: http://gerrit.cloudera.org:8080/24521 Reviewed-by: Csaba Ringhofer <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M .gitattributes M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/exec/parquet/parquet-struct-column-reader.h A be/src/exec/parquet/parquet-variant-column-reader.h M be/src/exprs/anyval-util.cc M be/src/exprs/scalar-expr-evaluator.cc M be/src/exprs/scalar-expr.cc M be/src/exprs/scalar-expr.inline.h M be/src/exprs/slot-ref.cc M be/src/runtime/descriptors.cc M be/src/runtime/raw-value.cc M be/src/runtime/raw-value.h M be/src/runtime/string-value.h M be/src/runtime/tuple.cc M be/src/runtime/types.cc M be/src/runtime/types.h M be/src/runtime/variant-value.cc M be/src/runtime/variant-value.h M be/src/service/hs2-util.cc M be/src/service/hs2-util.h M be/src/service/impala-beeswax-server.cc M be/src/service/query-result-set.cc M be/src/service/query-result-set.h M be/src/util/variant-util.cc M be/src/util/variant-util.h M common/thrift/Types.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CaseExpr.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/SortInfo.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java M fe/src/main/java/org/apache/impala/analysis/TypeDef.java M fe/src/main/java/org/apache/impala/catalog/Type.java A fe/src/main/java/org/apache/impala/catalog/VariantType.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/service/JdbcTest.java M testdata/data/README A testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/data/20260703_115517_00018_wa2kj-fc2bf4c0-5cb6-4bd2-abb2-2eac147adff9.parquet A testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/metadata/20260703_115517_00018_wa2kj-6375d4da-66f3-4adf-b0d6-fb7964f5b953.stats A testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/metadata/f8d5c650-108d-46aa-ae9f-e9d94e10adf4-m0.avro A testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/metadata/snap-5173553188971266738-1-f8d5c650-108d-46aa-ae9f-e9d94e10adf4.avro A testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/metadata/snap-5208362247809940782-1-10d5391c-e63e-4913-b898-ad7ef55cf306.avro A testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/metadata/version-hint.text A testdata/data/iceberg_test/iceberg_v3/trino_variant/data/20260629_095655_00023_wa2kj-0f39c7a5-6cc6-4be7-abc0-a45a62dacec5.parquet A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/20260629_095655_00023_wa2kj-bcc1a100-d380-42cf-bccf-2e32badf7ac5.stats A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/a874fc23-2c2c-42b5-9fc0-f72919a7449d-m0.avro A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/snap-6097489456815673056-1-a874fc23-2c2c-42b5-9fc0-f72919a7449d.avro A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/snap-8332248857076198293-1-03e34f74-1a29-4e53-ab2c-6f2c507d534d.avro A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/version-hint.text M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv A testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-variant.test M testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test M testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test M tests/custom_cluster/test_beeswax.py M tests/hs2/test_fetch.py M tests/query_test/test_iceberg.py M tests/util/iceberg_metadata_util.py 85 files changed, 2,239 insertions(+), 94 deletions(-) Approvals: Csaba Ringhofer: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/24521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie2f8a7c9b1d4e5f6a0c3b8d7e9f1a2b4c6d8e0f1 Gerrit-Change-Number: 24521 Gerrit-PatchSet: 17 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Arnab Karmakar <[email protected]> Gerrit-Reviewer: Balazs Hevele <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
