Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24882 )
Change subject: IMPALA-15141 (part 1): Keep codegen enabled for queries on VARIANT ...................................................................... IMPALA-15141 (part 1): Keep codegen enabled for queries on VARIANT The planner set the query-wide disable_codegen_hint for every query that scans a VARIANT column, so these queries lost codegen in all of their fragments, even in operators that never touch VARIANT values. This patch removes the hint and makes the backend safe without it. Expressions that involve VARIANT are still interpreted. The hint also protected codegen paths that cannot deal with VARIANT: * Exec nodes call ScalarExpr::GetCodegendComputeFn() directly (e.g. for conjuncts, hash/sort keys and partition exprs), bypassing ShouldCodegen(). GetCodegendComputeFn() now returns an expected error for expressions that involve VARIANT, so these callers fall back to interpretation, but only for the given operator. * LlvmCodeGen::GetSlotType() now knows the layout of VARIANT slots (two StringValues), so TupleDescriptor::GetLlvmStruct() works for tuples that have a VARIANT slot. * Only the Parquet scanner can read VARIANT, but Iceberg tables can also have Avro and ORC data files with VARIANT columns. The planner now rejects scans that read VARIANT from non-Parquet data files, similarly to UUID (IMPALA-15200). Tuple::CodegenMaterializeExprs() still falls back if the destination tuple has a VARIANT slot (TOP-N, UNION ALL). Testing: * added e2e tests to check that the planner does not disable codegen * added Iceberg V3 test tables with VARIANT in Avro, ORC and mixed data files, generated with the Iceberg Java API Change-Id: I6e3c59cc41bab7421e0dd450e08944ffc3889241 Assisted-by: Claude Fable 5.1 (Claude Code) Reviewed-on: http://gerrit.cloudera.org:8080/24882 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/codegen/llvm-codegen.cc M be/src/exprs/scalar-expr.cc M be/src/runtime/tuple.cc M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M testdata/data/README A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_avro/data/00000-0-variant.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_avro/metadata/4c32a9ec-306c-4e65-8646-5590e22b46d1-m0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_avro/metadata/snap-7108812184163398669-1-4c32a9ec-306c-4e65-8646-5590e22b46d1.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_avro/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_avro/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_avro/metadata/version-hint.text A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_mixed/data/p=1/00000-0-variant.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_mixed/data/p=2/00001-0-variant.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_mixed/metadata/23d64a68-204f-495c-a332-a66af7caa0c6-m0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_mixed/metadata/snap-8008792904878867789-1-23d64a68-204f-495c-a332-a66af7caa0c6.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_mixed/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_mixed/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_mixed/metadata/version-hint.text A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_orc/data/00000-0-variant.orc A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_orc/metadata/1bd3b0e1-d755-4627-9717-743e7ca7caa3-m0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_orc/metadata/snap-4259005618451124215-1-1bd3b0e1-d755-4627-9717-743e7ca7caa3.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_orc/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_orc/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_variant_orc/metadata/version-hint.text M testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-variant-get.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-variant.test M tests/query_test/test_iceberg.py 27 files changed, 133 insertions(+), 11 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/24882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6e3c59cc41bab7421e0dd450e08944ffc3889241 Gerrit-Change-Number: 24882 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
