Arnab Karmakar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/25041
Change subject: IMPALA-15212: Iceberg UUID ORC read support ...................................................................... IMPALA-15212: Iceberg UUID ORC read support Iceberg stores UUID columns in ORC files as BINARY with the iceberg.binary-type=UUID attribute. Impala now reads them into TYPE_UUID slots, so Iceberg tables with UUID columns can be scanned when they contain ORC data files. This works for UUIDs at the top level and inside STRUCT, ARRAY and MAP columns. Avro data files with UUID columns are still rejected. The ORC schema resolver maps BINARY to UUID table columns. A new OrcUuidColumnReader copies the 16 bytes of each value from the ORC batch into the inline slot. A value of any other length, or a dictionary-encoded batch, fails the scan as a corrupt file. UUID predicates (=, IN, range, IS NULL and IS NOT NULL) are pushed to the ORC reader as STRING search arguments on the 16 raw bytes. ORC BINARY columns have no min/max statistics, so the reader can only skip data by null counts and, for = and IN, by bloom filters, which hash the same raw bytes. Range predicates never skip data as the scan evaluates them in unsigned byte order. IN-list runtime filters do not support UUID. The planner's memory reservation estimate for ORC async reads now counts the LENGTH stream of UUID columns. Testing: - Regenerated the iceberg_uuid_test_orc fixture as a single ORC data file with a bloom filter on uuid_col, using the Iceberg Java API (version 1.10.1). - e2e tests on iceberg_uuid_test_orc: SELECT, =, !=, IN, NOT IN, range, IS NULL and IS NOT NULL predicates, ORDER BY, aggregates, CAST, a JOIN with the Parquet fixture, EXPLAIN checks that UUID predicates are ORC statistics predicates, and a check that an equality predicate on an absent value skips the data through the bloom filter. - Trino interop tests: Impala reads a table with both Parquet and ORC data files, and a Trino-written ORC table with UUIDs at the top level and in STRUCT, ARRAY and MAP, with values on both sides of the sign bit, including a JOIN with a Parquet table. Change-Id: Id736a9ea852892b610281df9105c82e4a4ce42da Assisted-by: Claude Opus 5.5 (Cursor) --- M be/src/exec/orc/hdfs-orc-scanner.cc M be/src/exec/orc/hdfs-orc-scanner.h M be/src/exec/orc/orc-column-readers.cc M be/src/exec/orc/orc-column-readers.h M be/src/exec/orc/orc-metadata-utils.cc M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M testdata/data/README D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-2e0fbfe3-d923-4a27-b5d4-f941a82fb1d1-1-00001.orc D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-896e60f8-2a60-47e3-8187-8f8bed145b7f-1-00001.orc D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-b9172fb6-1d4c-46e0-817c-b3d6377ccf31-1-00001.orc D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-d7da9c66-c7e5-45e0-a69a-0c2eb82e96c0-1-00001.orc D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-eef53fbc-bade-487e-b510-537354cf46e0-1-00001.orc A testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-ffed3f33-e038-42ff-8c17-8e03204fd3e0-1-00001.orc D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/22fa5c4b-83a3-46b2-bafa-3553189862bd-m0.avro D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/4abe1f4e-1220-4b05-a5e6-ebfc8fb92df3-m0.avro D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/547a17ba-a0d2-4921-956b-e06fbdadb60b-m0.avro R testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/92eaf5b5-cb0d-4589-a10d-a5622ead529f-m0.avro D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/d570e583-e57a-4879-8a99-052c9a85d706-m0.avro D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-5116163803842658326-1-4abe1f4e-1220-4b05-a5e6-ebfc8fb92df3.avro D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-5748673676392974692-1-75312a76-0d22-45dc-80c7-41c2cb712f98.avro D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-6680088367048559051-1-22fa5c4b-83a3-46b2-bafa-3553189862bd.avro D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-7102485323114575692-1-547a17ba-a0d2-4921-956b-e06fbdadb60b.avro R testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-7535938409123090183-1-92eaf5b5-cb0d-4589-a10d-a5622ead529f.avro M testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v1.metadata.json M testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v2.metadata.json D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v3.metadata.json D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v4.metadata.json D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v5.metadata.json D testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v6.metadata.json M testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/version-hint.text M testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-uuid.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-uuid-type.test M tests/custom_cluster/test_iceberg_trino_interop.py 34 files changed, 384 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/25041/1 -- To view, visit http://gerrit.cloudera.org:8080/25041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id736a9ea852892b610281df9105c82e4a4ce42da Gerrit-Change-Number: 25041 Gerrit-PatchSet: 1 Gerrit-Owner: Arnab Karmakar <[email protected]>
