wangsheng has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/16788 )
Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables ...................................................................... IMPALA-10361: Use field id to resolve columns for Iceberg tables We supported resolve column by field id for Iceberg table in this patch. We can use 'set PARQUET_FALLBACK_SCHEMA_RESOLUTION=FIELD_ID' or 'set PARQUET_FALLBACK_SCHEMA_RESOLUTION=2' to choose field id resolving. But pay attention, if you use this for non-Iceberg table, the result will be NULL. Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 --- M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/exec/parquet/parquet-metadata-utils.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/types.cc M be/src/runtime/types.h M be/src/service/query-options-test.cc M common/thrift/CatalogObjects.thrift M common/thrift/Descriptors.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/Types.thrift M fe/src/main/java/org/apache/impala/catalog/Column.java M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java A fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/StructType.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00001-1-13d79bd6-4b97-4680-b4e1-52e93b6ce04e-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00006-6-305c9b7a-f42d-4245-b806-dfa7a792593f-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00009-9-224fe2d6-b0d9-42d6-bc95-15f52ecb29ad-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00017-17-1a38e294-5992-48d9-a18e-08e129bb418c-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00023-23-74cfcf22-3de2-489a-b1ec-d5141e75a8e8-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00027-27-5f91dc85-b8f3-4cc2-a5c6-38b7fee49709-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00030-30-dc3510cc-e765-43bc-be03-c5561a8d50a3-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00031-31-364afc4a-b718-406d-a532-58fab5c8f85d-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00004-4-7a1a8e89-8aeb-4405-be64-76557432cf21-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00014-14-765d552a-fddc-42f3-adfd-ecba20a01d80-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00015-15-9957db43-3b9a-4a50-9946-d003cc1d461c-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00019-19-1e1895d0-1f42-4c30-989f-968802831077-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00020-20-bb59ac6d-aeee-4c35-9f8a-1a03127d33b8-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00028-28-44ba3ad9-737c-4416-a32c-501cc9a4aa90-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00003-3-31478795-ff6a-4a20-9fff-8dc4907c1ba7-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00007-7-3b4a35cb-22b5-4a5d-932b-89f222b0b2c7-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00011-11-a40c4c8c-de16-487a-89b3-ee8e58b4fd07-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00012-12-ed4ba706-2ace-4d41-b475-6bda3ba72306-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00022-22-57f5e226-5f8f-49b9-b998-039b8362b7a0-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00025-25-5fec3538-052d-493e-9479-b59fc8aece0f-00000.parquet A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/4ceee3ab-8653-423c-b8ac-0ad5f7b0579b-m0.avro A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/snap-8063525763252762178-1-4ceee3ab-8653-423c-b8ac-0ad5f7b0579b.avro A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/v1.metadata.json A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/v2.metadata.json A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/version-hint.text M testdata/datasets/functional/functional_schema_template.sql M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test M testdata/workloads/functional-query/queries/QueryTest/parquet-resolution-by-name.test M testdata/workloads/functional-query/queries/QueryTest/set.test M tests/query_test/test_scanners.py 53 files changed, 715 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16788/6 -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 6 Gerrit-Owner: wangsheng <sky...@163.com> Gerrit-Reviewer: Gabor Kaszab <gaborkas...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com> Gerrit-Reviewer: wangsheng <sky...@163.com>
