Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17877 )
Change subject: IMPALA-10935: Impala crashes on old Iceberg table property ...................................................................... IMPALA-10935: Impala crashes on old Iceberg table property With IMPALA-10627 we switched to use standard Iceberg table properties: https://iceberg.apache.org/configuration/ E.g. we switched from 'iceberg.file_format' to 'write.format.default'. For backward compatibility we also support 'iceberg.file_format'. Though the support is not perfect as it causes a crash in some cases. Impala crashes when the following conditions met: * local catalog mode is being used * Iceberg table is being queried * the data file format is ORC * 'iceberg.file_format' is set instead of 'write.format.default' table property * Query is "select count(*) from t;" Impala wrongly assumes that PARQUET is being used and tries to apply the count star optimization. It is not implemented for the ORC scanner and causes it to crash. This patch fixes the wrong assumption. Also it fixes the HdfsOrcScanner, so it won't crash in release mode but raise an error. This patch also enables UNSETting the file format table property for Iceberg tables. This table property was already enabled for modifications (changing the value via SET TBLPROPERTIES). Testing: * added e2e test for the above conditions Change-Id: Iafd9baef1c124d7356a14ba24c571567629a5e50 Reviewed-on: http://gerrit.cloudera.org:8080/17877 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M be/src/exec/hdfs-orc-scanner.cc M fe/src/main/java/org/apache/impala/analysis/AlterTableUnSetTblProperties.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test 9 files changed, 87 insertions(+), 25 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17877 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iafd9baef1c124d7356a14ba24c571567629a5e50 Gerrit-Change-Number: 17877 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com> Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
