Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19509 )
Change subject: IMPALA-11509: Prevent queries hanging when Iceberg metadata is missing. ...................................................................... IMPALA-11509: Prevent queries hanging when Iceberg metadata is missing. Traditionally table metadata is loaded by the catalog and sent as thrift to the Impala daemons. With Iceberg tables, some metadata, for example the org.apache.iceberg.Table, is loaded in the Coordinator at the same time as the thrift description is being deserialized. If the loading of the org.apache.iceberg.Table fails, perhaps because of missing Iceberg metadata, then the loading of the table fails. This can cause an infinite loop as StmtMetadataLoader.loadTables() waits hopefully for the catalog to send a new version of the table. Change some Iceberg table loading methods to throw IcebergTableLoadingException when a failure occurs. Prevent the hang by substituting in an IncompleteTable if an IcebergTableLoadingException occurs. The test test_drop_incomplete_table had previously been disabled because of IMPALA-11509. To re-enable this required a second change. The way that DROP TABLE is executed on an iceberg table depends on which Iceberg catalog is being used. If this Iceberg catalog is not a Hive catalog then the execution happens in two parts, first the Iceberg table is dropped, then the table is dropped in HMS. If this case, if the drop fails in Iceberg, we should still continue on to perform the drop in HMS. TESTING - Add a new test, originally developed for IMPALA-11330, which tests failures after deleting Iceberg metadata. - Re-enable test_drop_incomplete_table(). Change-Id: I695559e21c510615918a51a4b5057bc616ee5421 Reviewed-on: http://gerrit.cloudera.org:8080/19509 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalog.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalogs.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHadoopCatalog.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHadoopTables.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHiveCatalog.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test M tests/query_test/test_iceberg.py 10 files changed, 95 insertions(+), 34 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19509 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I695559e21c510615918a51a4b5057bc616ee5421 Gerrit-Change-Number: 19509 Gerrit-PatchSet: 11 Gerrit-Owner: Andrew Sherman <asher...@cloudera.com> Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com> Gerrit-Reviewer: Gabor Kaszab <gaborkas...@cloudera.com> Gerrit-Reviewer: Gergely Fürnstáhl <gfurnst...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>