Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18626 )
Change subject: IMPALA-9670: Fix unloaded views are shown as tables for GET_TABLES requests ...................................................................... IMPALA-9670: Fix unloaded views are shown as tables for GET_TABLES requests At startup, catalogd pulls the table names from HMS and tracks each table using an IncompleteTable which only contains the table name. The table types (TABLE/VIEW) and comments are unknown until the table/view is loaded in catalogd. GET_TABLES is a request of the HS2 protocol. It fetches all the tables with their types and comments. For unloaded tables/views, Impala always returns them with TABLE type (the default) and empty comments. This patch enables catalogd to always load the table types and comments along with the table names. This behavior is controlled by a catalogd-only flag, --pull_table_types_and_comments, which is false by default. When this flag is enabled, catalogd will load table types and comments at startup and in executing INVALIDATE METADATA commands. In other words, an unloaded table (IncompleteTable) now not just contains the table name, but also contains the correct table type and comment. This is implemented by using the getTableMetas HMS API when invalidating a table. The original behavior uses getAllTables to load all table names and uses tableExists to verify whether a table still exists. When the flag is set, we'll use getTableMetas instead to also load the table types and comments. Implementation: Add a new table type, UNLOADED_TABLE, in TTableType to identify tables that we just know it's not a view, but don’t know whether it's a Kudu or HDFS table since its full set of metadata is unloaded. When propagating catalog objects from catalogd to coordinators, views are sent using a catalog key explicitly prefixed by VIEW. So coordinators can create IncompleteTables/LocalIncompleteTables with the correct types. In most of the cases in creating an IncompleteTable, we have the table types and comments in the context. For instance, when adding an IncompleteTable for a CreateTable/CreateView request, we know exactly it's a table or view. So we can create IncompleteTables with the correct types. Test infra changes: - Adds get_tables() method for the hs2_client - Extends ImpalaTestSuite.create_client_for_nth_impalad() to support hs2 and hs2-http protocols. So we can create HS2 clients on all impalads. Tests: - Add custom cluster tests on all catalog modes (with/without local-catalog or event processor). Verify the table types and comments are always correct when pull_table_types_and_comments is true. Change-Id: I528bb20272ebdd66a0118c30efc2b0566f2b0e2f Reviewed-on: http://gerrit.cloudera.org:8080/18626 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeTable.java M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/View.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/FailedLoadLocalTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalView.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java M fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java M testdata/bin/generate-schema-statements.py M testdata/datasets/functional/functional_schema_template.sql M tests/common/impala_connection.py M tests/common/impala_test_suite.py A tests/custom_cluster/test_preload_table_types.py 34 files changed, 569 insertions(+), 202 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I528bb20272ebdd66a0118c30efc2b0566f2b0e2f Gerrit-Change-Number: 18626 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Aman Sinha <amsi...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Yu-Wen Lai <yu-wen....@cloudera.com>
