Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20367 )
Change subject: IMPALA-10976: Sync db/table to latest HMS event for all DDL/DMLs ...................................................................... Patch Set 16: (5 comments) http://gerrit.cloudera.org:8080/#/c/20367/16/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/20367/16/fe/src/main/java/org/apache/impala/catalog/Table.java@1073 PS16, Line 1073: lastRefreshEventId_ = eventId; 'lastRefreshEventId_' is also used in skipping events: https://github.com/apache/impala/blob/a6de494f24c47fbd679a037341ae0a34b9f696ff/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1075 Is it ok to update it if we haven't reload the full set of metadata (i.e. isSetLastSyncEventId=false)? http://gerrit.cloudera.org:8080/#/c/20367/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/20367/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2629 PS16, Line 2629: upt-to-date nit: "up-to-date" http://gerrit.cloudera.org:8080/#/c/20367/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3210 PS16, Line 3210: if (isTableBeingReplicated(hmsClient, hdfsTable)) { : String dbName = Preconditions.checkNotNull(hdfsTable.getDb()).getName(); : MetastoreShim.truncateTable(hmsClient, dbName, hdfsTable.getName(), null, : tblTxn.validWriteIds, tblTxn.writeId); I think we can reuse this when syncToLatestEventOnDdls is turned on. It generates HMS events so we can fetch and apply them. http://gerrit.cloudera.org:8080/#/c/20367/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3297 PS16, Line 3297: IcebergCatalogOpExecutor.truncateTable(iceTxn); Will this generate HMS events? http://gerrit.cloudera.org:8080/#/c/20367/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3322 PS16, Line 3322: if (isTableBeingReplicated(metaStoreClient.getHiveClient(), hdfsTable)) { : isTableBeingReplicated = true; : String dbName = Preconditions.checkNotNull(hdfsTable.getDb()).getName(); : metaStoreClient.getHiveClient() : .truncateTable(dbName, hdfsTable.getName(), null); Same here. I think we can reuse this when syncToLatestEventOnDdls is turned on. It generates HMS events so we can fetch and apply them. -- To view, visit http://gerrit.cloudera.org:8080/20367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf Gerrit-Change-Number: 20367 Gerrit-PatchSet: 16 Gerrit-Owner: Sai Hemanth Gantasala <saihema...@cloudera.com> Gerrit-Reviewer: Anonymous Coward <k.venureddy2...@gmail.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Sai Hemanth Gantasala <saihema...@cloudera.com> Gerrit-Comment-Date: Fri, 15 Dec 2023 08:17:50 +0000 Gerrit-HasComments: Yes
