Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21175 )
Change subject: IMPALA-12829: Skip processing transaction events if the table is HMS sync disabled. ...................................................................... Patch Set 19: (5 comments) http://gerrit.cloudera.org:8080/#/c/21175/19//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21175/19//COMMIT_MSG@24 PS19, Line 24: 1) CatalogServiceCatalog#reloadTableIfExists() didn't verify if the : current eventId is older than the table's lastSyncEventId which leads to : unecessary reloading of table for commit txns. : 2) Insert queries from impala didn't update the validWriteIdList for : transactional tables in the cache, so CommitTxn events triggered by : insert events are triggering reload on unpartitioned transactional : tables again while consuming these CommitTxn events. Fixed it by : updating the validWriteIdList in the cache. : 3) CommitTxn events generated after AlterTable events are leading to : incorrect results if file metadata reload is skipped in AlterTable : events. Reason being AlterTable event will update the writeId from : metastore but doesn't reload filemetadata which yields incorrect : results. This is fixed in HdfsTable class to not skip filemetadata : reload if writeId is changed. These seem a lot. It'd be nice if we split the patch to smaller fixes to ease backporting some of them. http://gerrit.cloudera.org:8080/#/c/21175/19/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/21175/19/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3143 PS19, Line 3143: writeIds nit: IIUC, these are committed writeIds, not open/aborted writeIds. If so, please rename it to 'committedWriteIds'. http://gerrit.cloudera.org:8080/#/c/21175/19/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@4568 PS19, Line 4568: !tbl.isSetParameters() If there are no parameters on this table, should we check the db parameters? http://gerrit.cloudera.org:8080/#/c/21175/19/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/21175/19/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1296 PS19, Line 1296: loadParams.isLoadPartitionFileMetadata() Do we need to check 'prevWriteIdChanged' here as well? http://gerrit.cloudera.org:8080/#/c/21175/19/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2973 PS19, Line 2973: toString().equals Is it the right way to compare ValidWriteIdList? The type of validWriteIds_ is MutableValidWriteIdList. Comparing a String with it always get false. -- To view, visit http://gerrit.cloudera.org:8080/21175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5d0ecb3b756755bc04c66a538a9ae6b88011a019 Gerrit-Change-Number: 21175 Gerrit-PatchSet: 19 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Thu, 19 Jun 2025 07:44:36 +0000 Gerrit-HasComments: Yes
