Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21175 )
Change subject: IMPALA-12829: Skip processing transaction events if the table is HMS sync disabled. ...................................................................... Patch Set 20: (5 comments) http://gerrit.cloudera.org:8080/#/c/21175/19//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21175/19//COMMIT_MSG@24 PS19, Line 24: 1) CatalogServiceCatalog#reloadTableIfExists() didn't verify if the : current eventId is older than the table's lastSyncEventId which leads to : unecessary reloading of table for commit txns. : 2) Insert queries from impala didn't update the validWriteIdList for : transactional tables in the cache, so CommitTxn events triggered by : insert events are triggering reload on unpartitioned transactional : tables again while consuming these CommitTxn events. Fixed it by : updating the validWriteIdList in the cache. : 3) CommitTxn events generated after AlterTable events are leading to : incorrect results if file metadata reload is skipped in AlterTable : events. Reason being AlterTable event will update the writeId from : metastore but doesn't reload filemetadata which yields incorrect : results. This is fixed in HdfsTable class to not skip filemetadata : reload if writeId is changed. > These seem a lot. It'd be nice if we split the patch to smaller fixes to ea I have tried moving out some fixes, but all of them stem from https://gerrit.cloudera.org/c/21175/19/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java#1019, which is essential for this test. I like to keep them together as all of them are interdependent. http://gerrit.cloudera.org:8080/#/c/21175/19/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/21175/19/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3143 PS19, Line 3143: committe > nit: IIUC, these are committed writeIds, not open/aborted writeIds. If so, It is either committedWriteIds or null (coming from https://gerrit.cloudera.org/c/21175/19/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#1116). I'll rename it to committedWriteIds for simplicity. http://gerrit.cloudera.org:8080/#/c/21175/19/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@4568 PS19, Line 4568: ventSyncDisabled(org.a > If there are no parameters on this table, should we check the db parameters Good point. MetastoreEvents.MetastoreTableEvent.getHmsSyncProperty(tbl) is anyway checking for table parameters. http://gerrit.cloudera.org:8080/#/c/21175/19/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/21175/19/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1296 PS19, Line 1296: loadParams.isLoadPartitionFileMetadata() > Do we need to check 'prevWriteIdChanged' here as well? This case is already handled for the partitioned table in loadFileMetadataForPartitions(). I have added a partitioned table in the test case to verify the same. http://gerrit.cloudera.org:8080/#/c/21175/19/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2973 PS19, Line 2973: eIdList != null) > Is it the right way to compare ValidWriteIdList? The type of validWriteIds_ Ack -- To view, visit http://gerrit.cloudera.org:8080/21175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5d0ecb3b756755bc04c66a538a9ae6b88011a019 Gerrit-Change-Number: 21175 Gerrit-PatchSet: 20 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Wed, 25 Jun 2025 21:25:28 +0000 Gerrit-HasComments: Yes
