Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20367 )
Change subject: IMPALA-10976: Sync db/table in catalogD to latest HMS event id after the DDL operation for all DDLS from Impala clients ...................................................................... Patch Set 6: (5 comments) http://gerrit.cloudera.org:8080/#/c/20367/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/20367/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@352 PS5, Line 352: Exceptio > Metrics is not thread-safe. We shouldn't use the one of event-processor. Ack http://gerrit.cloudera.org:8080/#/c/20367/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@358 PS5, Line 358: String.format("Write lock is not held on table %s by current thread", : tbl.getFullName())); : long lastEventId = tbl.getLastSyncedEventId(); : Preconditions.checkArgument(lastEventId > 0, "lastEvent " + : " Id %s for table %s should be greater than 0", lastEventId, tbl.getFullName()); : > nit: remove empty parameter comments. They seem verbose. Ack http://gerrit.cloudera.org:8080/#/c/20367/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/20367/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1276 PS4, Line 1276: reloadTableSchema = true; > This is used to identify if the events are from the current Impala service. Ack http://gerrit.cloudera.org:8080/#/c/20367/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@7004 PS4, Line 7004: "Action: Manually uncache directory %s via hdfs " + : "cacheAdmin.", part.getDbName(), part.getTableName(), : part.getValues(), part.getSd().getLocation()); : LOG.error(msg, e); : errorMessages.add(msg); : > +1 for this. I don't see this is addressed in the latest patch set (PS5). I I reverted the changes for the Insert operation in PS5, so I thought this point was not valid anymore. Since, in one of the comments you mentioned that all DDLs, and DMLs code paths should have this flag/feature, I'll now work on this comment. http://gerrit.cloudera.org:8080/#/c/20367/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@7040 PS4, Line 7040: "regarding the failed un/caching operations."); : response.getResult().setStatus( : new TStatus(TErrorCode.INTERNAL_ERROR, errorMessages)); : } else { : response.getResult().setStatus( : new TStatus(TErrorCode.OK, new ArrayList<String>())); : } : boolean syncToLatestEventId = : BackendConfig.INSTANCE.enableSyncToLatestEventOnDdls(); : / > I think to be consistent, we should do this for all catalog operations, inc ACK. Regarding, the Refresh/Invalidate query, >Invalidate marks the metadata object as stale and there is no point in >updating the cache for the metadata object and then marking it as stale. >Refresh currently updates the cache in place, so should we leave it as such? >since depending on meta store events processor is additional complexity? Do >you agree? -- To view, visit http://gerrit.cloudera.org:8080/20367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf Gerrit-Change-Number: 20367 Gerrit-PatchSet: 6 Gerrit-Owner: Sai Hemanth Gantasala <saihema...@cloudera.com> Gerrit-Reviewer: Anonymous Coward <k.venureddy2...@gmail.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Sai Hemanth Gantasala <saihema...@cloudera.com> Gerrit-Comment-Date: Sun, 05 Nov 2023 21:12:51 +0000 Gerrit-HasComments: Yes
