Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17308 )
Change subject: IMPALA-10502: Handle CREATE/DROP events correctly ...................................................................... Patch Set 15: (25 comments) This change makes a lot of sense. I mostly left nit comments. Looks good overall, but I'm planning to take another look in the following days. http://gerrit.cloudera.org:8080/#/c/17308/15//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17308/15//COMMIT_MSG@12 PS15, Line 12: For example if we have a sequence of : CREATE_TABLE, DROP_TABLE, CREATE_TABLE ... on the same table, it is : possible that when the create table event is being processed, the table : is not present in catalog because it was dropped recently. In such a : case, events processor does not have enough state information in : catalogd to determine that this table has been dropped from the : catalogd and the event should be ignored. Maybe it would be easier to understand if we had list the events here, e.g.: 1. CREATE_TABLE foo statement -> Catalog creates table foo 2. DROP_TABLE foo statement -> Catalog drops table foo ... a bit after event processor sees the following event: 3. CREATE_TABLE foo event -> foo is not in Catalog, cannot decide whether we should recreate the table or ignore this event. http://gerrit.cloudera.org:8080/#/c/17308/15//COMMIT_MSG@29 PS15, Line 29: parition typo: partition http://gerrit.cloudera.org:8080/#/c/17308/15//COMMIT_MSG@30 PS15, Line 30: the the duplicate 'the' http://gerrit.cloudera.org:8080/#/c/17308/15//COMMIT_MSG@35 PS15, Line 35: Added a new test Could you please the name of the test? http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2911 PS15, Line 2911: public TCatalogObject reloadHdfsPartition(HdfsTable hdfsTable, String partitionName, nit: add comment please http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java: http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@709 PS15, Line 709: -1L nit: for readability you could write: /*createEventId=*/-1L http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@899 PS15, Line 899: = nit: missing space after = http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java: http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@268 PS15, Line 268: null, reason); nit: fits prev line http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/TableLoader.java File fe/src/main/java/org/apache/impala/catalog/TableLoader.java: http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/TableLoader.java@86 PS15, Line 86: createdEventId nit: 'createdEventId' is only used at the callsite, so probably refer to 'eventId'. Same for next line. http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/TableLoader.java@87 PS15, Line 87: are not created by : // Impala not created by the current CatalogD process? http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java File fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java: http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java@33 PS15, Line 33: a typo: an http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java@41 PS15, Line 41: SortedMap optional: I'd assume that the events would come in more or less sorted order, therefore using a linked list might be more efficient. http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java@57 PS15, Line 57: a nit: an http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@589 PS15, Line 589: CatalogException e) { : if (e instanceof TableLoadingException : || e instanceof DatabaseNotFoundException) { Could be: catch (TableLoadingException | DatabaseNotFoundException e) http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@631 PS15, Line 631: }i no space between '}' and 'is' http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@632 PS15, Line 632: getFullyQualifiedTblName()); getFullyQualifiedTblName() is passed 2 times. http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1104 PS15, Line 1104: nit: unnecessary whitespace at the end. http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@108 PS15, Line 108: {code} nit: at other places in this file we are using <code> and </code> http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@268 PS15, Line 268: .getEvents(); nit: fits prev line http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@272 PS15, Line 272: ); probably add 'e' as well to get more information about the error. http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@547 PS15, Line 547: public List<NotificationEvent> getNextMetastoreEvents(final long eventId, nit: please add comment http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@580 PS15, Line 580: if is http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@764 PS15, Line 764: added a nit: adding an? http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@864 PS15, Line 864: such : // a event. nit: such event? http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2258 PS15, Line 2258: .equals(event.getEventType()) || nit: for readability this could go to the prev line, then the remaining parts of the condition could fit a single line -- To view, visit http://gerrit.cloudera.org:8080/17308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia2c5e96b48abac015240f20295b3ec3b1d71f24a Gerrit-Change-Number: 17308 Gerrit-PatchSet: 15 Gerrit-Owner: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Sourabh Goyal <soura...@cloudera.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com> Gerrit-Comment-Date: Mon, 05 Jul 2021 14:23:19 +0000 Gerrit-HasComments: Yes