Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )
Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate ...................................................................... Patch Set 9: (22 comments) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@294 PS9, Line 294: * Initializes Metastore event processor object if line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@46 PS9, Line 46: * <code>org.apache.hadoop.hive.metastore.api.NotificationEvent</code>. Metastore can be configured, line too long (100 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@47 PS9, Line 47: * to work with Listeners which are called on various DDL operations like create/alter/drop line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@48 PS9, Line 48: * operations on database, table, partition etc. Each event has a unique incremental id and the line too long (95 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@49 PS9, Line 49: * generated events are be fetched from Metastore to get incremental updates to the metadata stored line too long (99 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@50 PS9, Line 50: * in Hive metastore using the the public API <code>get_next_notification</code> These events could line too long (99 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@51 PS9, Line 51: * be generated by external Metastore clients like Apache Hive or Apache Spark as well as other line too long (95 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@54 PS9, Line 54: * This class is used to poll metastore for new events at a given frequency. By applying such events line too long (100 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@55 PS9, Line 55: * on the catalogD we can sync to external metadata operations by taking appropriate actions for line too long (96 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@56 PS9, Line 56: * each event type. We keep track of the last synced event id in each polling iteration so the next line too long (99 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@64 PS9, Line 64: * A new table/database is created in Catalog respectively. The newly created table/database are line too long (98 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@69 PS9, Line 69: * In case of alter table event, currently the code issues a invalidate table command. There is a line too long (99 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@70 PS9, Line 70: * special case of this event in case of renames, where the old table is removed and a new line too long (92 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@73 PS9, Line 73: * In case of alter database events, currently only the case of changing default location, owner line too long (98 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@76 PS9, Line 76: * Currently, in case of these events, we issue invalidate on the table. This can be optimized by line too long (99 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@162 PS9, Line 162: // fetch the current notification event id. We assume that the polling interval is small line too long (94 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@163 PS9, Line 163: // enough that most of these polling operations result in zero new events. In such a case, line too long (96 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@164 PS9, Line 164: // fetching current notification event id is much faster (and cheaper on HMS side) instead line too long (96 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@186 PS9, Line 186: .format("Unable to process notification event %d due to %s. Ignoring this event...", line too long (96 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@189 PS9, Line 189: // Make sure to update lastSyncedEventId_ in case there are errors while processing the line too long (95 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@190 PS9, Line 190: // event. Otherwise, the sync thread will be stuck forever at this bad event and keep line too long (93 > 90) http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@72 PS9, Line 72: eventsProcessor = MetastoreEventsProcessor.getOrCreate(catalog, currentNotificationId.getEventId(), 0L); line too long (110 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12118 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7 Gerrit-Change-Number: 12118 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward <vih...@cloudera.com> Gerrit-Reviewer: Anonymous Coward <vih...@cloudera.com> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Comment-Date: Thu, 27 Dec 2018 00:58:48 +0000 Gerrit-HasComments: Yes
