[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. Patch Set 15: Triggered dry-run external to verify that all the tests pass. -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 15 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 07:47:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Bharath Krishna has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. IMPALA-8419 : Validate event processing related configurations Using the Metastore API to get the configuration values, verify that the configurations needed for event processing are set correctly. Also check that the parameters required for event processing is not filtered out by the Hive config METASTORE_PARAMETER_EXCLUDE_PATTERNS. This validation is done while creating the event processor and throws CatalogException if the configuration is incorrect. Testing - Added unit tests Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 --- A fe/src/main/java/org/apache/impala/catalog/events/EventProcessorConfigValidator.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java 7 files changed, 530 insertions(+), 67 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/13019/15 -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 15 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Bharath Krishna has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. IMPALA-8419 : Validate event processing related configurations Using the Metastore API to get the configuration values, verify that the configurations needed for event processing are set correctly. Also check that the parameters required for event processing is not filtered out by the Hive config METASTORE_PARAMETER_EXCLUDE_PATTERNS. This validation is done while creating the event processor and throws CatalogException if the configuration is incorrect. Testing - Added unit tests Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 --- A fe/src/main/java/org/apache/impala/catalog/events/EventProcessorConfigValidator.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java 7 files changed, 530 insertions(+), 67 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/13019/14 -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 14 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Bharath Krishna has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. IMPALA-8419 : Validate event processing related configurations Using the Metastore API to get the configuration values, verify that the configurations needed for event processing are set correctly. Also check that the parameters required for event processing is not filtered out by the Hive config METASTORE_PARAMETER_EXCLUDE_PATTERNS. This validation is done while creating the event processor and throws CatalogException if the configuration is incorrect. Testing - Added unit tests Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 --- A fe/src/main/java/org/apache/impala/catalog/events/EventProcessorConfigValidator.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java 7 files changed, 529 insertions(+), 67 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/13019/13 -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 13 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/13019/9/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13019/9/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@274 PS9, Line 274: or (TestIncorrectMetastoreEventConfigs config : > maybe also assert the error message? Done -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 11 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 17:39:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Bharath Krishna has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. IMPALA-8419 : Validate event processing related configurations Using the Metastore API to get the configuration values, verify that the configurations needed for event processing are set correctly. Also check that the parameters required for event processing is not filtered out by the Hive config METASTORE_PARAMETER_EXCLUDE_PATTERNS. This validation is done while creating the event processor and throws CatalogException if the configuration is incorrect. Testing - Added unit tests Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 --- A fe/src/main/java/org/apache/impala/catalog/events/EventProcessorValidator.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java 7 files changed, 520 insertions(+), 61 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/13019/11 -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 11 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 5: Code-Review+1 Thanks Xiaomeng for the updates in patch. LGTM. -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 27 Apr 2019 00:03:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Bharath Krishna has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. IMPALA-8419 : Validate event processing related configurations Using the Metastore API to get the configuration values, verify that the configurations needed for event processing are set correctly. Also check that the parameters required for event processing is not filtered out by the Hive config METASTORE_PARAMETER_EXCLUDE_PATTERNS. This validation is done while creating the event processor and throws CatalogException if the configuration is incorrect. Testing - Added unit tests Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 --- A fe/src/main/java/org/apache/impala/catalog/events/EventProcessorValidator.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java 7 files changed, 507 insertions(+), 59 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/13019/9 -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Bharath Krishna has posted comments on this change. (
http://gerrit.cloudera.org:8080/13049 )
Change subject: IMPALA-8149 : Add support for alter_database events
..
Patch Set 4:
(4 comments)
Overall looks good, leaving a few comments..
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@757
PS4, Line 757:* If tblName is null, applies to database.
We can say:
if tblName is null, removes version number from database.
if tblName is not null and Table is not incomplete, removes version number from
Table.
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3805
PS1, Line 3805: if (db == null) {
> Remove it should be fine?
I think removing is fine. Probably we can change the Preconditions.checkState()
in catalog_.getDb to PreConditions.checkArguments() as Fredy suggested?
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@345
PS4, Line 345:
I feel we should add an additional test, here we are testing if the
Notification Processor is successfully processing the notification events
generated from Hive.
We are not triggering the AlterDatabase operation from Impala in this test. (We
need to do something similar to
MetastoreEventsProcessorTest#alterTableAddColsFromImpala).
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@1481
PS4, Line 1481: eventsProcessor_.processEvents();
Can we check if possible that the counter
MetastoreEventsProcessor.EVENTS_SKIPPED_METRIC is updated correctly here, as we
are skipping some events?
--
To view, visit http://gerrit.cloudera.org:8080/13049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b
Gerrit-Change-Number: 13049
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Krishna
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Thu, 25 Apr 2019 20:46:33 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Bharath Krishna has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. IMPALA-8419 : Validate event processing related configurations Using the Metastore API to get the configuration values, verify that the configurations needed for event processing are set correctly. Also check that the parameters required for event processing is not filtered out by the Hive config METASTORE_PARAMETER_EXCLUDE_PATTERNS. This validation is done while creating the event processor and throws CatalogException if the configuration is incorrect. Testing - Added unit tests Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 --- A fe/src/main/java/org/apache/impala/catalog/events/EventProcessorValidation.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java 7 files changed, 493 insertions(+), 59 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/13019/7 -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 7 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 21: How does this work for the tests in the MetastoreEventsProcessorTest , does it need dml.events = true as well? -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 21 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 22:41:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Bharath Krishna has posted comments on this change. (
http://gerrit.cloudera.org:8080/13049 )
Change subject: IMPALA-8149 : Add support for alter_database events
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@978
PS1, Line 978: public void process() throws DatabaseNotFoundException,
CatalogException {
Looks like DatabaseNotFoundException is already captured in CatalogException,
so probaly we can remove DatabaseNotFoundException from throws.
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3812
PS1, Line 3812: long newCatalogVersion =
catalog_.incrementAndGetCatalogVersion();
Just asking to understand this part:
Do we need to lock the DB when we update the version?
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3813
PS1, Line 3813: addCatalogServiceIdentifiers(db,
catalog_.getCatalogServiceId(), newCatalogVersion);
I see that the only alter operation here is SET_OWNER, so shouldn't we update
the catalogServiceIdentifiers only when the operation is ALTER DB SET OWNER?
Otherwise, why should we just update the version?
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3844
PS1, Line 3844: synchronized (metastoreDdlLock_) {
Should we do addCatalogServiceIdentifiers after acquiring the metastoreDdlLock_
?
--
To view, visit http://gerrit.cloudera.org:8080/13049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b
Gerrit-Change-Number: 13049
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Krishna
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Thu, 18 Apr 2019 03:29:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. Patch Set 4: (1 comment) Thanks Vihang for the comment. Bharath, I have updated the patch accordingly. http://gerrit.cloudera.org:8080/#/c/13058/3/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13058/3/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@700 PS3, Line 700: // Adding sleep here to make sure that the CREATION_TIME is not same : // as the previous CREATE_TABLE operation, so as to trigger the filtering logic : // based on CREATION_TIME in DROP_TABLE event processing. This is currently a : // limitation : the DROP_TABLE event filtering expects that while processing events, : // the CREATION_TIME of two tables with same name won't have the same : // creation timestamp > I think it would be good to add the following comment in the DropTableEvent Done -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 03:02:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Bharath Krishna has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase The test fails because of two Databases getting created with same CREATION_TIME. Hence, adding a sleep of 2 seconds to avoid this case. Also fixing other tests with similar use-case. Testing - Fixed the unit tests Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 40 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/13058/4 -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@278 PS1, Line 278: // Adding sleep here to make sure that the CREATION_TIME is not same : // as the previous CREATE_DB operation, so as to trigger the filtering logic : // based on CREATION_TIME in DROP_DB event processing. This is currently a : // limitation : the DROP_DB event filtering expects that while processing events, : // the CREATION_TIME of two Databases with same name won't have the same : // creation timestamp. : sleep(2000); > Like we discussed offlines, this is probably not needed. Consider removing Yes, I guess we don't need the sleep here because the first create will be filtered out, and the drop event will just be a no-op as the Table/DB is not yet created. -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 21:14:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8426: Logging error in DROP TABLE event processing
Bharath Krishna has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13056 ) Change subject: IMPALA-8426: Logging error in DROP_TABLE event processing .. IMPALA-8426: Logging error in DROP_TABLE event processing Fixing the bug in condition check while logging in DROP_TABLE event processing. Also updating EVENTS_SKIPPED metric to keep track of the number of drop table events skipped when CREATION_TIME matches. Testing: - Added metric check to unit test. - Ran existing unit tests. Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 14 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/13056/2 -- To view, visit http://gerrit.cloudera.org:8080/13056 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Gerrit-Change-Number: 13056 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. Patch Set 1: (2 comments) Hi Bharath, thanks for the review. I have added comments to explain the limitation about CREATION_TIME, let me know if it makes sense. http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@278 PS1, Line 278: // Adding sleep here to make sure that the CREATION_TIME is not same : // as the previous CREATE_DB operation, so as to trigger the filtering logic : // based on CREATION_TIME in DROP_DB event processing. This is currently a : // limitation : the DROP_DB event filtering expects that while processing events, : // the CREATION_TIME of two Databases with same name won't have the same : // creation timestamp. : sleep(2000); > Do you need this if all the creates are via Hive? (event processor is expec Yes, so the same applies to Hive as well. We don't want events to have same CREATION_TIME as mentioned below. http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@709 PS1, Line 709: This is currently a : // limitation : the DROP_TABLE event filtering expects that while processing events, : // the CREATION_TIME of two tables with same name won't have the same : // creation timestamp. > I'm not totally sure I understand how this is a limitation. Isn't the proce It is expected to rely on the ctime, but limitation is that we don't expect a CREATE_TABLE and DROP_TABLE on the same table happen in the same second. In that case, the logic would fail to filter out the event. -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 19:51:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Bharath Krishna has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13058 Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase The test fails because of two Databases getting created with same CREATION_TIME. Hence, adding a sleep of 2 seconds to avoid this case. Also fixing other tests with similar use-case. Testing - Fixed the unit tests Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d --- M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 1 file changed, 27 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/13058/1 -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8426: Logging error in DROP TABLE event processing
Bharath Krishna has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13056 Change subject: IMPALA-8426: Logging error in DROP_TABLE event processing .. IMPALA-8426: Logging error in DROP_TABLE event processing Fixing the bug in condition check while logging in DROP_TABLE event processing. Also updating EVENTS_SKIPPED metric to keep track of the number of drop table events skipped when CREATION_TIME matches. Testing: - Added metric check to unit test. - Ran existing unit tests. Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 14 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/13056/1 -- To view, visit http://gerrit.cloudera.org:8080/13056 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Gerrit-Change-Number: 13056 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8419 : Fetch metastore configuration values to detect misconfigured setups
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Fetch metastore configuration values to detect misconfigured setups .. Patch Set 4: Vihang, thanks for the review. I had a few questions on your comments so please let me know what you feel is a better choice. -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 04:28:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8419 : Fetch metastore configuration values to detect misconfigured setups
Bharath Krishna has posted comments on this change. (
http://gerrit.cloudera.org:8080/13019 )
Change subject: IMPALA-8419 : Fetch metastore configuration values to
detect misconfigured setups
..
Patch Set 4:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/13019/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/13019/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@315
PS4, Line 315: try {
> Use try-with-resources here to automatically close MetastoreClient once don
Do you mean to use try-with-resource in the above method which calls using
MetastoreClient?
I added the client as an input param so that I can Mock the test and extract
out this method as independent.
http://gerrit.cloudera.org:8080/#/c/13019/4/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
File fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java:
http://gerrit.cloudera.org:8080/#/c/13019/4/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java@88
PS4, Line 88: public static final String DEFAULT_METASTORE_CONFIG_VALUE =
> Why do we need this?
Will remove this if the below comment of mine doesn't sound like a good approach
http://gerrit.cloudera.org:8080/#/c/13019/4/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java@123
PS4, Line 123: public static String getMetastoreConfigValue(
> May be a more useful way is to add a input argument to provide a default va
I feel this is useful too when users can just compare the return value with
MetastoreUtil.DEFAULT_METASTORE_CONFIG_VALUE
Do you feel it would be better to add two methods :
getMetastoreConfigValue(client, config) (which calls the below method with
DEFAULT_METASTORE_CONFIG_VALUE)
and
getMetastoreConfigValue(client, config, defaultVal)
Or should I just have the latter one?
http://gerrit.cloudera.org:8080/#/c/13019/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/13019/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@128
PS1, Line 128: import org.slf4j.LoggerFactory;
Use only the required imports
http://gerrit.cloudera.org:8080/#/c/13019/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/13019/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@253
PS4, Line 253: when(mockIMetaStoreClient.getConfigValue(config.toString(),
: MetaStoreUtil.DEFAULT_METASTORE_CONFIG_VALUE))
: .thenReturn(config.getExpectedValue());
> Why do we care to revert the value on a mock client?
Here, I am iterating over each Config, and just toggling its value. If I don't
revert the value back, in the next iteration it will throw an exception in the
previous value itself. Please let me know if it makes sense to do that way or
if should consider some other alternative.
--
To view, visit http://gerrit.cloudera.org:8080/13019
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606
Gerrit-Change-Number: 13019
Gerrit-PatchSet: 4
Gerrit-Owner: Bharath Krishna
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Krishna
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Tue, 16 Apr 2019 04:26:48 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8419 : Fetch metastore configuration values to detect misconfigured setups
Bharath Krishna has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13019 Change subject: IMPALA-8419 : Fetch metastore configuration values to detect misconfigured setups .. IMPALA-8419 : Fetch metastore configuration values to detect misconfigured setups Using the Metastore API to get the configuration values, verify that the configurations needed for event processing are set correctly. This validation is done while creating the event processor and throws CatalogException if the configuration is incorrect. Testing - Added unit tests Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java 4 files changed, 135 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/13019/4 -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Krishna
[Impala-ASF-CR] IMPALA-8338 : Check CREATION TIME while processing DROP DATABASE events.
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/12938 ) Change subject: IMPALA-8338 : Check CREATION_TIME while processing DROP_DATABASE events. .. Patch Set 14: (3 comments) http://gerrit.cloudera.org:8080/#/c/12938/12//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12938/12//COMMIT_MSG@7 PS12, Line 7: Check CREATION_TIME while processing DROP_DATABASE events. : > wrap to single line? Something like "consider ctime while processing databa Done http://gerrit.cloudera.org:8080/#/c/12938/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/12938/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1546 PS12, Line 1546: if (catalogDb == null) return null; : : d > nit: single line Done http://gerrit.cloudera.org:8080/#/c/12938/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1552 PS12, Line 1552: = nu > nit: not needed Done -- To view, visit http://gerrit.cloudera.org:8080/12938 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c Gerrit-Change-Number: 12938 Gerrit-PatchSet: 14 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 15 Apr 2019 23:21:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8338 : Check CREATION TIME while processing DROP DATABASE events.
Bharath Krishna has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/12938 ) Change subject: IMPALA-8338 : Check CREATION_TIME while processing DROP_DATABASE events. .. IMPALA-8338 : Check CREATION_TIME while processing DROP_DATABASE events. Process the drop database event only if the CREATION_TIME of the catalog's database object is lesser than or equal to that of the database object present in the notification event. If the CREATION_TIME in the notification event object is lesser than the catalog's DB object, it means that the Database object present in the catalog is the latest and we can skip the event instead. Testing : - Added unit tests in MetastoreEventsProcessorTest. - Enabled testCreateDropCreateDatabaseFromImpala as we now have CREATION_TIME in the notification events. Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 130 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/12938/14 -- To view, visit http://gerrit.cloudera.org:8080/12938 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c Gerrit-Change-Number: 12938 Gerrit-PatchSet: 14 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8338 : Check CREATION TIME in event processor to avoid incorrect deletion of Database objects.
Bharath Krishna has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/12938 ) Change subject: IMPALA-8338 : Check CREATION_TIME in event processor to avoid incorrect deletion of Database objects. .. IMPALA-8338 : Check CREATION_TIME in event processor to avoid incorrect deletion of Database objects. Process the drop database event only if the CREATION_TIME of the catalog's database object is lesser than or equal to that of the database object present in the notification event. If the CREATION_TIME in the notification event object is lesser than the catalog's DB object, it means that the Database object present in the catalog is the latest and we can skip the event instead. Testing : - Added unit tests in MetastoreEventsProcessorTest. - Enabled testCreateDropCreateDatabaseFromImpala as we now have CREATION_TIME in the notification events. Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 131 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/12938/12 -- To view, visit http://gerrit.cloudera.org:8080/12938 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c Gerrit-Change-Number: 12938 Gerrit-PatchSet: 12 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8338 : Check CREATION TIME in event processor to avoid incorrect deletion of Database objects.
Bharath Krishna has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/12938 ) Change subject: IMPALA-8338 : Check CREATION_TIME in event processor to avoid incorrect deletion of Database objects. .. IMPALA-8338 : Check CREATION_TIME in event processor to avoid incorrect deletion of Database objects. Process the drop database event only if the CREATION_TIME of the catalog's database object is lesser than or equal to that of the database object present in the notification event. If the CREATION_TIME in the notification event object is lesser than the catalog's DB object, it means that the Database object present in the catalog is the latest and we can skip the event instead. Testing : - Added unit tests in MetastoreEventsProcessorTest. - Enabled testCreateDropCreateDatabaseFromImpala as we now have CREATION_TIME in the notification events. Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 137 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/12938/10 -- To view, visit http://gerrit.cloudera.org:8080/12938 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c Gerrit-Change-Number: 12938 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8338 : Check CREATION TIME in event processor to avoid incorrect deletion of Database objects.
Bharath Krishna has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/12938 ) Change subject: IMPALA-8338 : Check CREATION_TIME in event processor to avoid incorrect deletion of Database objects. .. IMPALA-8338 : Check CREATION_TIME in event processor to avoid incorrect deletion of Database objects. Process the drop database event only if the CREATION_TIME of the catalog's database object is lesser than or equal to that of the database object present in the notification event. If the CREATION_TIME in the notification event object is lesser than the catalog's DB object, it means that the Database object present in the catalog is the latest and we can skip the event instead. Testing : - Added unit tests in MetastoreEventsProcessorTest. - Enabled testCreateDropCreateDatabaseFromImpala as we now have CREATION_TIME in the notification events. Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 136 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/12938/9 -- To view, visit http://gerrit.cloudera.org:8080/12938 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c Gerrit-Change-Number: 12938 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 1009254
Bharath Krishna has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12965 ) Change subject: Bump CDH_BUILD_NUMBER to 1009254 .. Bump CDH_BUILD_NUMBER to 1009254 This change bumps the CDH_BUILD_VERSION to a version that includes the change to have full-thrift object for DropDatabaseMessage in Metastore notifications, added in HIVE-21526. This change is needed for the upcoming patch for IMPALA-8338. Testing: - Ran a full exhaustive build using the impala-private-parameterized job. Change-Id: Id38bae921c4d93421c6c72cdccef6a4783e2588e --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/12965/3 -- To view, visit http://gerrit.cloudera.org:8080/12965 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id38bae921c4d93421c6c72cdccef6a4783e2588e Gerrit-Change-Number: 12965 Gerrit-PatchSet: 3 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8338 : Check CREATION TIME in event processor to avoid incorrect deletion of Database objects.
Bharath Krishna has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/12938 ) Change subject: IMPALA-8338 : Check CREATION_TIME in event processor to avoid incorrect deletion of Database objects. .. IMPALA-8338 : Check CREATION_TIME in event processor to avoid incorrect deletion of Database objects. Process the drop database event only if the CREATION_TIME of the catalog's database object is lesser than or equal to that of the database object present in the notification event. If the CREATION_TIME in the notification event object is lesser than the catalog's DB object, it means that the Database object present in the catalog is the latest and we can skip the event instead. Testing : - Added unit tests in MetastoreEventsProcessorTest. - Enabled testCreateDropCreateDatabaseFromImpala as we now have CREATION_TIME in the notification events. Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 4 files changed, 129 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/12938/8 -- To view, visit http://gerrit.cloudera.org:8080/12938 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c Gerrit-Change-Number: 12938 Gerrit-PatchSet: 8 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8338 : Check CREATION TIME in event processor to avoid incorrect deletion of Database objects.
Bharath Krishna has posted comments on this change. (
http://gerrit.cloudera.org:8080/12938 )
Change subject: IMPALA-8338 : Check CREATION_TIME in event processor to avoid
incorrect deletion of Database objects.
..
Patch Set 7:
(14 comments)
Thanks for the review Vihang. Updated patch with comments resolved.
http://gerrit.cloudera.org:8080/#/c/12938/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12938/4//COMMIT_MSG@8
PS4, Line 8: incorrect deletion of Database
> I think this is not accurate. creation_time is not used for invalidating bu
Done
http://gerrit.cloudera.org:8080/#/c/12938/4//COMMIT_MSG@15
PS4, Line 15: we can skip the event
: instead.
> suggest you to change it to ".. and we can skip the event instead" which is
Done
http://gerrit.cloudera.org:8080/#/c/12938/4/be/src/util/event-metrics.h
File be/src/util/event-metrics.h:
http://gerrit.cloudera.org:8080/#/c/12938/4/be/src/util/event-metrics.h@40
PS4, Line 40: static DoubleGauge* EVENTS_FETCH_DURATION_MEAN;
> Is there any specific value in exposing this metric on the Catalog metrics
Done
http://gerrit.cloudera.org:8080/#/c/12938/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/12938/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@953
PS4, Line 953: public List getTableProperties(
> like suggested in my other comment related to race conditions , this API in
Removed this method
http://gerrit.cloudera.org:8080/#/c/12938/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/12938/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@208
PS4, Line 208:
metrics_.getCounter(MetastoreEventsProcessor.EVENTS_FILTERED_METRIC)
> Just having this line is sufficient to expose the metric in /events page. Y
Done
http://gerrit.cloudera.org:8080/#/c/12938/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@898
PS4, Line 898: vent's DB object, it means that the Database object present
> The process method does not invalidate so this description needs to be upda
Done
http://gerrit.cloudera.org:8080/#/c/12938/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@899
PS4, Line 899: * in the catalog is a later version and we can skip the
event.
> line too long (100 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/12938/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@903
PS4, Line 903: verride
: public void process() {
: Reference dbFound = new Reference<>();
: Reference dbMatched = new Reference<>();
: Db removedDb =
catalog_.removeDbIfExists(droppedDatabase_, dbFound, dbMatched);
: if (removedDb != null) {
: infoLog("Removed Database {} ", dbName_);
: } else if (!dbMatched.getRef()) {
: LOG.warn(debugString("Database %s was not removed
from catalog since "
: + "the creation time of the Database did not
match", dbName_));
:
metrics_.getCounter(MetastoreEventsProcessor.EVENTS_FILTERED_METRIC).inc();
: } else if (!dbFound.getRef()) {
: debugLog("Database {} was not removed since it did
not exist in catalog
> This code has potential race conditions. For example, the createTime of the
Refactored this method similar to DropTable. Now atomically checking
CREATE_TIME and removing DB
http://gerrit.cloudera.org:8080/#/c/12938/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/12938/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@272
PS4, Line 272:* DROP_DATABASE uses CREATION_TIME to filter events that try
to drop an
> add javadoc to describe what exactly this test is testing. Specifically, de
Done
http://gerrit.cloudera.org:8080/#/c/12938/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@284
PS4, Line 284:
> add javadoc
Done
http://gerrit.cloudera.org:8080/#/c/12938/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@285
PS4, Line 285: // Check that the database exists in Catalog
> line too long (126 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/12938/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@293
PS4, Line 293:*/
> line too long (91 > 90)
Done
[Impala-ASF-CR] IMPALA-8338 : Check CREATION TIME in event processor to avoid incorrect deletion of Database objects.
Bharath Krishna has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/12938 ) Change subject: IMPALA-8338 : Check CREATION_TIME in event processor to avoid incorrect deletion of Database objects. .. IMPALA-8338 : Check CREATION_TIME in event processor to avoid incorrect deletion of Database objects. Process the drop database event only if the CREATION_TIME of the catalog's database object is lesser than or equal to that of the database object present in the notification event. If the CREATION_TIME in the notification event object is lesser than the catalog's DB object, it means that the Database object present in the catalog is the latest and we can skip the event instead. Testing : - Added unit tests in MetastoreEventsProcessorTest. - Enabled testCreateDropCreateDatabaseFromImpala as we now have CREATION_TIME in the notification events. Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 4 files changed, 128 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/12938/7 -- To view, visit http://gerrit.cloudera.org:8080/12938 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c Gerrit-Change-Number: 12938 Gerrit-PatchSet: 7 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8338 : Check CREATION TIME of databases in event processor to avoid incorrect/redundant invalidates
Bharath Krishna has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12938 Change subject: IMPALA-8338 : Check CREATION_TIME of databases in event processor to avoid incorrect/redundant invalidates .. IMPALA-8338 : Check CREATION_TIME of databases in event processor to avoid incorrect/redundant invalidates Process the drop database event only if the CREATION_TIME of the catalog's database object is lesser than or equal to that of the database object present in the notification event. If the CREATION_TIME in the notification event object is lesser than the catalog's DB object, it means that the Database object present in the catalog is the latest and we do not need to invalidate it in this case. Testing : - Added unit tests in MetastoreEventsProcessorTest. - Enabled testCreateDropCreateDatabaseFromImpala as we now have CREATION_TIME in the notification events. Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c --- M be/src/util/event-metrics.cc M be/src/util/event-metrics.h M common/thrift/JniCatalog.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 8 files changed, 136 insertions(+), 26 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/12938/4 -- To view, visit http://gerrit.cloudera.org:8080/12938 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8fd3685cf7261e4953f4f884850489a47c5bbd6c Gerrit-Change-Number: 12938 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Krishna
[Impala-ASF-CR] This change bumps the CDH BUILD VERSION to a version that includes the change to have full-thrift object for DropDatabaseMessage in Metastore notifications. This change is needed for t
Bharath Krishna has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12965 Change subject: This change bumps the CDH_BUILD_VERSION to a version that includes the change to have full-thrift object for DropDatabaseMessage in Metastore notifications. This change is needed for the upcoming patch for IMPALA-8338. .. This change bumps the CDH_BUILD_VERSION to a version that includes the change to have full-thrift object for DropDatabaseMessage in Metastore notifications. This change is needed for the upcoming patch for IMPALA-8338. Change-Id: Id38bae921c4d93421c6c72cdccef6a4783e2588e --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/12965/1 -- To view, visit http://gerrit.cloudera.org:8080/12965 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id38bae921c4d93421c6c72cdccef6a4783e2588e Gerrit-Change-Number: 12965 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Bharath Krishna has posted comments on this change. (
http://gerrit.cloudera.org:8080/12889 )
Change subject: IMPALA-7971: Add support for insert events in event processor.
..
Patch Set 5:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@601
PS5, Line 601: Preconditions.checkNotNull(insertMessage.getTableObj());
These following two lines can be
Preconditions.checkNotNull(insertMessage.getTableObj());
msTbl_ = insertMessage.getTableObj();
written as :
msTbl_ = Preconditions.checkNotNull(insertMessage.getTableObj());
http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@204
PS5, Line 204: // Metric name for number of refreshes by event processor sofar
Typo sofar
http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3551
PS5, Line 3551: // For non-partitioned tables parts below will contain a
single value. The
nit : comma after tables.
http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3766
PS5, Line 3766: LOG.debug("Firing insert event... for %s", tbl.getName());
LOG.debug("Firing insert event for {}", tbl.getName());
http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3780
PS5, Line 3780: LOG.error("Failed to fire insert event. This may cause
some table refreshes to be"
I think it's better to have:
Some Tables might not be refreshed on other Impala clusters for this event.
--
To view, visit http://gerrit.cloudera.org:8080/12889
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52
Gerrit-Change-Number: 12889
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Krishna
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Tue, 02 Apr 2019 21:55:38 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12889/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/12889/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3778 PS2, Line 3778: rqst.setPartitionVals(partVals); > Should this condition be reversed, because it will be false by default? Done -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 01 Apr 2019 20:46:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 3: (5 comments) http://gerrit.cloudera.org:8080/#/c/12889/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/12889/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@597 PS2, Line 597: msTbl_ = insertMessage.getTableObj(); We can use Preconditions.checkNotNull(insertMessage.getTableObj()) http://gerrit.cloudera.org:8080/#/c/12889/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@606 PS2, Line 606: // Currently we do not check for self-events in Inserts. This means insert events I think we should add this as a javadoc comment for the process method. http://gerrit.cloudera.org:8080/#/c/12889/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/12889/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3716 PS2, Line 3716: } nit : typo in variable name existigPartNames http://gerrit.cloudera.org:8080/#/c/12889/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3770 PS2, Line 3770: FireEventRequestData data = new FireEventRequestData(); We can add Table name to the debug log. http://gerrit.cloudera.org:8080/#/c/12889/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3778 PS2, Line 3778: rqst.setPartitionVals(partVals); Should this condition be reversed, because it will be false by default? if (isOverwrite) insertData.setReplace(true); -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 29 Mar 2019 06:00:15 + Gerrit-HasComments: Yes
