[Impala-ASF-CR] IMPALA-9257: Last event id should be advanced if all events are skipped
Vihang Karajgaonkar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14916 ) Change subject: IMPALA-9257: Last event id should be advanced if all events are skipped .. IMPALA-9257: Last event id should be advanced if all events are skipped Events processor implements a filtering method which skips certain unnecessary events (eg. events on blacklisted dbs and tables). However, if the received batch has all the events which are filtered out, it fails to update its lastSyncedEventId. This causes unnecessary logs being printed in catalog and the same event batch being fetched repeatedly. Testing: Modified existing test to compare event id after events on blacklisted dbs and tables. Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 Reviewed-on: http://gerrit.cloudera.org:8080/14916 Reviewed-by: Anurag Mantripragada Reviewed-by: Quanlong Huang Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M tests/custom_cluster/test_event_processing.py 2 files changed, 9 insertions(+), 3 deletions(-) Approvals: Anurag Mantripragada: Looks good to me, but someone else must approve Quanlong Huang: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14916 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 Gerrit-Change-Number: 14916 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9257: Last event id should be advanced if all events are skipped
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14916 ) Change subject: IMPALA-9257: Last event id should be advanced if all events are skipped .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14916 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 Gerrit-Change-Number: 14916 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 18 Dec 2019 04:39:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9257: Last event id should be advanced if all events are skipped
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14916 ) Change subject: IMPALA-9257: Last event id should be advanced if all events are skipped .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5354/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14916 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 Gerrit-Change-Number: 14916 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 18 Dec 2019 00:09:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9257: Last event id should be advanced if all events are skipped
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14916 ) Change subject: IMPALA-9257: Last event id should be advanced if all events are skipped .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14916 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 Gerrit-Change-Number: 14916 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 17 Dec 2019 23:45:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9257: Last event id should be advanced if all events are skipped
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14916 ) Change subject: IMPALA-9257: Last event id should be advanced if all events are skipped .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5301/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14916 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 Gerrit-Change-Number: 14916 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 17 Dec 2019 19:51:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9257: Last event id should be advanced if all events are skipped
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14916 ) Change subject: IMPALA-9257: Last event id should be advanced if all events are skipped .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/14916 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 Gerrit-Change-Number: 14916 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 17 Dec 2019 19:42:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9257: Last event id should be advanced if all events are skipped
Vihang Karajgaonkar has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/14916 ) Change subject: IMPALA-9257: Last event id should be advanced if all events are skipped .. IMPALA-9257: Last event id should be advanced if all events are skipped Events processor implements a filtering method which skips certain unnecessary events (eg. events on blacklisted dbs and tables). However, if the received batch has all the events which are filtered out, it fails to update its lastSyncedEventId. This causes unnecessary logs being printed in catalog and the same event batch being fetched repeatedly. Testing: Modified existing test to compare event id after events on blacklisted dbs and tables. Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M tests/custom_cluster/test_event_processing.py 2 files changed, 9 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/14916/2 -- To view, visit http://gerrit.cloudera.org:8080/14916 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 Gerrit-Change-Number: 14916 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9257: Last event id should be advanced if all events are skipped
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14916 ) Change subject: IMPALA-9257: Last event id should be advanced if all events are skipped .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/14916/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/14916/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@588 PS1, Line 588: @return the last Notification event which was processed. > Remove this? Done -- To view, visit http://gerrit.cloudera.org:8080/14916 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 Gerrit-Change-Number: 14916 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 17 Dec 2019 19:20:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9257: Last event id should be advanced if all events are skipped
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14916 ) Change subject: IMPALA-9257: Last event id should be advanced if all events are skipped .. Patch Set 1: (1 comment) Looks good to me. Will let Quanlong approve it after you address his comments. http://gerrit.cloudera.org:8080/#/c/14916/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/14916/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@588 PS1, Line 588: @return the last Notification event which was processed. Remove this? -- To view, visit http://gerrit.cloudera.org:8080/14916 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 Gerrit-Change-Number: 14916 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 17 Dec 2019 18:52:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9257: Last event id should be advanced if all events are skipped
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14916 ) Change subject: IMPALA-9257: Last event id should be advanced if all events are skipped .. Patch Set 1: (1 comment) Nice find! http://gerrit.cloudera.org:8080/#/c/14916/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/14916/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@593 PS1, Line 593: if (events.isEmpty()) return; Should we still update EVENTS_RECEIVED_METRIC in this case? -- To view, visit http://gerrit.cloudera.org:8080/14916 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 Gerrit-Change-Number: 14916 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 17 Dec 2019 03:45:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9257: Last event id should be advanced if all events are skipped
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14916 ) Change subject: IMPALA-9257: Last event id should be advanced if all events are skipped .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5296/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14916 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 Gerrit-Change-Number: 14916 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 17 Dec 2019 01:28:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9257: Last event id should be advanced if all events are skipped
Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14916 Change subject: IMPALA-9257: Last event id should be advanced if all events are skipped .. IMPALA-9257: Last event id should be advanced if all events are skipped Events processor implements a filtering method which skips certain unnecessary events (eg. events on blacklisted dbs and tables). However, if the received batch has all the events which are filtered out, it fails to update its lastSyncedEventId. This causes unnecessary logs being printed in catalog and the same event batch being fetched repeatedly. Testing: Modified existing test to compare event id after events on blacklisted dbs and tables. Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M tests/custom_cluster/test_event_processing.py 2 files changed, 7 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/14916/1 -- To view, visit http://gerrit.cloudera.org:8080/14916 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32 Gerrit-Change-Number: 14916 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Quanlong Huang
