-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64955/
-----------------------------------------------------------
(Updated March 13, 2018, 11:26 p.m.)
Review request for sentry, Alexander Kolbasov, Arjun Mishra, Na Li, Sergio
Pena, and Vadim Spector.
Changes
-------
rebased the patch
Bugs: SENTRY-2109
https://issues.apache.org/jira/browse/SENTRY-2109
Repository: sentry
Description
-------
This patch does couple of things
1. Avoid triggering full snapshots when gaps are observed while fetching new
notifications. While fetching new notifications HMSFollower would would fetch
notifications with last event-id as well. When it gets the notifications and if
it doesn't get the notifications with event-id, full snpshot is triggered.
2. Functinalty to address gaps and out-of-sequence notificaitons by re-fetching
addtional notifications that were already fetched. This solution is not fool
proof. It does a best effort to reduse the chance of loosing notifications by
re-fetching the notifications.This approach will introduce an over head of
fetching addtional notifications that were already fetched. Overhead of DB
look-up is addressed by using a cache. This reduces additional DB lookups
needed to check if the notification was already processed.
2. Added looging to report duplicate events and out of order events for debug
purpose.
3. Added new e2e tests to verfy this behavior.
Diffs (updated)
-----
sentry-core/sentry-core-common/src/main/java/org/apache/sentry/core/common/exception/SentryOutOfSyncException.java
PRE-CREATION
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/HMSFollower.java
929e6be
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/SentryStore.java
4521ad4
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HiveNotificationFetcher.java
93cc34f
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryService.java
bf5d85b
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/ServiceConstants.java
7e02874
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestHMSFollower.java
61e3f06
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestHMSFollowerSentryStoreIntegration.java
91c90f9
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/service/thrift/TestHiveNotificationFetcher.java
83a1bec
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/service/thrift/TestHiveNotificationFetcherCache.java
PRE-CREATION
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestSnapshotCreation.java
PRE-CREATION
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestSnapshotCreationWithShorterHMSEventTtl.java
PRE-CREATION
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestSnapshotWithLongerHMSFollowerLongerInterval.java
PRE-CREATION
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationBase.java
9b0aeb2
Diff: https://reviews.apache.org/r/64955/diff/9/
Changes: https://reviews.apache.org/r/64955/diff/8-9/
Testing
-------
Made sure that tests pass.
Thanks,
kalyan kumar kalvagadda
