[
https://issues.apache.org/jira/browse/SENTRY-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liujianhui updated SENTRY-2104:
-------------------------------
Description:
when latestsentry notification id is larger than the current notification id on
HMS, it trigger hmsfollower to pull the full snapshot on hms, unfortunately, it
takes long time. eventually lead notification request timeout
What conditions will lead to the latestSentryNotificationId larger than
currentHmsNotificationId?
@Alexander Kolbasov
{code} private boolean isFullSnapshotRequired(long latestSentryNotificationId)
throws Exception {
if (sentryStore.isHmsNotificationEmpty()) {
return true;
}
long currentHmsNotificationId =
notificationFetcher.getCurrentNotificationId();
if (currentHmsNotificationId < latestSentryNotificationId) {
LOGGER.info("The latest notification ID on HMS is less than the latest
notification ID "
+ "processed by Sentry. Need to request a full HMS snapshot.");
return true;
}
return false;
}{code}
the log of pull full snapshot
{code}
2017-12-19 07:14:14,947 INFO org.apache.sentry.service.thrift.HMSFollower: The
latest notification ID on HMS is less than the latest notification ID processed
by Sentry. Need to request a full HMS snapshot.
2017-12-19 07:14:15,107 INFO org.apache.sentry.service.thrift.SentryHMSClient:
Request full HMS snapshot
2017-12-19 07:30:53,176 INFO org.apache.sentry.service.thrift.SentryHMSClient:
Obtained full HMS snapshot
2017-12-19 07:30:53,197 INFO org.apache.sentry.service.thrift.SentryHMSClient:
NotificationID, Before Snapshot: 301872, After Snapshot 303145
2017-12-19 07:30:53,197 INFO org.apache.sentry.service.thrift.SentryHMSClient:
Reconciling full snapshot - applying 1273 changes
2017-12-19 07:30:53,482 INFO org.apache.sentry.service.thrift.SentryHMSClient:
Successfully fetched hive full snapshot, Current NotificationID: 303145.
2017-12-19 07:30:53,486 INFO org.apache.sentry.service.thrift.HMSFollower:
Sentry HMS support is ready
{code}
was:
when latestsentry notification id is larger than the current notification id on
HMS, it trigger hmsfollower to pull the full snapshot on hms, unfortunately, it
takes long time. eventually lead notification request timeout
What conditions will lead to the latestSentryNotificationId larger than
currentHmsNotificationId?
{code} private boolean isFullSnapshotRequired(long latestSentryNotificationId)
throws Exception {
if (sentryStore.isHmsNotificationEmpty()) {
return true;
}
long currentHmsNotificationId =
notificationFetcher.getCurrentNotificationId();
if (currentHmsNotificationId < latestSentryNotificationId) {
LOGGER.info("The latest notification ID on HMS is less than the latest
notification ID "
+ "processed by Sentry. Need to request a full HMS snapshot.");
return true;
}
return false;
}{code}
the log of pull full snapshot
{code}
2017-12-19 07:14:14,947 INFO org.apache.sentry.service.thrift.HMSFollower: The
latest notification ID on HMS is less than the latest notification ID processed
by Sentry. Need to request a full HMS snapshot.
2017-12-19 07:14:15,107 INFO org.apache.sentry.service.thrift.SentryHMSClient:
Request full HMS snapshot
2017-12-19 07:30:53,176 INFO org.apache.sentry.service.thrift.SentryHMSClient:
Obtained full HMS snapshot
2017-12-19 07:30:53,197 INFO org.apache.sentry.service.thrift.SentryHMSClient:
NotificationID, Before Snapshot: 301872, After Snapshot 303145
2017-12-19 07:30:53,197 INFO org.apache.sentry.service.thrift.SentryHMSClient:
Reconciling full snapshot - applying 1273 changes
2017-12-19 07:30:53,482 INFO org.apache.sentry.service.thrift.SentryHMSClient:
Successfully fetched hive full snapshot, Current NotificationID: 303145.
2017-12-19 07:30:53,486 INFO org.apache.sentry.service.thrift.HMSFollower:
Sentry HMS support is ready
{code}
> it take long time to request full snapshot which lead to timeout
> ----------------------------------------------------------------
>
> Key: SENTRY-2104
> URL: https://issues.apache.org/jira/browse/SENTRY-2104
> Project: Sentry
> Issue Type: Bug
> Affects Versions: 1.7.0
> Reporter: liujianhui
> Priority: Minor
>
> when latestsentry notification id is larger than the current notification id
> on HMS, it trigger hmsfollower to pull the full snapshot on hms,
> unfortunately, it takes long time. eventually lead notification request
> timeout
> What conditions will lead to the latestSentryNotificationId larger than
> currentHmsNotificationId?
> @Alexander Kolbasov
> {code} private boolean isFullSnapshotRequired(long
> latestSentryNotificationId) throws Exception {
> if (sentryStore.isHmsNotificationEmpty()) {
> return true;
> }
> long currentHmsNotificationId =
> notificationFetcher.getCurrentNotificationId();
> if (currentHmsNotificationId < latestSentryNotificationId) {
> LOGGER.info("The latest notification ID on HMS is less than the latest
> notification ID "
> + "processed by Sentry. Need to request a full HMS snapshot.");
> return true;
> }
> return false;
> }{code}
> the log of pull full snapshot
> {code}
> 2017-12-19 07:14:14,947 INFO org.apache.sentry.service.thrift.HMSFollower:
> The latest notification ID on HMS is less than the latest notification ID
> processed by Sentry. Need to request a full HMS snapshot.
> 2017-12-19 07:14:15,107 INFO
> org.apache.sentry.service.thrift.SentryHMSClient: Request full HMS snapshot
> 2017-12-19 07:30:53,176 INFO
> org.apache.sentry.service.thrift.SentryHMSClient: Obtained full HMS snapshot
> 2017-12-19 07:30:53,197 INFO
> org.apache.sentry.service.thrift.SentryHMSClient: NotificationID, Before
> Snapshot: 301872, After Snapshot 303145
> 2017-12-19 07:30:53,197 INFO
> org.apache.sentry.service.thrift.SentryHMSClient: Reconciling full snapshot -
> applying 1273 changes
> 2017-12-19 07:30:53,482 INFO
> org.apache.sentry.service.thrift.SentryHMSClient: Successfully fetched hive
> full snapshot, Current NotificationID: 303145.
> 2017-12-19 07:30:53,486 INFO org.apache.sentry.service.thrift.HMSFollower:
> Sentry HMS support is ready
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
