[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16613765#comment-16613765
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12913741/SENTRY-2109-rebase-SENTRY-2106.014.patch
against master.
{color:red}Overall:{color} -1 due to an error
{color:red}ERROR:{color} failed to apply patch (exit code 1):
error:
a/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/HMSFollower.java:
does not exist in index
error:
a/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/SentryStore.java:
does not exist in index
error:
a/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HiveNotificationFetcher.java:
does not exist in index
error:
a/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryService.java:
does not exist in index
error:
a/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/ServiceConstants.java:
does not exist in index
error:
a/sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestHMSFollower.java:
does not exist in index
error:
a/sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestHMSFollowerSentryStoreIntegration.java:
does not exist in index
error:
a/sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/service/thrift/TestHiveNotificationFetcher.java:
does not exist in index
error:
a/sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationBase.java:
does not exist in index
error:
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/HMSFollower.java:
does not exist in index
error:
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/SentryStore.java:
does not exist in index
error:
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HiveNotificationFetcher.java:
does not exist in index
error:
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryService.java:
does not exist in index
error:
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/ServiceConstants.java:
does not exist in index
error:
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestHMSFollower.java:
does not exist in index
error:
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestHMSFollowerSentryStoreIntegration.java:
does not exist in index
error:
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/service/thrift/TestHiveNotificationFetcher.java:
does not exist in index
error: patch failed:
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationBase.java:874
Falling back to three-way merge...
Applied patch to
'sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationBase.java'
with conflicts.
error:
sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/HMSFollower.java:
does not exist in index
error:
sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/SentryStore.java:
does not exist in index
error:
sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HiveNotificationFetcher.java:
does not exist in index
error:
sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryService.java:
does not exist in index
error:
sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/ServiceConstants.java:
does not exist in index
error:
sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestHMSFollower.java:
does not exist in index
error:
sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestHMSFollowerSentryStoreIntegration.java:
does not exist in index
error:
sentry-provider-db/src/test/java/org/apache/sentry/service/thrift/TestHiveNotificationFetcher.java:
does not exist in index
error:
sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationBase.java:
does not exist in index
The patch does not appear to apply with p0, p1, or p2
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4103/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392803#comment-16392803 ] kalyan kumar kalvagadda commented on SENTRY-2109: - [~LinaAtAustin] Scenario you mentioned in your comment is already handled by the patch. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109-rebase-SENTRY-2106.014.patch, > SENTRY-2109.001.patch, SENTRY-2109.002.patch, SENTRY-2109.003.patch, > SENTRY-2109.004.patch, SENTRY-2109.005.patch, SENTRY-2109.006.patch, > SENTRY-2109.007.patch, SENTRY-2109.008.patch, SENTRY-2109.009.patch, > SENTRY-2109.010.patch, SENTRY-2109.010.patch, SENTRY-2109.011.patch, > SENTRY-2109.012.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.013.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392789#comment-16392789
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12913741/SENTRY-2109-rebase-SENTRY-2106.014.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3692/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-2109-rebase-SENTRY-2106.014.patch,
> SENTRY-2109.001.patch, SENTRY-2109.002.patch, SENTRY-2109.003.patch,
> SENTRY-2109.004.patch, SENTRY-2109.005.patch, SENTRY-2109.006.patch,
> SENTRY-2109.007.patch, SENTRY-2109.008.patch, SENTRY-2109.009.patch,
> SENTRY-2109.010.patch, SENTRY-2109.010.patch, SENTRY-2109.011.patch,
> SENTRY-2109.012.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch,
> SENTRY-2109.013.patch, Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392469#comment-16392469
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12913714/SENTRY-2109.013.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3691/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch,
> SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch,
> SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch,
> SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch,
> SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch,
> SENTRY-2109.012.patch, SENTRY-2109.013.patch,
> Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391976#comment-16391976
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12913660/SENTRY-2109.013.patch
against master.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.persistent.TestHMSFollower
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3690/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch,
> SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch,
> SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch,
> SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch,
> SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch,
> SENTRY-2109.012.patch, SENTRY-2109.013.patch,
> Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351014#comment-16351014 ] kalyan kumar kalvagadda commented on SENTRY-2109: - I have reverted the patch. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350906#comment-16350906 ] Alexander Kolbasov commented on SENTRY-2109: [~kkalyan] I would appreciate if you revert the patch and we complete the discussion review and then you commit (or not commit) whatever comes out of it. This patch had many iterations. There were several reviewers who had comments about earlier versions of the patch and some reviewers posted unanswered comments in this Jira so it is pretty clear that the review wasn't complete even though you got one +1 from a committer. I think that the best course of action is to revert the patch and complete the review. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350862#comment-16350862 ] kalyan kumar kalvagadda commented on SENTRY-2109: - [~akolb] This is what the patch does functionally. # Make sure that additional snapshots are not taken because of GAPS in the event-id's # Make sure that notification are not missed even when there are GAPS and out-of-sequence notifications. we know that issue of gaps and out-of-sequence in the events will be solved with Hive fix done for HIVE-18526 . Only justification to have this patch was to safe guard sentry and be able to handle cases where events are not received in proper order. It's reasonable not assuming that other components would behave correctly and have some functionality in place to handle the error scenarios. I will revert the patch if you say so. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350490#comment-16350490 ] Sergio Peña commented on SENTRY-2109: - [~vspec...@gmail.com] Thanks for the confirmation about keeping it on hold, but the patch was committed yesterday and there were still some concerns about it. Should we revert this patch to finish the tests and reviews with the HMS fix and see if that fixes the current problems? <- [~kkalyan] > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349562#comment-16349562 ] Alexander Kolbasov commented on SENTRY-2109: I asked the question multiple times in this JIRA and didn't get an answer - what problem does this fix address once the issue is fixed on HMS side? That's the basic question to answer before we dive into the implementation details. On Thu, Feb 1, 2018 at 4:09 PM, kalyan kumar kalvagadda (JIRA) < > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349549#comment-16349549 ] kalyan kumar kalvagadda commented on SENTRY-2109: - [~akolb], I have already re-opened the review. If you have comments, I will revert the change. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349522#comment-16349522 ] kalyan kumar kalvagadda commented on SENTRY-2109: - [~spena] We decided to hold this commit in CDH till 5.15 where we will have good testing coverage. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349520#comment-16349520 ] Alexander Kolbasov commented on SENTRY-2109: [~kkalyan] This is complicated patch that took 12 iterations to do - most of the patches didn't compile so it wasn't clear that there is a final patch available for review. Many previous patches didn't compile so I wasn't aware that you are ready for code review - and now it is committed. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349521#comment-16349521 ] Vadim Spector commented on SENTRY-2109: --- I think it's useful to have as many reviews as possible, so if [~akolb] wants to review it may be worth to reopen it. [~spena], it is correct, for now this logic is going to be on hold, so there is no rush in committing it. Since it's almost complete (unless the remaining reviewers find something wrong), it can be a legitimate plan B, if HMS fix is found to have any serious issues. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349519#comment-16349519 ] kalyan kumar kalvagadda commented on SENTRY-2109: - [~akolb] I did not know that you were reviewing the code change. I will re-open it for review. If you have comments I revert the change and re-submit it addressing your comments. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349508#comment-16349508 ] Sergio Peña commented on SENTRY-2109: - [~kkalyan] Isn't this logic going to be on hold now that we know it will be fixed on the HMS side? > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349502#comment-16349502 ] Alexander Kolbasov commented on SENTRY-2109: Why is it resolved so quickly? I was reviewing this as well and had some earlier comments, you didn't give me a chance to complete review. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347245#comment-16347245
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12908585/SENTRY-2109.012.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3651/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch,
> SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch,
> SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch,
> SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch,
> SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch,
> SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346157#comment-16346157
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12908461/SENTRY-2109.012.patch
against master.
{color:red}Overall:{color} -1 due to an error
{color:red}ERROR:{color} mvn test exited 1
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3650/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch,
> SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch,
> SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch,
> SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch,
> SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch,
> Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344074#comment-16344074
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12908183/SENTRY-2109.011.patch
against master.
{color:red}Overall:{color} -1 due to an error
{color:red}ERROR:{color} mvn test exited 1
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3647/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch,
> SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch,
> SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch,
> SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch,
> SENTRY-2109.011.patch, Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343670#comment-16343670
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12908154/SENTRY-2109.011.patch
against master.
{color:red}Overall:{color} -1 due to an error
{color:red}ERROR:{color} mvn test exited 1
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3645/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch,
> SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch,
> SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch,
> SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch,
> SENTRY-2109.011.patch, Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16339656#comment-16339656
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12907736/SENTRY-2109.010.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3644/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch,
> SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch,
> SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch,
> SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch,
> Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16339555#comment-16339555
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12907716/SENTRY-2109.010.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3643/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch,
> SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch,
> SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch,
> SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch,
> Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16339528#comment-16339528 ] Alexander Kolbasov commented on SENTRY-2109: [~kkalyan] ignore the title, it takes care of gaps as well. The fix strictly serializes all updates to notifications using row lock on the notification ID row, so it essentially works as a distributed lock. As a result, all artifacts related to concurrent modifications disappear. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16339526#comment-16339526 ] kalyan kumar kalvagadda commented on SENTRY-2109: - [~akolb] I did not look closer into changes done for HIVE-16886 but from the title it just handles duplicate event-id's. We still have issue with gaps in event-id's which this addressed. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16339448#comment-16339448 ] Alexander Kolbasov commented on SENTRY-2109: HIVE-18526 provides a very simple fix on the Hive side. I think it is better to fix the root cause of the problem rather then create a complicated workaround. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16339232#comment-16339232
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12907686/SENTRY-2109.009.patch
against master.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.service.thrift.TestHiveNotificationFetcher
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3642/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch,
> SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch,
> SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch,
> SENTRY-2109.009.patch, Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338147#comment-16338147 ] Alexander Kolbasov commented on SENTRY-2109: Added a link. I am not quite right though - the fix is in Hive 3.0 but not in Hive 2 branch. It should be backported to Hive 2. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338121#comment-16338121 ] kalyan kumar kalvagadda commented on SENTRY-2109: - [~akolb] Could you link the Jira here. Are you saying that sentry can assume that there will not be any duplicates and out-of-order notifications ? > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338109#comment-16338109 ] Alexander Kolbasov commented on SENTRY-2109: Note that the original problem (HMS notifications duplicates and other weirdness) is fixed in Hive upstream. So this is a work-around for a problem that is already fixed on the HMS side. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338103#comment-16338103 ] Alexander Kolbasov commented on SENTRY-2109: Can't build the latest patch. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16334630#comment-16334630
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12907158/SENTRY-2109.008.patch
against master.
{color:red}Overall:{color} -1 due to an error
{color:red}ERROR:{color} failed to build with patch (exit code 1)
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3636/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch,
> SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch,
> SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch,
> Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16334606#comment-16334606
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12907155/SENTRY-2109.008.patch
against master.
{color:red}Overall:{color} -1 due to an error
{color:red}ERROR:{color} failed to apply patch (exit code 1):
The patch does not appear to apply with p0, p1, or p2
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3635/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch,
> SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch,
> SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch,
> Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329881#comment-16329881
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12906511/SENTRY-2109.007.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3624/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch,
> SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch,
> SENTRY-2109.006.patch, SENTRY-2109.007.patch,
> Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329739#comment-16329739
]
Hadoop QA commented on SENTRY-2109:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12906493/SENTRY-2109.006.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3623/console
This message is automatically generated.
> Fix the logic of identifying HMS out of Sync and handle gaps and
> out-of-sequence notifications.
> ---
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch,
> SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch,
> SENTRY-2109.006.patch, SENTRY-2109.007.patch,
> Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
