[jira] [Updated] (SENTRY-1904) TransactionManager should limit the max time spent by transaction retry
[ https://issues.apache.org/jira/browse/SENTRY-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-1904: Attachment: (was: SENTRY-1904.001.patch) > TransactionManager should limit the max time spent by transaction retry > --- > > Key: SENTRY-1904 > URL: https://issues.apache.org/jira/browse/SENTRY-1904 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-1904.001.patch > > > The TransactionManager uses exponential backoff strategy for transaction > retries. This may cause some transactions to be delayed by a very long time. > We should also have a constraint on the max time for a transaction so that we > do not retry for too long. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2126) Transactional retry not needed for JDO exceptions on SENTRY_HMS_NOTIFICATION_ID
[ https://issues.apache.org/jira/browse/SENTRY-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2126: Status: Patch Available (was: In Progress) > Transactional retry not needed for JDO exceptions on > SENTRY_HMS_NOTIFICATION_ID > --- > > Key: SENTRY-2126 > URL: https://issues.apache.org/jira/browse/SENTRY-2126 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2126.001.patch > > > Transactional reties not needed for failures while persisting > MSentryHmsNotification which are because of duplicate key. > > > There are only inserts that happen on this table. If an insert failed because > of duplicate entry, it will likely fail in the subsequent retires. There is > no need to retry. > > Specially in HA mode when multiple sentry servers can act as leader (split > Brian situation) this will cause additional load on the database for no > reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2126) Transactional retry not needed for JDO exceptions on SENTRY_HMS_NOTIFICATION_ID
[ https://issues.apache.org/jira/browse/SENTRY-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2126: Attachment: SENTRY-2126.001.patch > Transactional retry not needed for JDO exceptions on > SENTRY_HMS_NOTIFICATION_ID > --- > > Key: SENTRY-2126 > URL: https://issues.apache.org/jira/browse/SENTRY-2126 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2126.001.patch > > > Transactional reties not needed for failures while persisting > MSentryHmsNotification which are because of duplicate key. > > > There are only inserts that happen on this table. If an insert failed because > of duplicate entry, it will likely fail in the subsequent retires. There is > no need to retry. > > Specially in HA mode when multiple sentry servers can act as leader (split > Brian situation) this will cause additional load on the database for no > reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2126) Transactional retry not needed for JDO exceptions on SENTRY_HMS_NOTIFICATION_ID
[ https://issues.apache.org/jira/browse/SENTRY-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2126: Description: Transactional reties not needed for failures while persisting MSentryHmsNotification which are because of duplicate key. There are only inserts that happen on this table. If an insert failed because of duplicate entry, it will likely fill in the subsequent retires. There is no need to retry. Specially in HA mode when multiple sentry servers can act as leader (split Brian situation) this will cause additional load on the database for no reason. was: Transactional reties not needed for failures while persisting MSentryHmsNotification which are because of duplicate key. There are only inserts that happen on this table. If an insert failed because of duplicate entry, it will likely fill in the subsequent retires. There is not no need to retry. Specially in HA mode when multiple sentry servers can act as leader (split Brian situation) this will cause additional load on the database for no reason. > Transactional retry not needed for JDO exceptions on > SENTRY_HMS_NOTIFICATION_ID > --- > > Key: SENTRY-2126 > URL: https://issues.apache.org/jira/browse/SENTRY-2126 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > > Transactional reties not needed for failures while persisting > MSentryHmsNotification which are because of duplicate key. > > > There are only inserts that happen on this table. If an insert failed because > of duplicate entry, it will likely fill in the subsequent retires. There is > no need to retry. > > Specially in HA mode when multiple sentry servers can act as leader (split > Brian situation) this will cause additional load on the database for no > reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2126) Transactional retry not needed for JDO exceptions on SENTRY_HMS_NOTIFICATION_ID
[ https://issues.apache.org/jira/browse/SENTRY-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2126: Description: Transactional reties not needed for failures while persisting MSentryHmsNotification which are because of duplicate key. There are only inserts that happen on this table. If an insert failed because of duplicate entry, it will likely fail in the subsequent retires. There is no need to retry. Specially in HA mode when multiple sentry servers can act as leader (split Brian situation) this will cause additional load on the database for no reason. was: Transactional reties not needed for failures while persisting MSentryHmsNotification which are because of duplicate key. There are only inserts that happen on this table. If an insert failed because of duplicate entry, it will likely fill in the subsequent retires. There is no need to retry. Specially in HA mode when multiple sentry servers can act as leader (split Brian situation) this will cause additional load on the database for no reason. > Transactional retry not needed for JDO exceptions on > SENTRY_HMS_NOTIFICATION_ID > --- > > Key: SENTRY-2126 > URL: https://issues.apache.org/jira/browse/SENTRY-2126 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > > Transactional reties not needed for failures while persisting > MSentryHmsNotification which are because of duplicate key. > > > There are only inserts that happen on this table. If an insert failed because > of duplicate entry, it will likely fail in the subsequent retires. There is > no need to retry. > > Specially in HA mode when multiple sentry servers can act as leader (split > Brian situation) this will cause additional load on the database for no > reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-1904) TransactionManager should limit the max time spent by transaction retry
[ https://issues.apache.org/jira/browse/SENTRY-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-1904: Status: Patch Available (was: In Progress) > TransactionManager should limit the max time spent by transaction retry > --- > > Key: SENTRY-1904 > URL: https://issues.apache.org/jira/browse/SENTRY-1904 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-1904.001.patch > > > The TransactionManager uses exponential backoff strategy for transaction > retries. This may cause some transactions to be delayed by a very long time. > We should also have a constraint on the max time for a transaction so that we > do not retry for too long. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-1904) TransactionManager should limit the max time spent by transaction retry
[ https://issues.apache.org/jira/browse/SENTRY-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-1904: Attachment: SENTRY-1904.001.patch > TransactionManager should limit the max time spent by transaction retry > --- > > Key: SENTRY-1904 > URL: https://issues.apache.org/jira/browse/SENTRY-1904 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-1904.001.patch > > > The TransactionManager uses exponential backoff strategy for transaction > retries. This may cause some transactions to be delayed by a very long time. > We should also have a constraint on the max time for a transaction so that we > do not retry for too long. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-1904) TransactionManager should limit the max time spent by transaction retry
[ https://issues.apache.org/jira/browse/SENTRY-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda reassigned SENTRY-1904: --- Assignee: kalyan kumar kalvagadda > TransactionManager should limit the max time spent by transaction retry > --- > > Key: SENTRY-1904 > URL: https://issues.apache.org/jira/browse/SENTRY-1904 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda >Priority: Major > > The TransactionManager uses exponential backoff strategy for transaction > retries. This may cause some transactions to be delayed by a very long time. > We should also have a constraint on the max time for a transaction so that we > do not retry for too long. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2126) Transactional retry not needed for JDO exceptions on SENTRY_HMS_NOTIFICATION_ID
[ https://issues.apache.org/jira/browse/SENTRY-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332938#comment-16332938 ] kalyan kumar kalvagadda commented on SENTRY-2126: - FYI [~spena], [~vspec...@gmail.com], [~arjunmishra13] and [~lina.li] > Transactional retry not needed for JDO exceptions on > SENTRY_HMS_NOTIFICATION_ID > --- > > Key: SENTRY-2126 > URL: https://issues.apache.org/jira/browse/SENTRY-2126 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > > Transactional reties not needed for failures while persisting > MSentryHmsNotification which are because of duplicate key. > > > There are only inserts that happen on this table. If an insert failed because > of duplicate entry, it will likely fill in the subsequent retires. There is > not no need to retry. > > Specially in HA mode when multiple sentry servers can act as leader (split > Brian situation) this will cause additional load on the database for no > reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SENTRY-2126) Transactional retry not needed for JDO exceptions on SENTRY_HMS_NOTIFICATION_ID
kalyan kumar kalvagadda created SENTRY-2126: --- Summary: Transactional retry not needed for JDO exceptions on SENTRY_HMS_NOTIFICATION_ID Key: SENTRY-2126 URL: https://issues.apache.org/jira/browse/SENTRY-2126 Project: Sentry Issue Type: Bug Components: Sentry Affects Versions: 2.1.0 Reporter: kalyan kumar kalvagadda Assignee: kalyan kumar kalvagadda Transactional reties not needed for failures while persisting MSentryHmsNotification which are because of duplicate key. There are only inserts that happen on this table. If an insert failed because of duplicate entry, it will likely fill in the subsequent retires. There is not no need to retry. Specially in HA mode when multiple sentry servers can act as leader (split Brian situation) this will cause additional load on the database for no reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2109: Attachment: SENTRY-2109.007.patch > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2109: Summary: Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications. (was: Fix the logic of identifying HMS out of Sync) > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2034) Add e2e tests for testing HMS notification processing.
[ https://issues.apache.org/jira/browse/SENTRY-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2034: Resolution: Fixed Status: Resolved (was: Patch Available) > Add e2e tests for testing HMS notification processing. > -- > > Key: SENTRY-2034 > URL: https://issues.apache.org/jira/browse/SENTRY-2034 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2034.001.patch, SENTRY-2034.002.patch, > SENTRY-2034.002.patch, SENTRY-2034.002.patch, SENTRY-2034.005.patch, > SENTRY-2034.007.patch > > > Currently, there are no e2e tests that test the functionality of pulling the > notifications from HMS and processing them. Which include updating the update > the permissions based on the HMS updates. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2034) Add e2e tests for testing HMS notification processing.
[ https://issues.apache.org/jira/browse/SENTRY-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327785#comment-16327785 ] kalyan kumar kalvagadda commented on SENTRY-2034: - [~lina.li] and [~spena] Thanks for the review. > Add e2e tests for testing HMS notification processing. > -- > > Key: SENTRY-2034 > URL: https://issues.apache.org/jira/browse/SENTRY-2034 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2034.001.patch, SENTRY-2034.002.patch, > SENTRY-2034.002.patch, SENTRY-2034.002.patch, SENTRY-2034.005.patch, > SENTRY-2034.007.patch > > > Currently, there are no e2e tests that test the functionality of pulling the > notifications from HMS and processing them. Which include updating the update > the permissions based on the HMS updates. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2115) Sentry will be out of sync with HMS on disabling and enabling HDFS synchronization.
[ https://issues.apache.org/jira/browse/SENTRY-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324549#comment-16324549 ] kalyan kumar kalvagadda commented on SENTRY-2115: - I prefer option-1 listed above. I will be submitting a patch for the same. > Sentry will be out of sync with HMS on disabling and enabling HDFS > synchronization. > --- > > Key: SENTRY-2115 > URL: https://issues.apache.org/jira/browse/SENTRY-2115 > Project: Sentry > Issue Type: Bug >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > On disabling HDFS sync, HMSFollower would update the > SENTRY_HMS_NOTIFICATION_ID table but not the MSentryPathChange table. > When HDFS sync is enabled again, HMSFollower would continue fetching new > notifications and process them and update the MSentryPathChange and > MAuthzPathsMapping info. This is not correct. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2034) Add e2e tests for testing HMS notification processing.
[ https://issues.apache.org/jira/browse/SENTRY-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2034: Attachment: SENTRY-2034.007.patch > Add e2e tests for testing HMS notification processing. > -- > > Key: SENTRY-2034 > URL: https://issues.apache.org/jira/browse/SENTRY-2034 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2034.001.patch, SENTRY-2034.002.patch, > SENTRY-2034.002.patch, SENTRY-2034.002.patch, SENTRY-2034.005.patch, > SENTRY-2034.007.patch > > > Currently, there are no e2e tests that test the functionality of pulling the > notifications from HMS and processing them. Which include updating the update > the permissions based on the HMS updates. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2115) Sentry will be out of sync with HMS on disabling and enabling HDFS synchronization.
[ https://issues.apache.org/jira/browse/SENTRY-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322610#comment-16322610 ] kalyan kumar kalvagadda commented on SENTRY-2115: - There are couple of ways to solve this 1. By clearing MAuthzPathsMapping and MAuthzPathsSnapshotId information when HDFS-SYNC is disabled. 2. Continue to update MAuthzPathsMapping and MAuthzPathsSnapshotId even when HDFS-SYNC is disabled. That way when the feature is enabled sentry-namenode plug-in gets the latest snapshot. Downside for this approach is that, sentry would be spending CPU cycles and memory processing and storing the patch changes in MSentryPathChange and MAuthzPathsMapping. > Sentry will be out of sync with HMS on disabling and enabling HDFS > synchronization. > --- > > Key: SENTRY-2115 > URL: https://issues.apache.org/jira/browse/SENTRY-2115 > Project: Sentry > Issue Type: Bug >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > On disabling HDFS sync, HMSFollower would update the > SENTRY_HMS_NOTIFICATION_ID table but not the MSentryPathChange table. > When HDFS sync is enabled again, HMSFollower would continue fetching new > notifications and process them and update the MSentryPathChange and > MAuthzPathsMapping info. This is not correct. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2121) Notifications processed during times when HDFS sync is disabled will not be applied as ACLs when later HDFS sync was to be enabled
[ https://issues.apache.org/jira/browse/SENTRY-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322597#comment-16322597 ] kalyan kumar kalvagadda commented on SENTRY-2121: - [~lina.li] and [~arjunmishra13] This is a duplicate of SENTRY-2115. > Notifications processed during times when HDFS sync is disabled will not be > applied as ACLs when later HDFS sync was to be enabled > -- > > Key: SENTRY-2121 > URL: https://issues.apache.org/jira/browse/SENTRY-2121 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra > Labels: triaged > > When HDFS sync is disabled, we don't update the AUTHZ_PATHS_MAPPING table. > However, when HDFS sync is enabled, ACLs that are generated are based on > entries in the AUTHZ_PATHS_MAPPING table. So this means that if we were to > disable HDFS sync and later enable HDFS sync, ACLs for all notifications that > were processed during the times when HDFS sycn was disabled won't be created -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2115) Sentry will be out of sync with HMS on disabling and enabling HDFS synchronization.
[ https://issues.apache.org/jira/browse/SENTRY-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2115: Description: On disabling HDFS sync, HMSFollower would update the SENTRY_HMS_NOTIFICATION_ID table but not the MSentryPathChange table. When HDFS sync is enabled again, HMSFollower would continue fetching new notifications and process them and update the MSentryPathChange and MAuthzPathsMapping info. This is not correct. was: On disabling HDFS sync, HMSFollower would update the SENTRY_HMS_NOTIFICATION_ID table but not the MSentryPathChange table. When HDFS sync is enabled again, notification information in SENTRY_HMS_NOTIFICATION_ID ahead of information in MSentryPathChange. If HMSFollower follows the current logic it will notifications that are fetched when the feature was turned off. > Sentry will be out of sync with HMS on disabling and enabling HDFS > synchronization. > --- > > Key: SENTRY-2115 > URL: https://issues.apache.org/jira/browse/SENTRY-2115 > Project: Sentry > Issue Type: Bug >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > On disabling HDFS sync, HMSFollower would update the > SENTRY_HMS_NOTIFICATION_ID table but not the MSentryPathChange table. > When HDFS sync is enabled again, HMSFollower would continue fetching new > notifications and process them and update the MSentryPathChange and > MAuthzPathsMapping info. This is not correct. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2119) HMSFollower may not fetch HMS notifications which are out of order
[ https://issues.apache.org/jira/browse/SENTRY-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322407#comment-16322407 ] kalyan kumar kalvagadda commented on SENTRY-2119: - [~lina.li] Notification processor process notifications in a sequence. If there is event-id is missed it is simple to identify. Can you put your question with an example to be clear? > HMSFollower may not fetch HMS notifications which are out of order > -- > > Key: SENTRY-2119 > URL: https://issues.apache.org/jira/browse/SENTRY-2119 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > With the current implementation of HMS and HMS-HA notifications inserted in > NOTIFICATION_LOG table can be out of order. That means, notifications with > smaller event-id's can be inserted into the table later and there is not > clear understanding on on the time difference between them. > When this happens HMSFollower will not be able to fetch these notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2119) HMSFollower may not fetch HMS notifications which are out of order
[ https://issues.apache.org/jira/browse/SENTRY-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318710#comment-16318710 ] kalyan kumar kalvagadda commented on SENTRY-2119: - [~spena][~arjunmishra13][~lina.li] FYI > HMSFollower may not fetch HMS notifications which are out of order > -- > > Key: SENTRY-2119 > URL: https://issues.apache.org/jira/browse/SENTRY-2119 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > With the current implementation of HMS and HMS-HA notifications inserted in > NOTIFICATION_LOG table can be out of order. That means, notifications with > smaller event-id's can be inserted into the table later and there is not > clear understanding on on the time difference between them. > When this happens HMSFollower will not be able to fetch these notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (SENTRY-2119) HMSFollower may not fetch HMS notifications which are out of order
[ https://issues.apache.org/jira/browse/SENTRY-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317338#comment-16317338 ] kalyan kumar kalvagadda edited comment on SENTRY-2119 at 1/8/18 11:34 PM: -- *Approach-2:* As HMSFollower process the notifications it would know if there are any notifications that are missed. HMSFollower needs to record/persist them in a new table. # Based on configured interval, HMSFollower should schedule a retry of missed notifications. # When HMSFollower is able to fetch it on this try, it should clean-up the entry from the persisted data. # when the missed event-ids are persisted HMSFollower needs to persist the current time stamp as well. # we need to have a time interval to purge this data. # When the event-id are purged we unblock any HMS threads if needed. was (Author: kkalyan): *Approach-2:* As HMSFollower process the notifications it would know if there are any notifications that are missed. HMSFollower needs to record/persist them . # Based on configured interval, HMSFollower should schedule a retry of missed notifications. # When HMSFollower is able to fetch it on this try, it should clean-up the entry from the persisted data. # when the missed event-ids are persisted HMSFollower needs to persist the current time stamp as well. # we need to have a time interval to purge this data. # When the event-id are purged we unblock any HMS threads if needed. > HMSFollower may not fetch HMS notifications which are out of order > -- > > Key: SENTRY-2119 > URL: https://issues.apache.org/jira/browse/SENTRY-2119 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > With the current implementation of HMS and HMS-HA notifications inserted in > NOTIFICATION_LOG table can be out of order. That means, notifications with > smaller event-id's can be inserted into the table later and there is not > clear understanding on on the time difference between them. > When this happens HMSFollower will not be able to fetch these notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (SENTRY-2119) HMSFollower may not fetch HMS notifications which are out of order
[ https://issues.apache.org/jira/browse/SENTRY-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317338#comment-16317338 ] kalyan kumar kalvagadda edited comment on SENTRY-2119 at 1/8/18 11:34 PM: -- *Approach-2:* As HMSFollower process the notifications it would know if there are any notifications that are missed. HMSFollower needs to record/persist them . # Based on configured interval, HMSFollower should schedule a retry of missed notifications. # When HMSFollower is able to fetch it on this try, it should clean-up the entry from the persisted data. # when the missed event-ids are persisted HMSFollower needs to persist the current time stamp as well. # we need to have a time interval to purge this data. # When the event-id are purged we unblock any HMS threads if needed. was (Author: kkalyan): *Approach-2:* As HMSFollower process the notifications it would know if there are any notifications that are missed. HMSFollower needs to record/persist them . # Based on configured interval HMSFollower should schedule a retry of missed notifications. # When HMSFollower is able to fetch it on this try, it should clean-up the entry from the persisted data. # when the missed event-ids are persisted HMSFollower needs to persist the current time stamp as well. # we need to have a time interval to purge this data. # When the event-id are purged we unblock any HMS threads if needed. > HMSFollower may not fetch HMS notifications which are out of order > -- > > Key: SENTRY-2119 > URL: https://issues.apache.org/jira/browse/SENTRY-2119 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > With the current implementation of HMS and HMS-HA notifications inserted in > NOTIFICATION_LOG table can be out of order. That means, notifications with > smaller event-id's can be inserted into the table later and there is not > clear understanding on on the time difference between them. > When this happens HMSFollower will not be able to fetch these notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (SENTRY-2119) HMSFollower may not fetch HMS notifications which are out of order
[ https://issues.apache.org/jira/browse/SENTRY-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317338#comment-16317338 ] kalyan kumar kalvagadda edited comment on SENTRY-2119 at 1/8/18 11:33 PM: -- *Approach-2:* As HMSFollower process the notifications it would know if there are any notifications that are missed. HMSFollower needs to record/persist them . # Based on configured interval HMSFollower should schedule a retry of missed notifications. # When HMSFollower is able to fetch it on this try, it should clean-up the entry from the persisted data. # # when the missed event-ids are persisted HMSFollower needs to persist the current time stamp as well. # we need to have a time interval to purge this data. # When the event-id are purged we unblock any HMS threads if needed. was (Author: kkalyan): *Approach-2:* As we process the notifications we HMSFollower would know if there are any notifications that are missed. We need to record/persist them . # We need to have configured interval to schedule a retry of missed notifications. # when the missed event-ids are persisted we need to persist the current time stamp as well # we need to have a time interval to purge this data. # When the event-id are purged we unblock any HMS threads if needed. > HMSFollower may not fetch HMS notifications which are out of order > -- > > Key: SENTRY-2119 > URL: https://issues.apache.org/jira/browse/SENTRY-2119 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > With the current implementation of HMS and HMS-HA notifications inserted in > NOTIFICATION_LOG table can be out of order. That means, notifications with > smaller event-id's can be inserted into the table later and there is not > clear understanding on on the time difference between them. > When this happens HMSFollower will not be able to fetch these notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (SENTRY-2119) HMSFollower may not fetch HMS notifications which are out of order
[ https://issues.apache.org/jira/browse/SENTRY-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317338#comment-16317338 ] kalyan kumar kalvagadda edited comment on SENTRY-2119 at 1/8/18 11:33 PM: -- *Approach-2:* As HMSFollower process the notifications it would know if there are any notifications that are missed. HMSFollower needs to record/persist them . # Based on configured interval HMSFollower should schedule a retry of missed notifications. # When HMSFollower is able to fetch it on this try, it should clean-up the entry from the persisted data. # when the missed event-ids are persisted HMSFollower needs to persist the current time stamp as well. # we need to have a time interval to purge this data. # When the event-id are purged we unblock any HMS threads if needed. was (Author: kkalyan): *Approach-2:* As HMSFollower process the notifications it would know if there are any notifications that are missed. HMSFollower needs to record/persist them . # Based on configured interval HMSFollower should schedule a retry of missed notifications. # When HMSFollower is able to fetch it on this try, it should clean-up the entry from the persisted data. # # when the missed event-ids are persisted HMSFollower needs to persist the current time stamp as well. # we need to have a time interval to purge this data. # When the event-id are purged we unblock any HMS threads if needed. > HMSFollower may not fetch HMS notifications which are out of order > -- > > Key: SENTRY-2119 > URL: https://issues.apache.org/jira/browse/SENTRY-2119 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > With the current implementation of HMS and HMS-HA notifications inserted in > NOTIFICATION_LOG table can be out of order. That means, notifications with > smaller event-id's can be inserted into the table later and there is not > clear understanding on on the time difference between them. > When this happens HMSFollower will not be able to fetch these notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (SENTRY-2119) HMSFollower may not fetch HMS notifications which are out of order
[ https://issues.apache.org/jira/browse/SENTRY-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317254#comment-16317254 ] kalyan kumar kalvagadda edited comment on SENTRY-2119 at 1/8/18 11:30 PM: -- *Approach-1:* When ever HMSFollower fetches notifications from HMS it can go back by configurable number of event-id's and fetch the notifications. That way, if there are any notifications that were missed in the previous fetch requests they can be fetched and processed later. This solution is not fool proof. Going back "N" event-id's and fetching older notifications may not solve the issue in all the cases but it can minimize the chase of it. was (Author: kkalyan): *Solution Approach:* When ever HMSFollower fetches notifications from HMS it can go back by configurable number of event-id's and fetch the notifications. That way, if there are any notifications that were missed in the previous fetch requests they can be fetched and processed later. This solution is not fool proof. Going back "N" event-id's and fetching older notifications may not solve the issue in all the cases but it can minimize the chase of it. > HMSFollower may not fetch HMS notifications which are out of order > -- > > Key: SENTRY-2119 > URL: https://issues.apache.org/jira/browse/SENTRY-2119 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > With the current implementation of HMS and HMS-HA notifications inserted in > NOTIFICATION_LOG table can be out of order. That means, notifications with > smaller event-id's can be inserted into the table later and there is not > clear understanding on on the time difference between them. > When this happens HMSFollower will not be able to fetch these notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2119) HMSFollower may not fetch HMS notifications which are out of order
[ https://issues.apache.org/jira/browse/SENTRY-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317338#comment-16317338 ] kalyan kumar kalvagadda commented on SENTRY-2119: - *Approach-2:* As we process the notifications we HMSFollower would know if there are any notifications that are missed. We need to record/persist them . # We need to have configured interval to schedule a retry of missed notifications. # when the missed event-ids are persisted we need to persist the current time stamp as well # we need to have a time interval to purge this data. # When the event-id are purged we unblock any HMS threads if needed. > HMSFollower may not fetch HMS notifications which are out of order > -- > > Key: SENTRY-2119 > URL: https://issues.apache.org/jira/browse/SENTRY-2119 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > With the current implementation of HMS and HMS-HA notifications inserted in > NOTIFICATION_LOG table can be out of order. That means, notifications with > smaller event-id's can be inserted into the table later and there is not > clear understanding on on the time difference between them. > When this happens HMSFollower will not be able to fetch these notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2119) HMSFollower may not fetch HMS notifications which are out of order
[ https://issues.apache.org/jira/browse/SENTRY-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317254#comment-16317254 ] kalyan kumar kalvagadda commented on SENTRY-2119: - *Solution Approach:* When ever HMSFollower fetches notifications from HMS it can go back by configurable number of event-id's and fetch the notifications. That way, if there are any notifications that were missed in the previous fetch requests they can be fetched and processed later. This solution is not fool proof. Going back "N" event-id's and fetching older notifications may not solve the issue in all the cases but it can minimize the chase of it. > HMSFollower may not fetch HMS notifications which are out of order > -- > > Key: SENTRY-2119 > URL: https://issues.apache.org/jira/browse/SENTRY-2119 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > With the current implementation of HMS and HMS-HA notifications inserted in > NOTIFICATION_LOG table can be out of order. That means, notifications with > smaller event-id's can be inserted into the table later and there is not > clear understanding on on the time difference between them. > When this happens HMSFollower will not be able to fetch these notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2119) HMSFollower may not fetch HMS notifications which are out of order
[ https://issues.apache.org/jira/browse/SENTRY-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317255#comment-16317255 ] kalyan kumar kalvagadda commented on SENTRY-2119: - [~vspec...@gmail.com] FYI > HMSFollower may not fetch HMS notifications which are out of order > -- > > Key: SENTRY-2119 > URL: https://issues.apache.org/jira/browse/SENTRY-2119 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > With the current implementation of HMS and HMS-HA notifications inserted in > NOTIFICATION_LOG table can be out of order. That means, notifications with > smaller event-id's can be inserted into the table later and there is not > clear understanding on on the time difference between them. > When this happens HMSFollower will not be able to fetch these notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2119) HMSFollower may not fetch HMS notifications which are out of order
kalyan kumar kalvagadda created SENTRY-2119: --- Summary: HMSFollower may not fetch HMS notifications which are out of order Key: SENTRY-2119 URL: https://issues.apache.org/jira/browse/SENTRY-2119 Project: Sentry Issue Type: Bug Components: Sentry Affects Versions: 2.1.0 Reporter: kalyan kumar kalvagadda Assignee: kalyan kumar kalvagadda With the current implementation of HMS and HMS-HA notifications inserted in NOTIFICATION_LOG table can be out of order. That means, notifications with smaller event-id's can be inserted into the table later and there is not clear understanding on on the time difference between them. When this happens HMSFollower will not be able to fetch these notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2117) event-id in notification log are wrongly called as notification id
kalyan kumar kalvagadda created SENTRY-2117: --- Summary: event-id in notification log are wrongly called as notification id Key: SENTRY-2117 URL: https://issues.apache.org/jira/browse/SENTRY-2117 Project: Sentry Issue Type: Bug Components: Sentry Affects Versions: 2.1.0 Reporter: kalyan kumar kalvagadda Assignee: kalyan kumar kalvagadda Fix For: 2.1.0 event-id in notification events is wrongly called as notification id. This is confusing. Code/Documentation should be updated accordingly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312048#comment-16312048 ] kalyan kumar kalvagadda commented on SENTRY-2109: - We need code change done for SENTRY-2113 to fix these test failures. > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311974#comment-16311974 ] kalyan kumar kalvagadda commented on SENTRY-2109: - [~lina.li] we can increase that number now we know that gaps are possible in event-id's > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2109: Attachment: SENTRY-2109.004.patch > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311782#comment-16311782 ] kalyan kumar kalvagadda commented on SENTRY-2109: - [~lina.li] Purging is not an issue as we don't purge all the notifications. we keep latest ~100 entries while purging the table. This patch addresses the scenario where Snapshot is triggered when there are gaps in between. Yes, there could be some notifications that could be missed. This will not be addressed with the patch. > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2113) MSentryHmsNotification should also hold the notification hash
[ https://issues.apache.org/jira/browse/SENTRY-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311777#comment-16311777 ] kalyan kumar kalvagadda commented on SENTRY-2113: - [~spena] As event-id in the notification is not unique, Sentry uses notification hash of the notification to verify it a notification is processed. Currently, this has is stored only in MSentryPathChange which is not designed to store all the notifications fetched from HMS. This jira is created to have notification has in MSentryHmsNotification, we it records every notification that is fetched from HMS. > MSentryHmsNotification should also hold the notification hash > -- > > Key: SENTRY-2113 > URL: https://issues.apache.org/jira/browse/SENTRY-2113 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Attachments: SENTRY-2113.001.patch, SENTRY-2113.002.patch, > SENTRY-2113.003.patch > > > Currently MSentryHmsNotification is holding notification-id but it should be > extended to hold the notification hash. > We are currently using MSentryPathChange to get the notification hash but > this information is not reliable as MSentryPathChange will store all the > notifications the HMSFollower gets from HMS. MSentryPathChange will only > store the notifications that needed to sync HDFS ACL's. > Unlike MSentryPathChange, MSentryHmsNotification is designed to record the > information of all the notifications that HMSFollower receives from HMS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2113) MSentryHmsNotification should also hold the notification hash
[ https://issues.apache.org/jira/browse/SENTRY-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2113: Attachment: SENTRY-2113.003.patch > MSentryHmsNotification should also hold the notification hash > -- > > Key: SENTRY-2113 > URL: https://issues.apache.org/jira/browse/SENTRY-2113 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Attachments: SENTRY-2113.001.patch, SENTRY-2113.002.patch, > SENTRY-2113.003.patch > > > Currently MSentryHmsNotification is holding notification-id but it should be > extended to hold the notification hash. > We are currently using MSentryPathChange to get the notification hash but > this information is not reliable as MSentryPathChange will store all the > notifications the HMSFollower gets from HMS. MSentryPathChange will only > store the notifications that needed to sync HDFS ACL's. > Unlike MSentryPathChange, MSentryHmsNotification is designed to record the > information of all the notifications that HMSFollower receives from HMS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2114) Fix behavior of HMSFollower when HDFS sync is not enabled
[ https://issues.apache.org/jira/browse/SENTRY-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311661#comment-16311661 ] kalyan kumar kalvagadda commented on SENTRY-2114: - [~lina.li],[~spena] Even when HDFS Synchronization is disabled, sentry need to do permission update based on the notifications fetched. so HMSFollower need to update the NOTIFICATION_ID table to be able to fetch new notifications on every run. > Fix behavior of HMSFollower when HDFS sync is not enabled > - > > Key: SENTRY-2114 > URL: https://issues.apache.org/jira/browse/SENTRY-2114 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > Currently behavior of HMSFollower when HDFS sync disabled is as below. > # Take full snapshot of HMS > # Take the Notification Id from the snapshot created. > # Persist the notification Id to SENTRY_HMS_NOTIFICATION_ID table > # Fetches notifications and process new notifications after that point. > *This doesn't make sense. Ideally, I think it should work as below.* > # Initially, fetch all the notifications with even-id > 0 and process them. > # Subsequently fetch and process new notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2113) MSentryHmsNotification should also hold the notification hash
[ https://issues.apache.org/jira/browse/SENTRY-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2113: Description: Currently MSentryHmsNotification is holding notification-id but it should be extended to hold the notification hash. We are currently using MSentryPathChange to get the notification hash but this information is not reliable as MSentryPathChange will store all the notifications the HMSFollower gets from HMS. MSentryPathChange will only store the notifications that needed to sync HDFS ACL's. Unlike MSentryPathChange, MSentryHmsNotification is designed to record the information of all the notifications that HMSFollower receives from HMS. was: Currently MSentryHmsNotification is holding notification-id but it should be extended to hold the notification hash. We are currently using MSentryPathChange to get the notification hash but this information is not reliable as MSentryPathChange will store all the notifications the HMSFollower gets from HMS. MSentryPathChange will only store the notifications that needed to sync HDFS ACL's. Unlike MSentryPathChange, MSentryHmsNotification is designed to record the information of all the notifications theat HMSFollower receives from HMS. > MSentryHmsNotification should also hold the notification hash > -- > > Key: SENTRY-2113 > URL: https://issues.apache.org/jira/browse/SENTRY-2113 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Attachments: SENTRY-2113.001.patch, SENTRY-2113.002.patch > > > Currently MSentryHmsNotification is holding notification-id but it should be > extended to hold the notification hash. > We are currently using MSentryPathChange to get the notification hash but > this information is not reliable as MSentryPathChange will store all the > notifications the HMSFollower gets from HMS. MSentryPathChange will only > store the notifications that needed to sync HDFS ACL's. > Unlike MSentryPathChange, MSentryHmsNotification is designed to record the > information of all the notifications that HMSFollower receives from HMS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (SENTRY-2114) Fix behavior of HMSFollower when HDFS sync is not enabled
[ https://issues.apache.org/jira/browse/SENTRY-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310384#comment-16310384 ] kalyan kumar kalvagadda edited comment on SENTRY-2114 at 1/4/18 2:33 PM: - [~lina.li] Even the current code doesn't handle this scenario properly. I was planning to create another jira to fix the functionality when 1. HDFS is disabled 2. HDFS is enabled after some time. I have created SENTRY-2115 for the same. was (Author: kkalyan): [~lina.li] Even the current code doesn't handle this scenario properly. I was planning to create another jira to fix the functionality when 1. HDFS is disabled 2. HDFS is enabled after some time. > Fix behavior of HMSFollower when HDFS sync is not enabled > - > > Key: SENTRY-2114 > URL: https://issues.apache.org/jira/browse/SENTRY-2114 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > Currently behavior of HMSFollower when HDFS sync disabled is as below. > # Take full snapshot of HMS > # Take the Notification Id from the snapshot created. > # Persist the notification Id to SENTRY_HMS_NOTIFICATION_ID table > # Fetches notifications and process new notifications after that point. > *This doesn't make sense. Ideally, I think it should work as below.* > # Initially, fetch all the notifications with even-id > 0 and process them. > # Subsequently fetch and process new notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2085) Sentry error handling exposes SentryGroupNotFoundException externally
[ https://issues.apache.org/jira/browse/SENTRY-2085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2085: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) > Sentry error handling exposes SentryGroupNotFoundException externally > - > > Key: SENTRY-2085 > URL: https://issues.apache.org/jira/browse/SENTRY-2085 > Project: Sentry > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Zach Amsden >Assignee: Zach Amsden > Fix For: 2.1.0 > > Attachments: SENTRY-2085.001.patch, SENTRY-2085.002.patch, > SENTRY-2085.003.patch, SENTRY-2085.004.patch, SENTRY-2085.006.patch > > > The fix for SENTRY-769 allowed better differentiation of Sentry errors but > also unnecessarily leaked exceptions outside of well defined API components. > This happens because the exception chosen, SentryGroupNotFoundException was > created of a subclass of RuntimeError, allowing it to propagate past API > boundaries without 'throws' or 'catch' clauses. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2085) Sentry error handling exposes SentryGroupNotFoundException externally
[ https://issues.apache.org/jira/browse/SENTRY-2085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310434#comment-16310434 ] kalyan kumar kalvagadda commented on SENTRY-2085: - [~zamsden_impala_ad21] Thanks for the contribution. > Sentry error handling exposes SentryGroupNotFoundException externally > - > > Key: SENTRY-2085 > URL: https://issues.apache.org/jira/browse/SENTRY-2085 > Project: Sentry > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Zach Amsden >Assignee: Zach Amsden > Attachments: SENTRY-2085.001.patch, SENTRY-2085.002.patch, > SENTRY-2085.003.patch, SENTRY-2085.004.patch, SENTRY-2085.006.patch > > > The fix for SENTRY-769 allowed better differentiation of Sentry errors but > also unnecessarily leaked exceptions outside of well defined API components. > This happens because the exception chosen, SentryGroupNotFoundException was > created of a subclass of RuntimeError, allowing it to propagate past API > boundaries without 'throws' or 'catch' clauses. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2115) Sentry will be out of sync with HMS on disabling and enabling HDFS synchronization.
kalyan kumar kalvagadda created SENTRY-2115: --- Summary: Sentry will be out of sync with HMS on disabling and enabling HDFS synchronization. Key: SENTRY-2115 URL: https://issues.apache.org/jira/browse/SENTRY-2115 Project: Sentry Issue Type: Bug Reporter: kalyan kumar kalvagadda Assignee: kalyan kumar kalvagadda On disabling HDFS sync, HMSFollower would update the SENTRY_HMS_NOTIFICATION_ID table but not the MSentryPathChange table. When HDFS sync is enabled again, notification information in SENTRY_HMS_NOTIFICATION_ID ahead of information in MSentryPathChange. If HMSFollower follows the current logic it will notifications that are fetched when the feature was turned off. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2114) Fix behavior of HMSFollower when HDFS sync is not enabled
[ https://issues.apache.org/jira/browse/SENTRY-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310384#comment-16310384 ] kalyan kumar kalvagadda commented on SENTRY-2114: - [~lina.li] Even the current code doesn't handle this scenario properly. I was planning to create another jira to fix the functionality when 1. HDFS is disabled 2. HDFS is enabled after some time. > Fix behavior of HMSFollower when HDFS sync is not enabled > - > > Key: SENTRY-2114 > URL: https://issues.apache.org/jira/browse/SENTRY-2114 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > Currently behavior of HMSFollower when HDFS sync disabled is as below. > # Take full snapshot of HMS > # Take the Notification Id from the snapshot created. > # Persist the notification Id to SENTRY_HMS_NOTIFICATION_ID table > # Fetches notifications and process new notifications after that point. > *This doesn't make sense. Ideally, I think it should work as below.* > # Initially, fetch all the notifications with even-id > 0 and process them. > # Subsequently fetch and process new notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2113) MSentryHmsNotification should also hold the notification hash
[ https://issues.apache.org/jira/browse/SENTRY-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2113: Attachment: SENTRY-2113.002.patch > MSentryHmsNotification should also hold the notification hash > -- > > Key: SENTRY-2113 > URL: https://issues.apache.org/jira/browse/SENTRY-2113 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Attachments: SENTRY-2113.001.patch, SENTRY-2113.002.patch > > > Currently MSentryHmsNotification is holding notification-id but it should be > extended to hold the notification hash. > We are currently using MSentryPathChange to get the notification hash but > this information is not reliable as MSentryPathChange will store all the > notifications the HMSFollower gets from HMS. MSentryPathChange will only > store the notifications that needed to sync HDFS ACL's. > Unlike MSentryPathChange, MSentryHmsNotification is designed to record the > information of all the notifications theat HMSFollower receives from HMS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2114) Fix behavior of HMSFollower when HDFS sync is not enabled
kalyan kumar kalvagadda created SENTRY-2114: --- Summary: Fix behavior of HMSFollower when HDFS sync is not enabled Key: SENTRY-2114 URL: https://issues.apache.org/jira/browse/SENTRY-2114 Project: Sentry Issue Type: Bug Components: Sentry Affects Versions: 2.1.0 Reporter: kalyan kumar kalvagadda Assignee: kalyan kumar kalvagadda Currently behavior of HMSFollower when HDFS sync disabled is as below. # Take full snapshot of HMS # Take the Notification Id from the snapshot created. # Persist the notification Id to SENTRY_HMS_NOTIFICATION_ID table # Fetches notifications and process new notifications after that point. *This doesn't make sense. Ideally, I think it should work as below.* # Initially, fetch all the notifications with even-id > 0 and process them. # Subsequently fetch and process new notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2113) MSentryHmsNotification should also hold the notification hash
[ https://issues.apache.org/jira/browse/SENTRY-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2113: Attachment: SENTRY-2113.001.patch > MSentryHmsNotification should also hold the notification hash > -- > > Key: SENTRY-2113 > URL: https://issues.apache.org/jira/browse/SENTRY-2113 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Attachments: SENTRY-2113.001.patch > > > Currently MSentryHmsNotification is holding notification-id but it should be > extended to hold the notification hash. > We are currently using MSentryPathChange to get the notification hash but > this information is not reliable as MSentryPathChange will store all the > notifications the HMSFollower gets from HMS. MSentryPathChange will only > store the notifications that needed to sync HDFS ACL's. > Unlike MSentryPathChange, MSentryHmsNotification is designed to record the > information of all the notifications theat HMSFollower receives from HMS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2113) MSentryHmsNotification should also hold the notification hash
[ https://issues.apache.org/jira/browse/SENTRY-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2113: Status: Patch Available (was: In Progress) > MSentryHmsNotification should also hold the notification hash > -- > > Key: SENTRY-2113 > URL: https://issues.apache.org/jira/browse/SENTRY-2113 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Attachments: SENTRY-2113.001.patch > > > Currently MSentryHmsNotification is holding notification-id but it should be > extended to hold the notification hash. > We are currently using MSentryPathChange to get the notification hash but > this information is not reliable as MSentryPathChange will store all the > notifications the HMSFollower gets from HMS. MSentryPathChange will only > store the notifications that needed to sync HDFS ACL's. > Unlike MSentryPathChange, MSentryHmsNotification is designed to record the > information of all the notifications theat HMSFollower receives from HMS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2085) Sentry error handling exposes SentryGroupNotFoundException externally
[ https://issues.apache.org/jira/browse/SENTRY-2085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2085: Attachment: SENTRY-2085.006.patch > Sentry error handling exposes SentryGroupNotFoundException externally > - > > Key: SENTRY-2085 > URL: https://issues.apache.org/jira/browse/SENTRY-2085 > Project: Sentry > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Zach Amsden >Assignee: Zach Amsden > Attachments: SENTRY-2085.001.patch, SENTRY-2085.002.patch, > SENTRY-2085.003.patch, SENTRY-2085.004.patch, SENTRY-2085.006.patch > > > The fix for SENTRY-769 allowed better differentiation of Sentry errors but > also unnecessarily leaked exceptions outside of well defined API components. > This happens because the exception chosen, SentryGroupNotFoundException was > created of a subclass of RuntimeError, allowing it to propagate past API > boundaries without 'throws' or 'catch' clauses. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2108) createFullSnapshot so frequently
[ https://issues.apache.org/jira/browse/SENTRY-2108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308487#comment-16308487 ] kalyan kumar kalvagadda commented on SENTRY-2108: - [~HuanWang] This is definitely as bug. Have you seen that in upstream version of sentry as well? If the issue is observed in sentry_1.5.2-cdh-5.13.0, this may not be right place to report the issue. > createFullSnapshot so frequently > - > > Key: SENTRY-2108 > URL: https://issues.apache.org/jira/browse/SENTRY-2108 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 1.8.0 >Reporter: HuanWang > > we are using sentry_1.5.2-cdh-5.13.0 consistent with sentry 1.8.0 in prod. > we found *HMSFollower* createFullSnapshot so frequently, here are some > timestamps I recorded > 2017-12-21 18:13:08 > 2017-12-21 18:15:44 > 2017-12-21 19:51:46 > 2017-12-21 19:54:12 > 2017-12-21 20:04:12 > 2017-12-21 20:06:39 > 2017-12-21 20:16:30 > 2017-12-21 20:18:58 > 2017-12-21 20:28:59 > 2017-12-21 20:31:25 > 2017-12-21 20:41:16 > 2017-12-21 20:43:43 > 2017-12-21 20:53:39 > 2017-12-21 20:56:07 > 2017-12-21 21:06:08 > 2017-12-21 21:08:37 > 2017-12-21 21:18:34 > 2017-12-21 21:21:01 > 2017-12-21 21:30:52 > 2017-12-21 21:33:20 > 2017-12-21 21:43:06 > 2017-12-21 21:45:31 > 2017-12-21 21:55:19 > 2017-12-21 21:57:43 > 2017-12-22 02:50:45 > 2017-12-22 02:53:19 > 2017-12-22 05:39:21 > 2017-12-22 05:41:55 > 2017-12-22 06:40:11 > 2017-12-22 06:42:48 > 2017-12-22 06:52:56 > 2017-12-22 06:55:27 > 2017-12-22 07:05:41 > 2017-12-22 07:05:47 Snapshot created failed > 2017-12-22 07:05:47 > 2017-12-22 07:08:27 > 2017-12-22 07:19:28 > 2017-12-22 07:21:03 Snapshot created failed > 2017-12-22 07:51:25 > 2017-12-22 07:53:53 > 2017-12-22 09:05:21 > 2017-12-22 09:07:55 > 2017-12-22 09:36:12 > 2017-12-22 09:38:53 > 2017-12-22 09:49:44 > 2017-12-22 09:52:17 > 2017-12-22 10:02:29 > 2017-12-22 10:05:05 > 2017-12-22 10:29:22 > 2017-12-22 10:30:13 > 2017-12-22 10:30:13 > 2017-12-22 10:32:47 > 2017-12-22 10:45:43 > 2017-12-22 10:46:08 Snapshot created failed > I confused with it，could somebody can explain it， or it could be a bug？ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2113) MSentryHmsNotification should also hold the notification hash
kalyan kumar kalvagadda created SENTRY-2113: --- Summary: MSentryHmsNotification should also hold the notification hash Key: SENTRY-2113 URL: https://issues.apache.org/jira/browse/SENTRY-2113 Project: Sentry Issue Type: Bug Components: Sentry Affects Versions: 2.1.0 Reporter: kalyan kumar kalvagadda Assignee: kalyan kumar kalvagadda Currently MSentryHmsNotification is holding notification-id but it should be extended to hold the notification hash. We are currently using MSentryPathChange to get the notification hash but this information is not reliable as MSentryPathChange will store all the notifications the HMSFollower gets from HMS. MSentryPathChange will only store the notifications that needed to sync HDFS ACL's. Unlike MSentryPathChange, MSentryHmsNotification is designed to record the information of all the notifications theat HMSFollower receives from HMS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2109: Attachment: SENTRY-2109.003.patch > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2109: Attachment: SENTRY-2109.003.patch > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2109: Attachment: SENTRY-2109.002.patch > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305618#comment-16305618 ] kalyan kumar kalvagadda edited comment on SENTRY-2109 at 12/28/17 6:08 PM: --- *Here is the assumption that HMSFollower is currently taking.* First event in list of notifications received from HMS greater than latest sentry HMS notification Id. *This is not true* Event_ID in the NOTIFICATION_LOG table are not always ascending. Order of sequence is not guaranteed. There can be gaps in between Event_ID's in the consecutive entries. [~vspector] I agree with you. Thus patch should be looked together with the patch for SENTRY-2106. I have created this patch to perform some tests locally using the patches for SENTRY-2106 and SENTRY-2109. *What does being Sentry being out of sync with HMS mean?* Sentry can not be up-to-date with HMS just by fetching new notifications and processing them. Sentry has to start from scratch and create a full snapshot. This patch does the following Note: latestSentryNotificationId is the MAX of all the event-id's that sentry processed. # While requesting updates HMSFollower would request notification from (latestSentryNotificationId -1) # Normally when Sentry and HMS are in Sync, filter provided should also process the notification with latestSentryNotificationId. # When Sentry and HMS are out of Sync where HMS will not have the notification event that was processed by sentry which is the notification event with event-id latestSentryNotificationId. This indicates that there are notifications missing. Note: This is an extension to current logic of identifying and processing notifications with same event-id's. was (Author: kkalyan): *Here is the assumption that HMSFollower is currently taking.* First event in list of notifications received from HMS greater than latest sentry HMS notification Id. *This is not true* Event_ID in the NOTIFICATION_LOG table are not always ascending. Order of sequence is not guaranteed. There can be gaps in between Event_ID's in the consecutive entries. [~vspector] I agree with you. Thus patch should be looked together with the patch for SENTRY-2106. I have created this patch to perform some tests locally using the patches for SENTRY-2106 and SENTRY-2109. *What does being Sentry being out of sync with HMS mean?* Sentry can not be up-to-date with HMS just by fetching new notifications and processing them. Sentry has to start from scratch and create a full snapshot. This patch does the following Note: latestSentryNotificationId is the MAX of all the event-id's that sentry processed. # While requesting updates HMSFollower would request notification from (latestSentryNotificationId -1) # Normally when Sentry and HMS are in Sync, filter provided should also process the notification with latestSentryNotificationId. # When Sentry and HMS are out of Sync where HMS will not have the notification event that was processed by sentry which is the notification event with event-id latestSentryNotificationId. This indicates that there are notifications missing. > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305618#comment-16305618 ] kalyan kumar kalvagadda edited comment on SENTRY-2109 at 12/28/17 6:08 PM: --- *Here is the assumption that HMSFollower is currently taking.* First event in list of notifications received from HMS greater than latest sentry HMS notification Id. *This is not true* Event_ID in the NOTIFICATION_LOG table are not always ascending. Order of sequence is not guaranteed. There can be gaps in between Event_ID's in the consecutive entries. [~vspector] I agree with you. Thus patch should be looked together with the patch for SENTRY-2106. I have created this patch to perform some tests locally using the patches for SENTRY-2106 and SENTRY-2109. *What does being Sentry being out of sync with HMS mean?* Sentry can not be up-to-date with HMS just by fetching new notifications and processing them. Sentry has to start from scratch and create a full snapshot. This patch does the following Note: latestSentryNotificationId is the MAX of all the event-id's that sentry processed. # While requesting updates HMSFollower would request notification from (latestSentryNotificationId -1) # Normally when Sentry and HMS are in Sync, filter provided should also process the notification with latestSentryNotificationId. # When Sentry and HMS are out of Sync where HMS will not have the notification event that was processed by sentry which is the notification event with event-id latestSentryNotificationId. This indicates that there are notifications missing. *Note:* This is an extension to current logic of identifying and processing notifications with same event-id's. was (Author: kkalyan): *Here is the assumption that HMSFollower is currently taking.* First event in list of notifications received from HMS greater than latest sentry HMS notification Id. *This is not true* Event_ID in the NOTIFICATION_LOG table are not always ascending. Order of sequence is not guaranteed. There can be gaps in between Event_ID's in the consecutive entries. [~vspector] I agree with you. Thus patch should be looked together with the patch for SENTRY-2106. I have created this patch to perform some tests locally using the patches for SENTRY-2106 and SENTRY-2109. *What does being Sentry being out of sync with HMS mean?* Sentry can not be up-to-date with HMS just by fetching new notifications and processing them. Sentry has to start from scratch and create a full snapshot. This patch does the following Note: latestSentryNotificationId is the MAX of all the event-id's that sentry processed. # While requesting updates HMSFollower would request notification from (latestSentryNotificationId -1) # Normally when Sentry and HMS are in Sync, filter provided should also process the notification with latestSentryNotificationId. # When Sentry and HMS are out of Sync where HMS will not have the notification event that was processed by sentry which is the notification event with event-id latestSentryNotificationId. This indicates that there are notifications missing. Note: This is an extension to current logic of identifying and processing notifications with same event-id's. > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305618#comment-16305618 ] kalyan kumar kalvagadda commented on SENTRY-2109: - *Here is the assumption that HMSFollower is currently taking.* First event in list of notifications received from HMS greater than latest sentry HMS notification Id. *This is not true* Event_ID in the NOTIFICATION_LOG table are not always ascending. Order of sequence is not guaranteed. There can be gaps in between Event_ID's in the consecutive entries. [~vspector] I agree with you. Thus patch should be looked together with the patch for SENTRY-2106. I have created this patch to perform some tests locally using the patches for SENTRY-2106 and SENTRY-2109. *What does being Sentry being out of sync with HMS mean?* Sentry can not be up-to-date with HMS just by fetching new notifications and processing them. Sentry has to start from scratch and create a full snapshot. This patch does the following Note: latestSentryNotificationId is the MAX of all the event-id's that sentry processed. # While requesting updates HMSFollower would request notification from (latestSentryNotificationId -1) # Normally when Sentry and HMS are in Sync, filter provided should also process the notification with latestSentryNotificationId. # When Sentry and HMS are out of Sync where HMS will not have the notification event that was processed by sentry which is the notification event with event-id latestSentryNotificationId. This indicates that there are notifications missing. > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2106) Sentry ahead full snapshot retry logic
[
https://issues.apache.org/jira/browse/SENTRY-2106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305572#comment-16305572
]
kalyan kumar kalvagadda commented on SENTRY-2106:
-
[~arjunmishra13] I have couple of comments on the approach taken. Before going
into that, I would re-iterate couple of assumptions that HMSFollower has taken
# Method getCurrentNotificationEventId on Hmsclient would always return the
last issued notification event id, which HMSFollower assumes to be the biggest.
Which doesn't seems to be true. We don't have any conclusive answers on this
behavior of HMS.
# First event in list of notifications received from HMS greater than latest
sentry HMS notification Id. This is not true
## Event_ID in the NOTIFICATION_LOG table are not always ascending. Order of
sequence is not guaranteed.
## There can be gaps in between Event_ID's in the consecutive entries.
Here is what I think.
# when getCurrentNotificationEventId is not dependable why should HMSFollower
use it in the first place. Even with the fix that you provided HMSFollower
would wait for 5 sec(Default) to see if getCurrentNotificationEventId returns a
value greater than the last event id that sentry has processed. If not, full
snapshot is initiated. There is no guaranty that it would be greater by that
time. Sentry is not out of sync even if getCurrentNotificationEventId returns
an Id that is smaller than the last event id that sentry has processed. *Do you
agree?*
{color:blue}I prefer removing code in HMSFollower that depends on
getCurrentNotificationEventId to see if it out of sync with HMS.{color}
# Fix for SENTRY-2109 will address the 2nd assumption taken.
> Sentry ahead full snapshot retry logic
> --
>
> Key: SENTRY-2106
> URL: https://issues.apache.org/jira/browse/SENTRY-2106
> Project: Sentry
> Issue Type: New Feature
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Arjun Mishra
>Assignee: Arjun Mishra
> Labels: features
> Fix For: 2.1.0
>
> Attachments: SENTRY-2106.01.patch, SENTRY-2106.02.patch,
> SENTRY-2106.03.patch
>
>
> Once SentryHMSNotification table is not empty, there are two cases when a
> full snapshot is triggered.
> # If first event in list of notifications received from HMS greater than
> latest sentry hms notification Id
> # If latest sentry notification Id is greater than current hms notification Id
> The later is a unexpected case where for some reason sentry gets ahead of
> HMS. We should add a logic to trigger a full snapshot in case 2 only after a
> configurable number of retries. This will avoid unnecessary full snapshot
> triggers as they are resource intensive
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SENTRY-2103) Authorization check for CREATEALIAS command doesn't check permissions for the alias name
[ https://issues.apache.org/jira/browse/SENTRY-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305526#comment-16305526 ] kalyan kumar kalvagadda commented on SENTRY-2103: - [~hgadre] Code changes look good. Please post the changes for review in review board for a formal review. > Authorization check for CREATEALIAS command doesn't check permissions for the > alias name > > > Key: SENTRY-2103 > URL: https://issues.apache.org/jira/browse/SENTRY-2103 > Project: Sentry > Issue Type: Bug > Components: Solr Plugin >Affects Versions: 2.0.0 >Reporter: Hrishikesh Gadre >Assignee: Hrishikesh Gadre > Attachments: SENTRY-2103.1.patch > > > As part of SENTRY-1475, we reimplemented Sentry/Solr integration to base on > Solr authorization framework. The original implementation of Sentry/Solr > integration used to validate the collection level permissions for the alias > name during CREATEALIAS admin operation. During the refactoring done in > SENTRY-1475, we missed that check. This jira is to rectify this mistake. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2109: Status: Patch Available (was: In Progress) > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302476#comment-16302476 ] kalyan kumar kalvagadda edited comment on SENTRY-2109 at 12/23/17 3:01 PM: --- I have attached a patch to fix the issue and also added e2e tests around snapshot creation but there is still some work pending. This new change is causing unit test failures as they are using mocking to test specific class. There is effort involved in fixing those tests. This patch comments them out for now. was (Author: kkalyan): I have attached a patch to fix the issue and also added e2e tests around snapshot creation but there is still some work pending. This new change is causing unit test failures as they are using mocking to test specific class. There is effort involved in fixing those tests. This patch comments them out got now. > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302476#comment-16302476 ] kalyan kumar kalvagadda commented on SENTRY-2109: - I have attached a patch to fix the issue and also added e2e tests around snapshot creation but there is still some work pending. This new change is causing unit test failures as they are using mocking to test specific class. There is effort involved in fixing those tests. This patch comments them out got now. > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2109: Attachment: SENTRY-2109.001.patch > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301807#comment-16301807 ] kalyan kumar kalvagadda commented on SENTRY-2109: - [~vspec...@gmail.com] Sure, I will add logging for that as well. > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301666#comment-16301666 ] kalyan kumar kalvagadda commented on SENTRY-2109: - Working on a patch to avoid the assumption mentioned above. New logic would make sure that the last notification processed by sentry is still in HMS_NOTIFICATION_LOG table to make sure that sentry is still in sync with HMS. > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2109: Attachment: Screenshot_HMS_NOTIFICATION_LOG.png > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301652#comment-16301652
]
kalyan kumar kalvagadda commented on SENTRY-2109:
-
With this below logic HMSFollower excepts that event-id for the first
notification that is fetched should be greater than the last notification
processed by 1. This may not be the case.
{noformat}
public boolean areNotificationsOutOfSync(Collection events,
long latestProcessedId) {
if (events.isEmpty()) {
return false;
}
/*
* If the sequence of notifications has a gap, then an out-of-sync might
* have happened due to the following issue:
*
* - HDFS sync was disabled or Sentry was shutdown for a time period longer
than
* the HMS notification clean-up thread causing old notifications to be
deleted.
*
* HMS notifications may contain both gaps in the sequence and duplicates
* (the same ID repeated more then once for different events).
*
* To accept duplicates (see NotificationFetcher for more info), then a gap
is found
* if the 1st notification received is higher than the current ID processed
+ 1.
* i.e.
* 1st ID = 3, latest ID = 3 (duplicate found but no gap detected)
* 1st ID = 4, latest ID = 3 (consecutive ID found but no gap detected)
* 1st ID = 5, latest ID = 3 (a gap is detected)
*/
List eventList = (List) events;
long firstNotificationId = eventList.get(0).getEventId();
if (firstNotificationId > (latestProcessedId + 1)) {
LOGGER.info("Current HMS notifications are out-of-sync with latest Sentry
processed"
+ "notifications. Need to request a full HMS snapshot.");
return true;
}
return false;
}
{noformat}
I have attached the snapshot for notification_log table for reference.
> Fix the logic of identifying HMS out of Sync
>
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
> Fix For: 2.1.0
>
> Attachments: Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
kalyan kumar kalvagadda created SENTRY-2109: --- Summary: Fix the logic of identifying HMS out of Sync Key: SENTRY-2109 URL: https://issues.apache.org/jira/browse/SENTRY-2109 Project: Sentry Issue Type: Bug Components: Sentry Affects Versions: 2.1.0 Reporter: kalyan kumar kalvagadda Assignee: kalyan kumar kalvagadda Fix For: 2.1.0 Currently HMSFollower proactively checks if sentry is out of sync with HMS and initiates full snapshot, if needed. There will be false positives with the current logic if there are gaps in the event-id in the notification log sequence. This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2105) Add impersonator info to Solr audit logs
[ https://issues.apache.org/jira/browse/SENTRY-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16300146#comment-16300146 ] kalyan kumar kalvagadda commented on SENTRY-2105: - This issue can not be fixed with out SOLR-11781. FYI [~hgadre] > Add impersonator info to Solr audit logs > > > Key: SENTRY-2105 > URL: https://issues.apache.org/jira/browse/SENTRY-2105 > Project: Sentry > Issue Type: Improvement > Components: Solr Plugin >Affects Versions: 2.0.0 >Reporter: Hrishikesh Gadre >Assignee: Hrishikesh Gadre >Priority: Minor > > SENTRY-1475 reimplemented Sentry/Solr integration to comply with Solr > authorization framework (including solr audit logs). The current > implementation of audit log is not filling the impersonator information. > SOLR-11781 provides the support for passing impersonator info via > AuthorizationContext. This jira is implement Sentry side changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2097) Sentry privileges model: Can Sentry take a database privileges away from a server privileges?
[
https://issues.apache.org/jira/browse/SENTRY-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295731#comment-16295731
]
kalyan kumar kalvagadda commented on SENTRY-2097:
-
Jest for reference
https://cwiki.apache.org/confluence/display/SENTRY/Sentry+Privileges
> Sentry privileges model: Can Sentry take a database privileges away from a
> server privileges?
> -
>
> Key: SENTRY-2097
> URL: https://issues.apache.org/jira/browse/SENTRY-2097
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Reporter: Sergio Peña
>Assignee: Na Li
>Priority: Minor
>
> Assume I have a user |jack| and a group |datateam|. The
> user |jack| belongs to group |datateam|.
> Use Sentry for authorization.
> |create role admin; grant role admin to group datateam; grant all on
> server server1 to role admin; |
> Now the role |admin| has the following priveleges.
> {noformat}
> |+---+++-+-+-++---+---+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+---+--+--+
> | * | | | | admin | ROLE | * | false | 1480985013185000 | -- |
> +---+++-+-+-++---+---+--+--+|
> {noformat}
> Assume I have this database.
> |create database testdb; |
> It is successful. User |jack| created a database |testdb|.
> Use Sentry to revoke the privileges on |testdb|;
> |revoke all on database `testdb` from role admin; |
> The priveleges is still the same.
> {noformat}
> |+---+++-+-+-++---+---+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+---+--+--+
> | * | | | | admin | ROLE | * | false | 1480985013185000 | -- |
> +---+++-+-+-++---+---+--+--+|
> {noformat}
> Shouldn't Sentry take the privileges on database |testdb| away from the
> server |server1|?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (SENTRY-2097) Sentry privileges model: Can Sentry take a database privileges away from a server privileges?
[
https://issues.apache.org/jira/browse/SENTRY-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295731#comment-16295731
]
kalyan kumar kalvagadda edited comment on SENTRY-2097 at 12/18/17 9:53 PM:
---
Just for reference
https://cwiki.apache.org/confluence/display/SENTRY/Sentry+Privileges
was (Author: kkalyan):
Jest for reference
https://cwiki.apache.org/confluence/display/SENTRY/Sentry+Privileges
> Sentry privileges model: Can Sentry take a database privileges away from a
> server privileges?
> -
>
> Key: SENTRY-2097
> URL: https://issues.apache.org/jira/browse/SENTRY-2097
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Reporter: Sergio Peña
>Assignee: Na Li
>Priority: Minor
>
> Assume I have a user |jack| and a group |datateam|. The
> user |jack| belongs to group |datateam|.
> Use Sentry for authorization.
> |create role admin; grant role admin to group datateam; grant all on
> server server1 to role admin; |
> Now the role |admin| has the following priveleges.
> {noformat}
> |+---+++-+-+-++---+---+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+---+--+--+
> | * | | | | admin | ROLE | * | false | 1480985013185000 | -- |
> +---+++-+-+-++---+---+--+--+|
> {noformat}
> Assume I have this database.
> |create database testdb; |
> It is successful. User |jack| created a database |testdb|.
> Use Sentry to revoke the privileges on |testdb|;
> |revoke all on database `testdb` from role admin; |
> The priveleges is still the same.
> {noformat}
> |+---+++-+-+-++---+---+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+---+--+--+
> | * | | | | admin | ROLE | * | false | 1480985013185000 | -- |
> +---+++-+-+-++---+---+--+--+|
> {noformat}
> Shouldn't Sentry take the privileges on database |testdb| away from the
> server |server1|?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (SENTRY-2034) Add e2e tests for testing HMS notification processing.
[ https://issues.apache.org/jira/browse/SENTRY-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286336#comment-16286336 ] kalyan kumar kalvagadda edited comment on SENTRY-2034 at 12/11/17 5:47 PM: --- I have re-submitted the patch as the test failure seems no where related to the patch that is submitted. was (Author: kkalyan): I have re-submitted the patch as the fail seems no where related to the patch that is submitted. > Add e2e tests for testing HMS notification processing. > -- > > Key: SENTRY-2034 > URL: https://issues.apache.org/jira/browse/SENTRY-2034 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2034.001.patch, SENTRY-2034.002.patch, > SENTRY-2034.002.patch, SENTRY-2034.002.patch, SENTRY-2034.005.patch > > > Currently, there are no e2e tests that test the functionality of pulling the > notifications from HMS and processing them. Which include updating the update > the permissions based on the HMS updates. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2034) Add e2e tests for testing HMS notification processing.
[ https://issues.apache.org/jira/browse/SENTRY-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2034: Attachment: (was: SENTRY-2034.005.patch) > Add e2e tests for testing HMS notification processing. > -- > > Key: SENTRY-2034 > URL: https://issues.apache.org/jira/browse/SENTRY-2034 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2034.001.patch, SENTRY-2034.002.patch, > SENTRY-2034.002.patch, SENTRY-2034.002.patch, SENTRY-2034.005.patch > > > Currently, there are no e2e tests that test the functionality of pulling the > notifications from HMS and processing them. Which include updating the update > the permissions based on the HMS updates. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2034) Add e2e tests for testing HMS notification processing.
[ https://issues.apache.org/jira/browse/SENTRY-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2034: Attachment: SENTRY-2034.005.patch > Add e2e tests for testing HMS notification processing. > -- > > Key: SENTRY-2034 > URL: https://issues.apache.org/jira/browse/SENTRY-2034 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2034.001.patch, SENTRY-2034.002.patch, > SENTRY-2034.002.patch, SENTRY-2034.002.patch, SENTRY-2034.005.patch > > > Currently, there are no e2e tests that test the functionality of pulling the > notifications from HMS and processing them. Which include updating the update > the permissions based on the HMS updates. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2034) Add e2e tests for testing HMS notification processing.
[ https://issues.apache.org/jira/browse/SENTRY-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286336#comment-16286336 ] kalyan kumar kalvagadda commented on SENTRY-2034: - I have re-submitted the patch as the fail seems no where related to the patch that is submitted. > Add e2e tests for testing HMS notification processing. > -- > > Key: SENTRY-2034 > URL: https://issues.apache.org/jira/browse/SENTRY-2034 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2034.001.patch, SENTRY-2034.002.patch, > SENTRY-2034.002.patch, SENTRY-2034.002.patch, SENTRY-2034.005.patch > > > Currently, there are no e2e tests that test the functionality of pulling the > notifications from HMS and processing them. Which include updating the update > the permissions based on the HMS updates. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (SENTRY-2096) Fail unit tests at end
[
https://issues.apache.org/jira/browse/SENTRY-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285936#comment-16285936
]
kalyan kumar kalvagadda edited comment on SENTRY-2096 at 12/11/17 1:53 PM:
---
[~lina.li]
--fail-at-end is not limited to tests.
{noformat}
Only fail the build afterwards; allow all non-impacted builds to continue
{noformat}
It is better to use an option an option that is limited to tests. Please look
into below option
{noformat}
-Dmaven.test.failure.ignore=true
{noformat}
was (Author: kkalyan):
--fail-at-end is not limited to tests.
{noformat}
Only fail the build afterwards; allow all non-impacted builds to continue
{noformat}
It is better to use an option an option that is limited to tests. Please look
into below option
{noformat}
-Dmaven.test.failure.ignore=true
{noformat}
> Fail unit tests at end
> --
>
> Key: SENTRY-2096
> URL: https://issues.apache.org/jira/browse/SENTRY-2096
> Project: Sentry
> Issue Type: Task
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2096.001.patch
>
>
> Currently, when the unit tests fail in one module, the test stops and report
> failure. It is better to run tests in all modules and then report all failed
> tests. So we can fix test failure at once.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SENTRY-2096) Fail unit tests at end
[
https://issues.apache.org/jira/browse/SENTRY-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285936#comment-16285936
]
kalyan kumar kalvagadda commented on SENTRY-2096:
-
--fail-at-end is not limited to tests.
{noformat}
Only fail the build afterwards; allow all non-impacted builds to continue
{noformat}
It is better to use an option an option that is limited to tests. Please look
into below option
{noformat}
-Dmaven.test.failure.ignore=true
{noformat}
> Fail unit tests at end
> --
>
> Key: SENTRY-2096
> URL: https://issues.apache.org/jira/browse/SENTRY-2096
> Project: Sentry
> Issue Type: Task
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2096.001.patch
>
>
> Currently, when the unit tests fail in one module, the test stops and report
> failure. It is better to run tests in all modules and then report all failed
> tests. So we can fix test failure at once.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SENTRY-2004) Create Release-2.0.0
[ https://issues.apache.org/jira/browse/SENTRY-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16282457#comment-16282457 ] kalyan kumar kalvagadda commented on SENTRY-2004: - release-2.0.0 is ready and the voting for the release passed. > Create Release-2.0.0 > > > Key: SENTRY-2004 > URL: https://issues.apache.org/jira/browse/SENTRY-2004 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Labels: release > Fix For: 2.0.0 > > > Once release-2.0.0 is ready and no fixes are needed, then we should create > jars, sign them and upload them to the repo. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2093) Close JIRA version for 2.0.0
kalyan kumar kalvagadda created SENTRY-2093: --- Summary: Close JIRA version for 2.0.0 Key: SENTRY-2093 URL: https://issues.apache.org/jira/browse/SENTRY-2093 Project: Sentry Issue Type: Sub-task Components: Sentry Affects Versions: 2.0.0 Reporter: kalyan kumar kalvagadda Fix For: 2.0.0 version 2.0.0 should be closed in jira so that no one uses it for fixVersion. [~spena] Can you help with this? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SENTRY-2004) Create Release-2.0.0
[ https://issues.apache.org/jira/browse/SENTRY-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda resolved SENTRY-2004. - Resolution: Fixed > Create Release-2.0.0 > > > Key: SENTRY-2004 > URL: https://issues.apache.org/jira/browse/SENTRY-2004 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Labels: release > Fix For: 2.0.0 > > > Once release-2.0.0 is ready and no fixes are needed, then we should create > jars, sign them and upload them to the repo. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SENTRY-2005) Run vote on Release-2.0.0
[ https://issues.apache.org/jira/browse/SENTRY-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda resolved SENTRY-2005. - Resolution: Fixed Assignee: kalyan kumar kalvagadda Voting process is complete. > Run vote on Release-2.0.0 > - > > Key: SENTRY-2005 > URL: https://issues.apache.org/jira/browse/SENTRY-2005 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Labels: release > Fix For: 2.0.0 > > > Ask for vote on mailing list. > We need at least 3 PMC members +1 on the release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (SENTRY-2004) Create Release-2.0.0
[ https://issues.apache.org/jira/browse/SENTRY-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda reassigned SENTRY-2004: --- Assignee: kalyan kumar kalvagadda > Create Release-2.0.0 > > > Key: SENTRY-2004 > URL: https://issues.apache.org/jira/browse/SENTRY-2004 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Labels: release > Fix For: 2.0.0 > > > Once release-2.0.0 is ready and no fixes are needed, then we should create > jars, sign them and upload them to the repo. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2034) Add e2e tests for testing HMS notification processing.
[ https://issues.apache.org/jira/browse/SENTRY-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2034: Attachment: SENTRY-2034.005.patch > Add e2e tests for testing HMS notification processing. > -- > > Key: SENTRY-2034 > URL: https://issues.apache.org/jira/browse/SENTRY-2034 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2034.001.patch, SENTRY-2034.002.patch, > SENTRY-2034.002.patch, SENTRY-2034.002.patch, SENTRY-2034.005.patch > > > Currently, there are no e2e tests that test the functionality of pulling the > notifications from HMS and processing them. Which include updating the update > the permissions based on the HMS updates. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2086) Sentry distribution source does not build to to missing modules
[ https://issues.apache.org/jira/browse/SENTRY-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277113#comment-16277113 ] kalyan kumar kalvagadda commented on SENTRY-2086: - [~coheig] sure, I will send a mail cancelling the vote. FYI, branch 2.0.0 was deleted last week. please update you local repo. > Sentry distribution source does not build to to missing modules > --- > > Key: SENTRY-2086 > URL: https://issues.apache.org/jira/browse/SENTRY-2086 > Project: Sentry > Issue Type: Bug >Reporter: Colm O hEigeartaigh >Assignee: Colm O hEigeartaigh > Fix For: 2.0.0 > > Attachments: SENTRY-2086.patch > > > The Sentry distribution source does not build due to some missing modules. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2086) Sentry distribution source does not build to to missing modules
[ https://issues.apache.org/jira/browse/SENTRY-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277103#comment-16277103 ] kalyan kumar kalvagadda commented on SENTRY-2086: - +1 . It works with this change. > Sentry distribution source does not build to to missing modules > --- > > Key: SENTRY-2086 > URL: https://issues.apache.org/jira/browse/SENTRY-2086 > Project: Sentry > Issue Type: Bug >Reporter: Colm O hEigeartaigh >Assignee: Colm O hEigeartaigh > Fix For: 2.0.0 > > Attachments: SENTRY-2086.patch > > > The Sentry distribution source does not build due to some missing modules. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2081) Automate the generation LICENSE.txt based on distributed jars
[ https://issues.apache.org/jira/browse/SENTRY-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276783#comment-16276783 ] kalyan kumar kalvagadda commented on SENTRY-2081: - [~coheig], [~spena], [~akolb], Thanks for the review. > Automate the generation LICENSE.txt based on distributed jars > - > > Key: SENTRY-2081 > URL: https://issues.apache.org/jira/browse/SENTRY-2081 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0, 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.0.0, 2.1.0 > > Attachments: SENTRY-2081.001.patch, SENTRY-2081.003.patch, > SENTRY-2081.005.patch, SENTRY-2081.006.patch, SENTRY-2081.007.patch, > SENTRY-2081.008.patch, SENTRY-2081.009.patch > > > As per https://www.apache.org/dev/licensing-howto.html , sentry should update > the LICENSE.txt file with license information of all the jars that sentry is > distributing along with the pointer to the LICENSE files of the dependencies. > we need to make sure that the license file is generated with sentry is built > based on the jar that it is distributing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2084) Exclude javax.jms:jms from sentry distribution
[ https://issues.apache.org/jira/browse/SENTRY-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276780#comment-16276780 ] kalyan kumar kalvagadda commented on SENTRY-2084: - [~spena] Thanks for the review. > Exclude javax.jms:jms from sentry distribution > -- > > Key: SENTRY-2084 > URL: https://issues.apache.org/jira/browse/SENTRY-2084 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0, 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.0.0, 2.1.0 > > Attachments: SENTRY-2084.001.patch > > > we need to stop distributing jms artifact with group id: javax.jms as these > no valid valid license information available for this jar and secondly it is > a transitive dependency which sentry is not dependent on. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2081) Automate the generation LICENSE.txt based on distributed jars
[ https://issues.apache.org/jira/browse/SENTRY-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2081: Resolution: Fixed Status: Resolved (was: Patch Available) > Automate the generation LICENSE.txt based on distributed jars > - > > Key: SENTRY-2081 > URL: https://issues.apache.org/jira/browse/SENTRY-2081 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0, 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.0.0, 2.1.0 > > Attachments: SENTRY-2081.001.patch, SENTRY-2081.003.patch, > SENTRY-2081.005.patch, SENTRY-2081.006.patch, SENTRY-2081.007.patch, > SENTRY-2081.008.patch, SENTRY-2081.009.patch > > > As per https://www.apache.org/dev/licensing-howto.html , sentry should update > the LICENSE.txt file with license information of all the jars that sentry is > distributing along with the pointer to the LICENSE files of the dependencies. > we need to make sure that the license file is generated with sentry is built > based on the jar that it is distributing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2076) Some test artifacts are not defined at test scope
[ https://issues.apache.org/jira/browse/SENTRY-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2076: Affects Version/s: 2.1.0 2.0.0 > Some test artifacts are not defined at test scope > - > > Key: SENTRY-2076 > URL: https://issues.apache.org/jira/browse/SENTRY-2076 > Project: Sentry > Issue Type: Improvement >Affects Versions: 2.0.0, 2.1.0 >Reporter: Colm O hEigeartaigh >Assignee: Colm O hEigeartaigh >Priority: Minor > Fix For: 2.0.0, 2.1.0 > > Attachments: SENTRY-2076.patch > > > Some test artifacts in Sentry are not defined in test scope. This means that > they get included in the distribution lib directory incorrectly. With the > attached patch, the following jars no longer appear in the distribution lib: > Only in lib/: junit4-ant-2.5.3.jar > Only in lib/: lucene-test-framework-7.1.0.jar > Only in lib/: mockito-all-1.8.5.jar > Only in lib/: randomizedtesting-runner-2.5.3.jar > Only in lib/: solr-test-framework-7.1.0.jar -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2076) Some test artifacts are not defined at test scope
[ https://issues.apache.org/jira/browse/SENTRY-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2076: Fix Version/s: 2.1.0 > Some test artifacts are not defined at test scope > - > > Key: SENTRY-2076 > URL: https://issues.apache.org/jira/browse/SENTRY-2076 > Project: Sentry > Issue Type: Improvement >Reporter: Colm O hEigeartaigh >Assignee: Colm O hEigeartaigh >Priority: Minor > Fix For: 2.0.0, 2.1.0 > > Attachments: SENTRY-2076.patch > > > Some test artifacts in Sentry are not defined in test scope. This means that > they get included in the distribution lib directory incorrectly. With the > attached patch, the following jars no longer appear in the distribution lib: > Only in lib/: junit4-ant-2.5.3.jar > Only in lib/: lucene-test-framework-7.1.0.jar > Only in lib/: mockito-all-1.8.5.jar > Only in lib/: randomizedtesting-runner-2.5.3.jar > Only in lib/: solr-test-framework-7.1.0.jar -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2082) Exclude javax.servlet-3.0.0.v201112011016.jar from Sentry dist
[
https://issues.apache.org/jira/browse/SENTRY-2082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kalyan kumar kalvagadda updated SENTRY-2082:
Fix Version/s: 2.1.0
> Exclude javax.servlet-3.0.0.v201112011016.jar from Sentry dist
> --
>
> Key: SENTRY-2082
> URL: https://issues.apache.org/jira/browse/SENTRY-2082
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Blocker
> Fix For: 2.0.0, 2.1.0
>
> Attachments: SENTRY-2082.1.patch, SENTRY-2082.1.patch
>
>
> The javax.servlet-3.0.0.v201112011016.jar transitive dependency is causing
> some error exceptions on the Sentry 2.0 distribution in some Mac environments.
> {noformat}
> 17/11/29 10:13:25 ERROR thrift.SentryService: Error starting server
> java.lang.SecurityException: class "javax.servlet.DispatcherType"'s signer
> information does not match signer information of other classes in the same
> package
> at java.lang.ClassLoader.checkCerts(ClassLoader.java:898)
> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:668)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:761)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.sentry.service.thrift.SentryService.startSentryWebServer(SentryService.java:422)
> at
> org.apache.sentry.service.thrift.SentryService.runServer(SentryService.java:268)
> at
> org.apache.sentry.service.thrift.SentryService.call(SentryService.java:198)
> at
> org.apache.sentry.service.thrift.SentryService.call(SentryService.java:76)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Exception in thread "main" java.util.concurrent.ExecutionException:
> java.lang.Exception: Error starting server
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.sentry.service.thrift.SentryService$CommandImpl.run(SentryService.java:591)
> at org.apache.sentry.SentryMain.main(SentryMain.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> Caused by: java.lang.Exception: Error starting server
> at
> org.apache.sentry.service.thrift.SentryService.call(SentryService.java:202)
> at
> org.apache.sentry.service.thrift.SentryService.call(SentryService.java:76)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.SecurityException: class
> "javax.servlet.DispatcherType"'s signer information does not match signer
> information of other classes in the same package
> at java.lang.ClassLoader.checkCerts(ClassLoader.java:898)
> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:668)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:761)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at
[jira] [Updated] (SENTRY-2079) Sentry HA leader monitor does not work due to a mix of curator versions in the classpath
[ https://issues.apache.org/jira/browse/SENTRY-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2079: Fix Version/s: 2.1.0 > Sentry HA leader monitor does not work due to a mix of curator versions in > the classpath > > > Key: SENTRY-2079 > URL: https://issues.apache.org/jira/browse/SENTRY-2079 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Blocker > Fix For: 2.0.0, 2.1.0 > > Attachments: SENTRY-2079.1.patch > > > While testing Sentry 2.0, I couldn't enable HA mode because connections to > Zookeeper were stuck due to a mix of Apache curator dependencies > (curator-client-2.8.0 and curator-client-2.11.1). > We need to exclude the old curator dependencies and also shade the curator > jars so that it does not affect other components. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2081) Automate the generation LICENSE.txt based on distributed jars
[ https://issues.apache.org/jira/browse/SENTRY-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2081: Fix Version/s: 2.1.0 2.0.0 > Automate the generation LICENSE.txt based on distributed jars > - > > Key: SENTRY-2081 > URL: https://issues.apache.org/jira/browse/SENTRY-2081 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0, 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.0.0, 2.1.0 > > Attachments: SENTRY-2081.001.patch, SENTRY-2081.003.patch, > SENTRY-2081.005.patch, SENTRY-2081.006.patch, SENTRY-2081.007.patch, > SENTRY-2081.008.patch, SENTRY-2081.009.patch > > > As per https://www.apache.org/dev/licensing-howto.html , sentry should update > the LICENSE.txt file with license information of all the jars that sentry is > distributing along with the pointer to the LICENSE files of the dependencies. > we need to make sure that the license file is generated with sentry is built > based on the jar that it is distributing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2084) Exclude javax.jms:jms from sentry distribution
[ https://issues.apache.org/jira/browse/SENTRY-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2084: Resolution: Fixed Fix Version/s: 2.1.0 2.0.0 Status: Resolved (was: Patch Available) > Exclude javax.jms:jms from sentry distribution > -- > > Key: SENTRY-2084 > URL: https://issues.apache.org/jira/browse/SENTRY-2084 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0, 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.0.0, 2.1.0 > > Attachments: SENTRY-2084.001.patch > > > we need to stop distributing jms artifact with group id: javax.jms as these > no valid valid license information available for this jar and secondly it is > a transitive dependency which sentry is not dependent on. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2084) Exclude javax.jms:jms from sentry distribution
[ https://issues.apache.org/jira/browse/SENTRY-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2084: Status: Patch Available (was: In Progress) > Exclude javax.jms:jms from sentry distribution > -- > > Key: SENTRY-2084 > URL: https://issues.apache.org/jira/browse/SENTRY-2084 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0, 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Attachments: SENTRY-2084.001.patch > > > we need to stop distributing jms artifact with group id: javax.jms as these > no valid valid license information available for this jar and secondly it is > a transitive dependency which sentry is not dependent on. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2081) Automate the generation LICENSE.txt based on distributed jars
[ https://issues.apache.org/jira/browse/SENTRY-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2081: Description: As per https://www.apache.org/dev/licensing-howto.html , sentry should update the LICENSE.txt file with license information of all the jars that sentry is distributing along with the pointer to the LICENSE files of the dependencies. we need to make sure that the license file is generated with sentry is built based on the jar that it is distributing. was:As per https://www.apache.org/dev/licensing-howto.html , sentry should update the LICENSE.txt file with license information of all the jars that sentry is distributing along with the pointer to the LICENSE files of the dependencies. > Automate the generation LICENSE.txt based on distributed jars > - > > Key: SENTRY-2081 > URL: https://issues.apache.org/jira/browse/SENTRY-2081 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0, 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Attachments: SENTRY-2081.001.patch, SENTRY-2081.003.patch, > SENTRY-2081.005.patch, SENTRY-2081.006.patch, SENTRY-2081.007.patch, > SENTRY-2081.008.patch, SENTRY-2081.009.patch > > > As per https://www.apache.org/dev/licensing-howto.html , sentry should update > the LICENSE.txt file with license information of all the jars that sentry is > distributing along with the pointer to the LICENSE files of the dependencies. > we need to make sure that the license file is generated with sentry is built > based on the jar that it is distributing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2081) Automate the generation LICENSE.txt based on distributed jars
[ https://issues.apache.org/jira/browse/SENTRY-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2081: Summary: Automate the generation LICENSE.txt based on distributed jars (was: Update the LICENSE.txt with the license information of distributed jars) > Automate the generation LICENSE.txt based on distributed jars > - > > Key: SENTRY-2081 > URL: https://issues.apache.org/jira/browse/SENTRY-2081 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0, 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Attachments: SENTRY-2081.001.patch, SENTRY-2081.003.patch, > SENTRY-2081.005.patch, SENTRY-2081.006.patch, SENTRY-2081.007.patch, > SENTRY-2081.008.patch, SENTRY-2081.009.patch > > > As per https://www.apache.org/dev/licensing-howto.html , sentry should update > the LICENSE.txt file with license information of all the jars that sentry is > distributing along with the pointer to the LICENSE files of the dependencies. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2084) Exclude javax.jms:jms from sentry distribution
[ https://issues.apache.org/jira/browse/SENTRY-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2084: Attachment: SENTRY-2084.001.patch > Exclude javax.jms:jms from sentry distribution > -- > > Key: SENTRY-2084 > URL: https://issues.apache.org/jira/browse/SENTRY-2084 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0, 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Attachments: SENTRY-2084.001.patch > > > we need to stop distributing jms artifact with group id: javax.jms as these > no valid valid license information available for this jar and secondly it is > a transitive dependency which sentry is not dependent on. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2084) Exclude javax.jms:jms from sentry distribution
kalyan kumar kalvagadda created SENTRY-2084: --- Summary: Exclude javax.jms:jms from sentry distribution Key: SENTRY-2084 URL: https://issues.apache.org/jira/browse/SENTRY-2084 Project: Sentry Issue Type: Bug Components: Sentry Affects Versions: 2.0.0, 2.1.0 Reporter: kalyan kumar kalvagadda Assignee: kalyan kumar kalvagadda we need to stop distributing jms artifact with group id: javax.jms as these no valid valid license information available for this jar and secondly it is a transitive dependency which sentry is not dependent on. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2081) Update the LICENSE.txt with the license information of distributed jars
[ https://issues.apache.org/jira/browse/SENTRY-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2081: Attachment: SENTRY-2081.008.patch > Update the LICENSE.txt with the license information of distributed jars > --- > > Key: SENTRY-2081 > URL: https://issues.apache.org/jira/browse/SENTRY-2081 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0, 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Attachments: SENTRY-2081.001.patch, SENTRY-2081.003.patch, > SENTRY-2081.005.patch, SENTRY-2081.006.patch, SENTRY-2081.007.patch, > SENTRY-2081.008.patch > > > As per https://www.apache.org/dev/licensing-howto.html , sentry should update > the LICENSE.txt file with license information of all the jars that sentry is > distributing along with the pointer to the LICENSE files of the dependencies. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
