[jira] [Commented] (SENTRY-2558) Issue in creating full snapshot when the storage descriptor for a table is null.
[
https://issues.apache.org/jira/browse/SENTRY-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198502#comment-17198502
]
Na Li commented on SENTRY-2558:
---
[~kalyan] why the tests were not triggered?
> Issue in creating full snapshot when the storage descriptor for a table is
> null.
>
>
> Key: SENTRY-2558
> URL: https://issues.apache.org/jira/browse/SENTRY-2558
> Project: Sentry
> Issue Type: Bug
>Reporter: Kalyan Kalvagadda
>Assignee: Kalyan Kalvagadda
>Priority: Major
> Fix For: 2.3.0
>
> Attachments: SENTRY-2558.001.patch
>
>
> When the storage descriptor for the table is null, sentry throws NullPointer
> Exception and fails to create a snapshot.
> {noformat}
> 020-09-17 15:08:43,647 ERROR
> org.apache.sentry.service.thrift.FullUpdateInitializer: Failed to execute
> task java.lang.NullPointerException at
> org.apache.sentry.service.thrift.FullUpdateInitializer$TableTask.doTask(FullUpdateInitializer.java:369)
> at
> org.apache.sentry.service.thrift.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:222)
> at
> org.apache.sentry.service.thrift.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:258)
> at
> org.apache.sentry.service.thrift.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:190)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) 2020-09-17 15:08:43,650
> INFO org.apache.sentry.service.thrift.FullUpdateInitializer: For db =
> test_db4 table = test_tb5 total number of partitions = 0 2020-09-17
> 15:08:43,657 INFO hive.metastore: Closed a connection to metastore, current
> connections: 8 2020-09-17 15:08:43,657 ERROR
> org.apache.sentry.service.thrift.FullUpdateInitializer: Failed to execute
> task java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@68c2c9a6 rejected from
> java.util.concurrent.ThreadPoolExecutor@1a289856[Shutting down, pool size =
> 7, active threads = 7, queued tasks = 0, completed tasks = 9] at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
> at
> org.apache.sentry.service.thrift.FullUpdateInitializer$TableTask.doTask(FullUpdateInitializer.java:366)
> at
> org.apache.sentry.service.thrift.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:222)
> at
> org.apache.sentry.service.thrift.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:258)
> at
> org.apache.sentry.service.thrift.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:190)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> Instead, it should ignore the object and move-on.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li closed SENTRY-2422.
-
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Fix For: 2.3.0
>
> Attachments: SENTRY-2422.002.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2422:
--
Fix Version/s: 2.3.0
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Fix For: 2.3.0
>
> Attachments: SENTRY-2422.002.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2422:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2422.002.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2422:
--
Attachment: (was: SENTRY-2422.001.patch)
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2422.002.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2422:
--
Attachment: (was: SENTRY-2422.001.patch)
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2422.002.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2422:
--
Attachment: (was: SENTRY-2422.001.patch)
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2422.002.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2422:
--
Attachment: (was: SENTRY-2422.003.patch)
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2422.002.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2422:
--
Attachment: (was: SENTRY-2422.001.patch)
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2422.002.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2422:
--
Attachment: SENTRY-2422.003.patch
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2422.001.patch, SENTRY-2422.001.patch,
> SENTRY-2422.001.patch, SENTRY-2422.001.patch, SENTRY-2422.002.patch,
> SENTRY-2422.003.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2422:
--
Attachment: SENTRY-2422.002.patch
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2422.001.patch, SENTRY-2422.001.patch,
> SENTRY-2422.001.patch, SENTRY-2422.001.patch, SENTRY-2422.002.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2422:
--
Attachment: SENTRY-2422.001.patch
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2422.001.patch, SENTRY-2422.001.patch,
> SENTRY-2422.001.patch, SENTRY-2422.001.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2422:
--
Attachment: SENTRY-2422.001.patch
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2422.001.patch, SENTRY-2422.001.patch,
> SENTRY-2422.001.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164490#comment-17164490
]
Na Li commented on SENTRY-2422:
---
The failed test
"org.apache.sentry.hdfs.TestSentryHDFSServiceProcessor.testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate"
passed locally. It is likely failed due to timing. Re-trigger the test.
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2422.001.patch, SENTRY-2422.001.patch,
> SENTRY-2422.001.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2422:
--
Attachment: SENTRY-2422.001.patch
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2422.001.patch, SENTRY-2422.001.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2422:
--
Affects Version/s: 2.1.0
Status: Patch Available (was: In Progress)
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.0.1, 2.1.0
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2422.001.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2422:
--
Attachment: SENTRY-2422.001.patch
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.0.1
>Reporter: Brian Towles
>Assignee: Krishna Kalyan
>Priority: Major
> Attachments: SENTRY-2422.001.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li reassigned SENTRY-2422:
-
Assignee: Na Li (was: Krishna Kalyan)
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.0.1
>Reporter: Brian Towles
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2422.001.patch
>
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SENTRY-2422) HMS synchronization is causing multiple entries of the same ID in SENTRY_HMS_NOTIFICATION_ID
[
https://issues.apache.org/jira/browse/SENTRY-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161497#comment-17161497
]
Na Li commented on SENTRY-2422:
---
The code change should be at
[https://github.com/apache/sentry/blob/master/sentry-service/sentry-service-server/src/main/java/org/apache/sentry/provider/db/service/persistent/SentryStore.java#L4490]
1) in the same transaction
1.1) Get the current max notification ID using getLastProcessedChangeIDCore
1.2) If the input notificationId is smaller than or equal to the max value,
simple return. Otherwise, persist the value
This should avoid duplicated values in notification ID in table
SENTRY_HMS_NOTIFICATION_ID
> HMS synchronization is causing multiple entries of the same ID in
> SENTRY_HMS_NOTIFICATION_ID
>
>
> Key: SENTRY-2422
> URL: https://issues.apache.org/jira/browse/SENTRY-2422
> Project: Sentry
> Issue Type: Improvement
> Components: Core
>Affects Versions: 2.0.1
>Reporter: Brian Towles
>Assignee: Krishna Kalyan
>Priority: Major
>
> When processing updates for the HMS notifications, multiple entries of the
> same ID are being put into the SENTRY_HMS_NOTIFICATION_ID table.
> This can be seen after doing a non location changing ALTER TABLE in hive for
> example.
>
> ALTER TABLE sometable SET TBLPROPERTIES ("someproperty"="somevalue");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2549) Only groups but not users are passed to PrivilegeCache
[ https://issues.apache.org/jira/browse/SENTRY-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2549: -- Attachment: SENTRY-2549.001.patch > Only groups but not users are passed to PrivilegeCache > -- > > Key: SENTRY-2549 > URL: https://issues.apache.org/jira/browse/SENTRY-2549 > Project: Sentry > Issue Type: Bug >Reporter: Csaba Ringhofer >Priority: Critical > Attachments: SENTRY-2549.001.patch, SENTRY-2549.001.patch, > SENTRY-2549.001.patch > > > SENTRY-2539 introduced this issue, see line 83 of > SimpleDBProviderBackend.java at > https://github.com/apache/sentry/commit/843368fc2aeb154f9cf7b18d191c128644295ce5#diff-171087e076ec1622cb69536426d35b4a > I guess that this means that privileges assigned to groups are respected, but > user privileges are ignored. I think that this is the reason behind recent > test failures in Apache Impala. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2545) Rolling back Privilege Cache to SimplePrivilegeCache does not work
[ https://issues.apache.org/jira/browse/SENTRY-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2545: -- Attachment: SENTRY-2545.002.patch > Rolling back Privilege Cache to SimplePrivilegeCache does not work > -- > > Key: SENTRY-2545 > URL: https://issues.apache.org/jira/browse/SENTRY-2545 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2545.001.patch, SENTRY-2545.002.patch > > > The change in SENTRY-2539 uses TreePrivilegeCache as default implementation > of privilege cache to improve authorization performance. However, rolling > back to SimplePrivilegeCache does not work due to how Sentry creates the > privilege cache. > The solution is to create the privilege cache based on configured class name > and handle different constructor inputs properly. > More test cases are added to verify that rollback works. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2545) Rolling back Privilege Cache to SimplePrivilegeCache does not work
[ https://issues.apache.org/jira/browse/SENTRY-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2545: -- Status: Patch Available (was: Open) > Rolling back Privilege Cache to SimplePrivilegeCache does not work > -- > > Key: SENTRY-2545 > URL: https://issues.apache.org/jira/browse/SENTRY-2545 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2545.001.patch > > > The change in SENTRY-2539 uses TreePrivilegeCache as default implementation > of privilege cache to improve authorization performance. However, rolling > back to SimplePrivilegeCache does not work due to how Sentry creates the > privilege cache. > The solution is to create the privilege cache based on configured class name > and handle different constructor inputs properly. > More test cases are added to verify that rollback works. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2545) Rolling back Privilege Cache to SimplePrivilegeCache does not work
[ https://issues.apache.org/jira/browse/SENTRY-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2545: -- Attachment: SENTRY-2545.001.patch > Rolling back Privilege Cache to SimplePrivilegeCache does not work > -- > > Key: SENTRY-2545 > URL: https://issues.apache.org/jira/browse/SENTRY-2545 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2545.001.patch > > > The change in SENTRY-2539 uses TreePrivilegeCache as default implementation > of privilege cache to improve authorization performance. However, rolling > back to SimplePrivilegeCache does not work due to how Sentry creates the > privilege cache. > The solution is to create the privilege cache based on configured class name > and handle different constructor inputs properly. > More test cases are added to verify that rollback works. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2545) Rolling back Privilege Cache to SimplePrivilegeCache does not work
[ https://issues.apache.org/jira/browse/SENTRY-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2545: -- Summary: Rolling back Privilege Cache to SimplePrivilegeCache does not work (was: Rollback Privilege Cache to SimplePrivilegeCache does not work) > Rolling back Privilege Cache to SimplePrivilegeCache does not work > -- > > Key: SENTRY-2545 > URL: https://issues.apache.org/jira/browse/SENTRY-2545 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > > The change in SENTRY-2539 uses TreePrivilegeCache as default implementation > of privilege cache to improve authorization performance. However, rolling > back to SimplePrivilegeCache does not work due to how Sentry creates the > privilege cache. > The solution is to create the privilege cache based on configured class name > and handle different constructor inputs properly. > More test cases are added to verify that rollback works. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (SENTRY-2545) Rollback Privilege Cache to SimplePrivilegeCache does not work
Na Li created SENTRY-2545: - Summary: Rollback Privilege Cache to SimplePrivilegeCache does not work Key: SENTRY-2545 URL: https://issues.apache.org/jira/browse/SENTRY-2545 Project: Sentry Issue Type: Bug Components: Sentry Affects Versions: 2.1 Reporter: Na Li Assignee: Na Li The change in SENTRY-2539 uses TreePrivilegeCache as default implementation of privilege cache to improve authorization performance. However, rolling back to SimplePrivilegeCache does not work due to how Sentry creates the privilege cache. The solution is to create the privilege cache based on configured class name and handle different constructor inputs properly. More test cases are added to verify that rollback works. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.014.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch, SENTRY-2539.006.patch, SENTRY-2539.007.patch, > SENTRY-2539.008.patch, SENTRY-2539.008.patch, SENTRY-2539.008.patch, > SENTRY-2539.009.patch, SENTRY-2539.010.patch, SENTRY-2539.010.patch, > SENTRY-2539.013.patch, SENTRY-2539.013.patch, SENTRY-2539.013.patch, > SENTRY-2539.013.patch, SENTRY-2539.014.patch > > > *Problem*: > Right now, for a command such "show databases", Sentry has to perform > authorization checks on each database. When there are many databases, like > 12000 databases in the system, the authorization checks of a single command > in Sentry could be very slow. There are two main factors that slow down > authorization checks in Sentry even when caching is enabled: > 1) Cache returns the list of privileges in the form of String. As a result, > every authorization check has to convert the privilege string to privilege > object. > 2) When cache is enabled, the cache returns all privileges of a given user > regardless what resource to check. > 2.1) for example, a user has 2000 privileges assigned and the resource to > check is "server=server1, database=db_1, table=table_1". The cache returns > all 2000 privileges including unrelated privileges such like > "server=server1->database=db_2->action=ALL". > 2.2) Returning unrelated privileges has two side effects: > 2.2.1) Converting privileges from String to Object overhead is > proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > 2.2.2) Authorization check goes through each privilege, and its overhead > is proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > *Solution*: > 1) Add a new function listPrivilegeObjects that lets authorization provider > get privilege objects when checking the authorization. This avoids the > conversion overhead. All the interfaces from policy engine (PolicyEngine) to > the cache (PrivilegeCache) have to be changed to add this new function. > 2) Implement a new cache TreePrivilegeCache. It converts the privilege from > String format to Privilege object at beginning, and directly return the > privilege objects in listPrivilegeObjects at authorization check. This avoids > the overhead of conversion at each authorization check. > 3) TreePrivilegeCache organizes the privileges based on the resource > hierarchy, like a tree. Therefore, it can return only related privileges > based on the resource to check. This reduces the authorization check > overhead. > 3.1) For example, a user has 2000 privileges assigned, and the resource to > check is "server=server1, database=db_1, table=table_1". the cache > TreePrivilegeCache returns only related privileges excluding unrelated > privileges such like "server=server1->database=db_2->action=ALL". > 3.2) SENTRY-1291 was to address the problem 2). However, it did not address > the problem 1). And its implementation SimplePrivilegeCache is not memory > efficient (the key of the map contains the whole resource hierarchy, and many > keys share large portion of the same content), nor operational efficient (for > each authorization check, SimplePrivilegeCache .listPrivileges() has to > construct a large amount of keys in order to find all related privileges in a > map). > 4) Use TreePrivilegeCache instead of SimplePrivilegeCache for caching. Note, > this solution is built on top of SENTRY-1291, and utilizes the changes > SENTRY-1291 made, such as providing resource hierarchy when getting > privileges for authorization check. > > *Major Behavior Change* > 1) Create a new Interface FilteredPrivilegeCache, which extends from > PrivilegeCache. > 2) Move the function added by SENTRY-1291 in PrivilegeCache to > FilteredPrivilegeCache. Add additional functions in this solution to > FilteredPrivilegeCache. In this way, there is no change in PrivilegeCache, > and we are backward compatible with old implementation before SENTRY-1291. > 3) Move all changed in SimplePrivilegeCache (implements PrivilegeCache) from > SENTRY-1291 to a new class SimpleFilteredPrivilegeCache, which implements > FilteredPrivilegeCache. > 4) Instead of hard-coding the privileg
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.013.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch, SENTRY-2539.006.patch, SENTRY-2539.007.patch, > SENTRY-2539.008.patch, SENTRY-2539.008.patch, SENTRY-2539.008.patch, > SENTRY-2539.009.patch, SENTRY-2539.010.patch, SENTRY-2539.010.patch, > SENTRY-2539.013.patch, SENTRY-2539.013.patch, SENTRY-2539.013.patch, > SENTRY-2539.013.patch > > > *Problem*: > Right now, for a command such "show databases", Sentry has to perform > authorization checks on each database. When there are many databases, like > 12000 databases in the system, the authorization checks of a single command > in Sentry could be very slow. There are two main factors that slow down > authorization checks in Sentry even when caching is enabled: > 1) Cache returns the list of privileges in the form of String. As a result, > every authorization check has to convert the privilege string to privilege > object. > 2) When cache is enabled, the cache returns all privileges of a given user > regardless what resource to check. > 2.1) for example, a user has 2000 privileges assigned and the resource to > check is "server=server1, database=db_1, table=table_1". The cache returns > all 2000 privileges including unrelated privileges such like > "server=server1->database=db_2->action=ALL". > 2.2) Returning unrelated privileges has two side effects: > 2.2.1) Converting privileges from String to Object overhead is > proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > 2.2.2) Authorization check goes through each privilege, and its overhead > is proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > *Solution*: > 1) Add a new function listPrivilegeObjects that lets authorization provider > get privilege objects when checking the authorization. This avoids the > conversion overhead. All the interfaces from policy engine (PolicyEngine) to > the cache (PrivilegeCache) have to be changed to add this new function. > 2) Implement a new cache TreePrivilegeCache. It converts the privilege from > String format to Privilege object at beginning, and directly return the > privilege objects in listPrivilegeObjects at authorization check. This avoids > the overhead of conversion at each authorization check. > 3) TreePrivilegeCache organizes the privileges based on the resource > hierarchy, like a tree. Therefore, it can return only related privileges > based on the resource to check. This reduces the authorization check > overhead. > 3.1) For example, a user has 2000 privileges assigned, and the resource to > check is "server=server1, database=db_1, table=table_1". the cache > TreePrivilegeCache returns only related privileges excluding unrelated > privileges such like "server=server1->database=db_2->action=ALL". > 3.2) SENTRY-1291 was to address the problem 2). However, it did not address > the problem 1). And its implementation SimplePrivilegeCache is not memory > efficient (the key of the map contains the whole resource hierarchy, and many > keys share large portion of the same content), nor operational efficient (for > each authorization check, SimplePrivilegeCache .listPrivileges() has to > construct a large amount of keys in order to find all related privileges in a > map). > 4) Use TreePrivilegeCache instead of SimplePrivilegeCache for caching. Note, > this solution is built on top of SENTRY-1291, and utilizes the changes > SENTRY-1291 made, such as providing resource hierarchy when getting > privileges for authorization check. > > *Major Behavior Change* > 1) Create a new Interface FilteredPrivilegeCache, which extends from > PrivilegeCache. > 2) Move the function added by SENTRY-1291 in PrivilegeCache to > FilteredPrivilegeCache. Add additional functions in this solution to > FilteredPrivilegeCache. In this way, there is no change in PrivilegeCache, > and we are backward compatible with old implementation before SENTRY-1291. > 3) Move all changed in SimplePrivilegeCache (implements PrivilegeCache) from > SENTRY-1291 to a new class SimpleFilteredPrivilegeCache, which implements > FilteredPrivilegeCache. > 4) Instead of hard-coding the privilege cache class, use conf
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.013.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch, SENTRY-2539.006.patch, SENTRY-2539.007.patch, > SENTRY-2539.008.patch, SENTRY-2539.008.patch, SENTRY-2539.008.patch, > SENTRY-2539.009.patch, SENTRY-2539.010.patch, SENTRY-2539.010.patch, > SENTRY-2539.013.patch, SENTRY-2539.013.patch, SENTRY-2539.013.patch > > > *Problem*: > Right now, for a command such "show databases", Sentry has to perform > authorization checks on each database. When there are many databases, like > 12000 databases in the system, the authorization checks of a single command > in Sentry could be very slow. There are two main factors that slow down > authorization checks in Sentry even when caching is enabled: > 1) Cache returns the list of privileges in the form of String. As a result, > every authorization check has to convert the privilege string to privilege > object. > 2) When cache is enabled, the cache returns all privileges of a given user > regardless what resource to check. > 2.1) for example, a user has 2000 privileges assigned and the resource to > check is "server=server1, database=db_1, table=table_1". The cache returns > all 2000 privileges including unrelated privileges such like > "server=server1->database=db_2->action=ALL". > 2.2) Returning unrelated privileges has two side effects: > 2.2.1) Converting privileges from String to Object overhead is > proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > 2.2.2) Authorization check goes through each privilege, and its overhead > is proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > *Solution*: > 1) Add a new function listPrivilegeObjects that lets authorization provider > get privilege objects when checking the authorization. This avoids the > conversion overhead. All the interfaces from policy engine (PolicyEngine) to > the cache (PrivilegeCache) have to be changed to add this new function. > 2) Implement a new cache TreePrivilegeCache. It converts the privilege from > String format to Privilege object at beginning, and directly return the > privilege objects in listPrivilegeObjects at authorization check. This avoids > the overhead of conversion at each authorization check. > 3) TreePrivilegeCache organizes the privileges based on the resource > hierarchy, like a tree. Therefore, it can return only related privileges > based on the resource to check. This reduces the authorization check > overhead. > 3.1) For example, a user has 2000 privileges assigned, and the resource to > check is "server=server1, database=db_1, table=table_1". the cache > TreePrivilegeCache returns only related privileges excluding unrelated > privileges such like "server=server1->database=db_2->action=ALL". > 3.2) SENTRY-1291 was to address the problem 2). However, it did not address > the problem 1). And its implementation SimplePrivilegeCache is not memory > efficient (the key of the map contains the whole resource hierarchy, and many > keys share large portion of the same content), nor operational efficient (for > each authorization check, SimplePrivilegeCache .listPrivileges() has to > construct a large amount of keys in order to find all related privileges in a > map). > 4) Use TreePrivilegeCache instead of SimplePrivilegeCache for caching. Note, > this solution is built on top of SENTRY-1291, and utilizes the changes > SENTRY-1291 made, such as providing resource hierarchy when getting > privileges for authorization check. > > *Major Behavior Change* > 1) Create a new Interface FilteredPrivilegeCache, which extends from > PrivilegeCache. > 2) Move the function added by SENTRY-1291 in PrivilegeCache to > FilteredPrivilegeCache. Add additional functions in this solution to > FilteredPrivilegeCache. In this way, there is no change in PrivilegeCache, > and we are backward compatible with old implementation before SENTRY-1291. > 3) Move all changed in SimplePrivilegeCache (implements PrivilegeCache) from > SENTRY-1291 to a new class SimpleFilteredPrivilegeCache, which implements > FilteredPrivilegeCache. > 4) Instead of hard-coding the privilege cache class, use configuration > AuthzConfVars
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.013.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch, SENTRY-2539.006.patch, SENTRY-2539.007.patch, > SENTRY-2539.008.patch, SENTRY-2539.008.patch, SENTRY-2539.008.patch, > SENTRY-2539.009.patch, SENTRY-2539.010.patch, SENTRY-2539.010.patch, > SENTRY-2539.013.patch, SENTRY-2539.013.patch > > > *Problem*: > Right now, for a command such "show databases", Sentry has to perform > authorization checks on each database. When there are many databases, like > 12000 databases in the system, the authorization checks of a single command > in Sentry could be very slow. There are two main factors that slow down > authorization checks in Sentry even when caching is enabled: > 1) Cache returns the list of privileges in the form of String. As a result, > every authorization check has to convert the privilege string to privilege > object. > 2) When cache is enabled, the cache returns all privileges of a given user > regardless what resource to check. > 2.1) for example, a user has 2000 privileges assigned and the resource to > check is "server=server1, database=db_1, table=table_1". The cache returns > all 2000 privileges including unrelated privileges such like > "server=server1->database=db_2->action=ALL". > 2.2) Returning unrelated privileges has two side effects: > 2.2.1) Converting privileges from String to Object overhead is > proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > 2.2.2) Authorization check goes through each privilege, and its overhead > is proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > *Solution*: > 1) Add a new function listPrivilegeObjects that lets authorization provider > get privilege objects when checking the authorization. This avoids the > conversion overhead. All the interfaces from policy engine (PolicyEngine) to > the cache (PrivilegeCache) have to be changed to add this new function. > 2) Implement a new cache TreePrivilegeCache. It converts the privilege from > String format to Privilege object at beginning, and directly return the > privilege objects in listPrivilegeObjects at authorization check. This avoids > the overhead of conversion at each authorization check. > 3) TreePrivilegeCache organizes the privileges based on the resource > hierarchy, like a tree. Therefore, it can return only related privileges > based on the resource to check. This reduces the authorization check > overhead. > 3.1) For example, a user has 2000 privileges assigned, and the resource to > check is "server=server1, database=db_1, table=table_1". the cache > TreePrivilegeCache returns only related privileges excluding unrelated > privileges such like "server=server1->database=db_2->action=ALL". > 3.2) SENTRY-1291 was to address the problem 2). However, it did not address > the problem 1). And its implementation SimplePrivilegeCache is not memory > efficient (the key of the map contains the whole resource hierarchy, and many > keys share large portion of the same content), nor operational efficient (for > each authorization check, SimplePrivilegeCache .listPrivileges() has to > construct a large amount of keys in order to find all related privileges in a > map). > 4) Use TreePrivilegeCache instead of SimplePrivilegeCache for caching. Note, > this solution is built on top of SENTRY-1291, and utilizes the changes > SENTRY-1291 made, such as providing resource hierarchy when getting > privileges for authorization check. > > *Major Behavior Change* > 1) Create a new Interface FilteredPrivilegeCache, which extends from > PrivilegeCache. > 2) Move the function added by SENTRY-1291 in PrivilegeCache to > FilteredPrivilegeCache. Add additional functions in this solution to > FilteredPrivilegeCache. In this way, there is no change in PrivilegeCache, > and we are backward compatible with old implementation before SENTRY-1291. > 3) Move all changed in SimplePrivilegeCache (implements PrivilegeCache) from > SENTRY-1291 to a new class SimpleFilteredPrivilegeCache, which implements > FilteredPrivilegeCache. > 4) Instead of hard-coding the privilege cache class, use configuration > AuthzConfVars.AUTHZ_PRIVILEGE_CACHE
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.013.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch, SENTRY-2539.006.patch, SENTRY-2539.007.patch, > SENTRY-2539.008.patch, SENTRY-2539.008.patch, SENTRY-2539.008.patch, > SENTRY-2539.009.patch, SENTRY-2539.010.patch, SENTRY-2539.010.patch, > SENTRY-2539.013.patch, SENTRY-2539.013.patch > > > *Problem*: > Right now, for a command such "show databases", Sentry has to perform > authorization checks on each database. When there are many databases, like > 12000 databases in the system, the authorization checks of a single command > in Sentry could be very slow. There are two main factors that slow down > authorization checks in Sentry even when caching is enabled: > 1) Cache returns the list of privileges in the form of String. As a result, > every authorization check has to convert the privilege string to privilege > object. > 2) When cache is enabled, the cache returns all privileges of a given user > regardless what resource to check. > 2.1) for example, a user has 2000 privileges assigned and the resource to > check is "server=server1, database=db_1, table=table_1". The cache returns > all 2000 privileges including unrelated privileges such like > "server=server1->database=db_2->action=ALL". > 2.2) Returning unrelated privileges has two side effects: > 2.2.1) Converting privileges from String to Object overhead is > proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > 2.2.2) Authorization check goes through each privilege, and its overhead > is proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > *Solution*: > 1) Add a new function listPrivilegeObjects that lets authorization provider > get privilege objects when checking the authorization. This avoids the > conversion overhead. All the interfaces from policy engine (PolicyEngine) to > the cache (PrivilegeCache) have to be changed to add this new function. > 2) Implement a new cache TreePrivilegeCache. It converts the privilege from > String format to Privilege object at beginning, and directly return the > privilege objects in listPrivilegeObjects at authorization check. This avoids > the overhead of conversion at each authorization check. > 3) TreePrivilegeCache organizes the privileges based on the resource > hierarchy, like a tree. Therefore, it can return only related privileges > based on the resource to check. This reduces the authorization check > overhead. > 3.1) For example, a user has 2000 privileges assigned, and the resource to > check is "server=server1, database=db_1, table=table_1". the cache > TreePrivilegeCache returns only related privileges excluding unrelated > privileges such like "server=server1->database=db_2->action=ALL". > 3.2) SENTRY-1291 was to address the problem 2). However, it did not address > the problem 1). And its implementation SimplePrivilegeCache is not memory > efficient (the key of the map contains the whole resource hierarchy, and many > keys share large portion of the same content), nor operational efficient (for > each authorization check, SimplePrivilegeCache .listPrivileges() has to > construct a large amount of keys in order to find all related privileges in a > map). > 4) Use TreePrivilegeCache instead of SimplePrivilegeCache for caching. Note, > this solution is built on top of SENTRY-1291, and utilizes the changes > SENTRY-1291 made, such as providing resource hierarchy when getting > privileges for authorization check. > > *Major Behavior Change* > 1) Create a new Interface FilteredPrivilegeCache, which extends from > PrivilegeCache. > 2) Move the function added by SENTRY-1291 in PrivilegeCache to > FilteredPrivilegeCache. Add additional functions in this solution to > FilteredPrivilegeCache. In this way, there is no change in PrivilegeCache, > and we are backward compatible with old implementation before SENTRY-1291. > 3) Move all changed in SimplePrivilegeCache (implements PrivilegeCache) from > SENTRY-1291 to a new class SimpleFilteredPrivilegeCache, which implements > FilteredPrivilegeCache. > 4) Instead of hard-coding the privilege cache class, use configuration > AuthzConfVars.AUTHZ_PRIVILEGE_CACHE
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[
https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2539:
--
Description:
*Problem*:
Right now, for a command such "show databases", Sentry has to perform
authorization checks on each database. When there are many databases, like
12000 databases in the system, the authorization checks of a single command in
Sentry could be very slow. There are two main factors that slow down
authorization checks in Sentry even when caching is enabled:
1) Cache returns the list of privileges in the form of String. As a result,
every authorization check has to convert the privilege string to privilege
object.
2) When cache is enabled, the cache returns all privileges of a given user
regardless what resource to check.
2.1) for example, a user has 2000 privileges assigned and the resource to
check is "server=server1, database=db_1, table=table_1". The cache returns all
2000 privileges including unrelated privileges such like
"server=server1->database=db_2->action=ALL".
2.2) Returning unrelated privileges has two side effects:
2.2.1) Converting privileges from String to Object overhead is proportional
to the number of returned privileges from cache. Converting unrelated
privileges cost time, but no benefit.
2.2.2) Authorization check goes through each privilege, and its overhead is
proportional to the number of returned privileges from cache. Converting
unrelated privileges cost time, but no benefit.
*Solution*:
1) Add a new function listPrivilegeObjects that lets authorization provider get
privilege objects when checking the authorization. This avoids the conversion
overhead. All the interfaces from policy engine (PolicyEngine) to the cache
(PrivilegeCache) have to be changed to add this new function.
2) Implement a new cache TreePrivilegeCache. It converts the privilege from
String format to Privilege object at beginning, and directly return the
privilege objects in listPrivilegeObjects at authorization check. This avoids
the overhead of conversion at each authorization check.
3) TreePrivilegeCache organizes the privileges based on the resource hierarchy,
like a tree. Therefore, it can return only related privileges based on the
resource to check. This reduces the authorization check overhead.
3.1) For example, a user has 2000 privileges assigned, and the resource to
check is "server=server1, database=db_1, table=table_1". the cache
TreePrivilegeCache returns only related privileges excluding unrelated
privileges such like "server=server1->database=db_2->action=ALL".
3.2) SENTRY-1291 was to address the problem 2). However, it did not address
the problem 1). And its implementation SimplePrivilegeCache is not memory
efficient (the key of the map contains the whole resource hierarchy, and many
keys share large portion of the same content), nor operational efficient (for
each authorization check, SimplePrivilegeCache .listPrivileges() has to
construct a large amount of keys in order to find all related privileges in a
map).
4) Use TreePrivilegeCache instead of SimplePrivilegeCache for caching. Note,
this solution is built on top of SENTRY-1291, and utilizes the changes
SENTRY-1291 made, such as providing resource hierarchy when getting privileges
for authorization check.
*Major Behavior Change*
1) Create a new Interface FilteredPrivilegeCache, which extends from
PrivilegeCache.
2) Move the function added by SENTRY-1291 in PrivilegeCache to
FilteredPrivilegeCache. Add additional functions in this solution to
FilteredPrivilegeCache. In this way, there is no change in PrivilegeCache, and
we are backward compatible with old implementation before SENTRY-1291.
3) Move all changed in SimplePrivilegeCache (implements PrivilegeCache) from
SENTRY-1291 to a new class SimpleFilteredPrivilegeCache, which implements
FilteredPrivilegeCache.
4) Instead of hard-coding the privilege cache class, use configuration
AuthzConfVars.AUTHZ_PRIVILEGE_CACHE ("sentry.hive.privilege.cache") to specify
the privilege cache class name. The default value is
"org.apache.sentry.provider.cache.TreePrivilegeCache". User can change to
another cache implementation in sentry-site.xml at a service (such as hive
server or HMS). The options are
4.1) org.apache.sentry.provider.cache.SimplePrivilegeCache (the original
cache implementation before SENTRY-1291)
4.2) org.apache.sentry.provider.cache.SimpleFilteredPrivilegeCache (the cache
implemented in SENTRY-1291)
4.3) org.apache.sentry.provider.cache.TreePrivilegeCache (the cache
implemented in this Jira SENTRY-2539)
was:
*Problem*:
Right now, for a command such "show databases", Sentry has to perform
authorization checks on each database. When there are many databases, like
12000 databases in the system, the authorization checks of a single command in
Sentry could be very slow. There
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.010.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch, SENTRY-2539.006.patch, SENTRY-2539.007.patch, > SENTRY-2539.008.patch, SENTRY-2539.008.patch, SENTRY-2539.008.patch, > SENTRY-2539.009.patch, SENTRY-2539.010.patch, SENTRY-2539.010.patch > > > *Problem*: > Right now, for a command such "show databases", Sentry has to perform > authorization checks on each database. When there are many databases, like > 12000 databases in the system, the authorization checks of a single command > in Sentry could be very slow. There are two main factors that slow down > authorization checks in Sentry even when caching is enabled: > 1) Cache returns the list of privileges in the form of String. As a result, > every authorization check has to convert the privilege string to privilege > object. > 2) When cache is enabled, the cache returns all privileges of a given user > regardless what resource to check. > 2.1) for example, a user has 2000 privileges assigned and the resource to > check is "server=server1, database=db_1, table=table_1". The cache returns > all 2000 privileges including unrelated privileges such like > "server=server1->database=db_2->action=ALL". > 2.2) Returning unrelated privileges has two side effects: > 2.2.1) Converting privileges from String to Object overhead is > proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > 2.2.2) Authorization check goes through each privilege, and its overhead > is proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > *Solution*: > 1) Add a new function listPrivilegeObjects that lets authorization provider > get privilege objects when checking the authorization. This avoids the > conversion overhead. All the interfaces from policy engine (PolicyEngine) to > the cache (PrivilegeCache) have to be changed to add this new function. > 2) Implement a new cache TreePrivilegeCache. It converts the privilege from > String format to Privilege object at beginning, and directly return the > privilege objects in listPrivilegeObjects at authorization check. This avoids > the overhead of conversion at each authorization check. > 3) TreePrivilegeCache organizes the privileges based on the resource > hierarchy, like a tree. Therefore, it can return only related privileges > based on the resource to check. This reduces the authorization check > overhead. > 3.1) For example, a user has 2000 privileges assigned, and the resource to > check is "server=server1, database=db_1, table=table_1". the cache > TreePrivilegeCache returns only related privileges excluding unrelated > privileges such like "server=server1->database=db_2->action=ALL". > 3.2) SENTRY-1291 was to address the problem 2). However, it did not address > the problem 1). And its implementation SimplePrivilegeCache is not memory > efficient (the key of the map contains the whole resource hierarchy, and many > keys share large portion of the same content), nor operational efficient (for > each authorization check, SimplePrivilegeCache .listPrivileges() has to > construct a large amount of keys in order to find all related privileges in a > map). > 4) Use TreePrivilegeCache instead of SimplePrivilegeCache for caching. Note, > this solution is built on top of SENTRY-1291, and utilizes the changes > SENTRY-1291 made, such as providing resource hierarchy when getting > privileges for authorization check. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.010.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch, SENTRY-2539.006.patch, SENTRY-2539.007.patch, > SENTRY-2539.008.patch, SENTRY-2539.008.patch, SENTRY-2539.008.patch, > SENTRY-2539.009.patch, SENTRY-2539.010.patch > > > *Problem*: > Right now, for a command such "show databases", Sentry has to perform > authorization checks on each database. When there are many databases, like > 12000 databases in the system, the authorization checks of a single command > in Sentry could be very slow. There are two main factors that slow down > authorization checks in Sentry even when caching is enabled: > 1) Cache returns the list of privileges in the form of String. As a result, > every authorization check has to convert the privilege string to privilege > object. > 2) When cache is enabled, the cache returns all privileges of a given user > regardless what resource to check. > 2.1) for example, a user has 2000 privileges assigned and the resource to > check is "server=server1, database=db_1, table=table_1". The cache returns > all 2000 privileges including unrelated privileges such like > "server=server1->database=db_2->action=ALL". > 2.2) Returning unrelated privileges has two side effects: > 2.2.1) Converting privileges from String to Object overhead is > proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > 2.2.2) Authorization check goes through each privilege, and its overhead > is proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > *Solution*: > 1) Add a new function listPrivilegeObjects that lets authorization provider > get privilege objects when checking the authorization. This avoids the > conversion overhead. All the interfaces from policy engine (PolicyEngine) to > the cache (PrivilegeCache) have to be changed to add this new function. > 2) Implement a new cache TreePrivilegeCache. It converts the privilege from > String format to Privilege object at beginning, and directly return the > privilege objects in listPrivilegeObjects at authorization check. This avoids > the overhead of conversion at each authorization check. > 3) TreePrivilegeCache organizes the privileges based on the resource > hierarchy, like a tree. Therefore, it can return only related privileges > based on the resource to check. This reduces the authorization check > overhead. > 3.1) For example, a user has 2000 privileges assigned, and the resource to > check is "server=server1, database=db_1, table=table_1". the cache > TreePrivilegeCache returns only related privileges excluding unrelated > privileges such like "server=server1->database=db_2->action=ALL". > 3.2) SENTRY-1291 was to address the problem 2). However, it did not address > the problem 1). And its implementation SimplePrivilegeCache is not memory > efficient (the key of the map contains the whole resource hierarchy, and many > keys share large portion of the same content), nor operational efficient (for > each authorization check, SimplePrivilegeCache .listPrivileges() has to > construct a large amount of keys in order to find all related privileges in a > map). > 4) Use TreePrivilegeCache instead of SimplePrivilegeCache for caching. Note, > this solution is built on top of SENTRY-1291, and utilizes the changes > SENTRY-1291 made, such as providing resource hierarchy when getting > privileges for authorization check. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.009.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch, SENTRY-2539.006.patch, SENTRY-2539.007.patch, > SENTRY-2539.008.patch, SENTRY-2539.008.patch, SENTRY-2539.008.patch, > SENTRY-2539.009.patch > > > *Problem*: > Right now, for a command such "show databases", Sentry has to perform > authorization checks on each database. When there are many databases, like > 12000 databases in the system, the authorization checks of a single command > in Sentry could be very slow. There are two main factors that slow down > authorization checks in Sentry even when caching is enabled: > 1) Cache returns the list of privileges in the form of String. As a result, > every authorization check has to convert the privilege string to privilege > object. > 2) When cache is enabled, the cache returns all privileges of a given user > regardless what resource to check. > 2.1) for example, a user has 2000 privileges assigned and the resource to > check is "server=server1, database=db_1, table=table_1". The cache returns > all 2000 privileges including unrelated privileges such like > "server=server1->database=db_2->action=ALL". > 2.2) Returning unrelated privileges has two side effects: > 2.2.1) Converting privileges from String to Object overhead is > proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > 2.2.2) Authorization check goes through each privilege, and its overhead > is proportional to the number of returned privileges from cache. Converting > unrelated privileges cost time, but no benefit. > *Solution*: > 1) Add a new function listPrivilegeObjects that lets authorization provider > get privilege objects when checking the authorization. This avoids the > conversion overhead. All the interfaces from policy engine (PolicyEngine) to > the cache (PrivilegeCache) have to be changed to add this new function. > 2) Implement a new cache TreePrivilegeCache. It converts the privilege from > String format to Privilege object at beginning, and directly return the > privilege objects in listPrivilegeObjects at authorization check. This avoids > the overhead of conversion at each authorization check. > 3) TreePrivilegeCache organizes the privileges based on the resource > hierarchy, like a tree. Therefore, it can return only related privileges > based on the resource to check. This reduces the authorization check > overhead. > 3.1) For example, a user has 2000 privileges assigned, and the resource to > check is "server=server1, database=db_1, table=table_1". the cache > TreePrivilegeCache returns only related privileges excluding unrelated > privileges such like "server=server1->database=db_2->action=ALL". > 3.2) SENTRY-1291 was to address the problem 2). However, it did not address > the problem 1). And its implementation SimplePrivilegeCache is not memory > efficient (the key of the map contains the whole resource hierarchy, and many > keys share large portion of the same content), nor operational efficient (for > each authorization check, SimplePrivilegeCache .listPrivileges() has to > construct a large amount of keys in order to find all related privileges in a > map). > 4) Use TreePrivilegeCache instead of SimplePrivilegeCache for caching. Note, > this solution is built on top of SENTRY-1291, and utilizes the changes > SENTRY-1291 made, such as providing resource hierarchy when getting > privileges for authorization check. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Description: *Problem*: Right now, for a command such "show databases", Sentry has to perform authorization checks on each database. When there are many databases, like 12000 databases in the system, the authorization checks of a single command in Sentry could be very slow. There are two main factors that slow down authorization checks in Sentry even when caching is enabled: 1) Cache returns the list of privileges in the form of String. As a result, every authorization check has to convert the privilege string to privilege object. 2) When cache is enabled, the cache returns all privileges of a given user regardless what resource to check. 2.1) for example, a user has 2000 privileges assigned and the resource to check is "server=server1, database=db_1, table=table_1". The cache returns all 2000 privileges including unrelated privileges such like "server=server1->database=db_2->action=ALL". 2.2) Returning unrelated privileges has two side effects: 2.2.1) Converting privileges from String to Object overhead is proportional to the number of returned privileges from cache. Converting unrelated privileges cost time, but no benefit. 2.2.2) Authorization check goes through each privilege, and its overhead is proportional to the number of returned privileges from cache. Converting unrelated privileges cost time, but no benefit. *Solution*: 1) Add a new function listPrivilegeObjects that lets authorization provider get privilege objects when checking the authorization. This avoids the conversion overhead. All the interfaces from policy engine (PolicyEngine) to the cache (PrivilegeCache) have to be changed to add this new function. 2) Implement a new cache TreePrivilegeCache. It converts the privilege from String format to Privilege object at beginning, and directly return the privilege objects in listPrivilegeObjects at authorization check. This avoids the overhead of conversion at each authorization check. 3) TreePrivilegeCache organizes the privileges based on the resource hierarchy, like a tree. Therefore, it can return only related privileges based on the resource to check. This reduces the authorization check overhead. 3.1) For example, a user has 2000 privileges assigned, and the resource to check is "server=server1, database=db_1, table=table_1". the cache TreePrivilegeCache returns only related privileges excluding unrelated privileges such like "server=server1->database=db_2->action=ALL". 3.2) SENTRY-1291 was to address the problem 2). However, it did not address the problem 1). And its implementation SimplePrivilegeCache is not memory efficient (the key of the map contains the whole resource hierarchy, and many keys share large portion of the same content), nor operational efficient (for each authorization check, SimplePrivilegeCache .listPrivileges() has to construct a large amount of keys in order to find all related privileges in a map). 4) Use TreePrivilegeCache instead of SimplePrivilegeCache for caching. Note, this solution is built on top of SENTRY-1291, and utilizes the changes SENTRY-1291 made, such as providing resource hierarchy when getting privileges for authorization check. was: Right now, for a command such "show databases", Sentry has to perform authorization checks on each database. When there are many databases, like 12000 databases in the system, the authorization checks of a single command in Sentry could be very slow. There are two main factors that slow down authorization checks in Sentry even when caching is enabled: 1) Cache returns the list of privileges in the form of String. As a result, every authorization check has to convert the privilege string to privilege object. 2) When cache is enabled, the cache returns all privileges of a given user regardless what resource to check. 2.1) for example, a user has 2000 privileges assigned and the resource to check is "server=server1, database=db_1, table=table_1". The cache returns all 2000 privileges including unrelated privileges such like "server=server1->database=db_2->action=ALL". 2.2) Returning unrelated privileges has two side effects: 2.2.1) Converting privileges from String to Object overhead is proportional to the number of returned privileges from cache. Converting unrelated privileges cost time, but no benefit. 2.2.2) Authorization check goes through each privilege, and its overhead is proportional to the number of returned privileges from cache. Converting unrelated privileges cost time, but no benefit. Solution: 1) Add a new function listPrivilegeObjects that lets authorization provider get privilege objects when checking the authorization. This avoids the conversion overhead. All the interfaces from policy engine (PolicyEngine) to
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Description: Right now, for a command such "show databases", Sentry has to perform authorization checks on each database. When there are many databases, like 12000 databases in the system, the authorization checks of a single command in Sentry could be very slow. There are two main factors that slow down authorization checks in Sentry even when caching is enabled: 1) Cache returns the list of privileges in the form of String. As a result, every authorization check has to convert the privilege string to privilege object. 2) When cache is enabled, the cache returns all privileges of a given user regardless what resource to check. 2.1) for example, a user has 2000 privileges assigned and the resource to check is "server=server1, database=db_1, table=table_1". The cache returns all 2000 privileges including unrelated privileges such like "server=server1->database=db_2->action=ALL". 2.2) Returning unrelated privileges has two side effects: 2.2.1) Converting privileges from String to Object overhead is proportional to the number of returned privileges from cache. Converting unrelated privileges cost time, but no benefit. 2.2.2) Authorization check goes through each privilege, and its overhead is proportional to the number of returned privileges from cache. Converting unrelated privileges cost time, but no benefit. Solution: 1) Add a new function listPrivilegeObjects that lets authorization provider get privilege objects when checking the authorization. This avoids the conversion overhead. All the interfaces from policy engine (PolicyEngine) to the cache (PrivilegeCache) have to be changed to add this new function. 2) Implement a new cache TreePrivilegeCache. It converts the privilege from String format to Privilege object at beginning, and directly return the privilege objects in listPrivilegeObjects at authorization check. This avoids the overhead of conversion at each authorization check. 3) TreePrivilegeCache organizes the privileges based on the resource hierarchy, like a tree. Therefore, it can return only related privileges based on the resource to check. This reduces the authorization check overhead. 3.1) For example, a user has 2000 privileges assigned, and the resource to check is "server=server1, database=db_1, table=table_1". the cache TreePrivilegeCache returns only related privileges excluding unrelated privileges such like "server=server1->database=db_2->action=ALL". 3.2) SENTRY-1291 was to address the problem 2). However, it did not address the problem 1). And its implementation SimplePrivilegeCache is not memory efficient (the key of the map contains the whole resource hierarchy, and many keys share large portion of the same content), nor operational efficient (for each authorization check, SimplePrivilegeCache .listPrivileges() has to construct a large amount of keys in order to find all related privileges in a map). 4) Use TreePrivilegeCache instead of SimplePrivilegeCache for caching. Note, this solution is built on top of SENTRY-1291, and utilizes the changes SENTRY-1291 made, such as providing resource hierarchy when getting privileges for authorization check. was: Right now, the PolicyEngine Interface only returns the list of privileges in the form of String. As a result, every authorization check has to convert the privilege string to privilege object even though the cached privilege objects are of the correct type already. We should add a new function that returns privilege object directly to avoid the overhead of conversion. > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch, SENTRY-2539.006.patch, SENTRY-2539.007.patch, > SENTRY-2539.008.patch, SENTRY-2539.008.patch, SENTRY-2539.008.patch > > > Right now, for a command such "show databases", Sentry has to perform > authorization checks on each database. When there are many databases, like > 12000 databases in the system, the authorization checks of a single command > in Sentry could be very slow. There are two main factors that slow down > authorization checks in Sentry even when caching is enabled: > 1) Cache returns the list of privileges in the form of String. As a result, > every authorization check has to convert the privilege string to privilege > object. > 2) When cache is enabled, the c
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.008.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch, SENTRY-2539.006.patch, SENTRY-2539.007.patch, > SENTRY-2539.008.patch, SENTRY-2539.008.patch, SENTRY-2539.008.patch > > > Right now, the PolicyEngine Interface only returns the list of privileges in > the form of String. As a result, every authorization check has to convert the > privilege string to privilege object even though the cached privilege objects > are of the correct type already. > We should add a new function that returns privilege object directly to avoid > the overhead of conversion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.008.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch, SENTRY-2539.006.patch, SENTRY-2539.007.patch, > SENTRY-2539.008.patch, SENTRY-2539.008.patch > > > Right now, the PolicyEngine Interface only returns the list of privileges in > the form of String. As a result, every authorization check has to convert the > privilege string to privilege object even though the cached privilege objects > are of the correct type already. > We should add a new function that returns privilege object directly to avoid > the overhead of conversion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.008.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch, SENTRY-2539.006.patch, SENTRY-2539.007.patch, > SENTRY-2539.008.patch > > > Right now, the PolicyEngine Interface only returns the list of privileges in > the form of String. As a result, every authorization check has to convert the > privilege string to privilege object even though the cached privilege objects > are of the correct type already. > We should add a new function that returns privilege object directly to avoid > the overhead of conversion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.007.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch, SENTRY-2539.006.patch, SENTRY-2539.007.patch > > > Right now, the PolicyEngine Interface only returns the list of privileges in > the form of String. As a result, every authorization check has to convert the > privilege string to privilege object even though the cached privilege objects > are of the correct type already. > We should add a new function that returns privilege object directly to avoid > the overhead of conversion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.006.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch, SENTRY-2539.006.patch > > > Right now, the PolicyEngine Interface only returns the list of privileges in > the form of String. As a result, every authorization check has to convert the > privilege string to privilege object even though the cached privilege objects > are of the correct type already. > We should add a new function that returns privilege object directly to avoid > the overhead of conversion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.005.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch, > SENTRY-2539.005.patch > > > Right now, the PolicyEngine Interface only returns the list of privileges in > the form of String. As a result, every authorization check has to convert the > privilege string to privilege object even though the cached privilege objects > are of the correct type already. > We should add a new function that returns privilege object directly to avoid > the overhead of conversion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.003.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch, SENTRY-2539.003.patch > > > Right now, the PolicyEngine Interface only returns the list of privileges in > the form of String. As a result, every authorization check has to convert the > privilege string to privilege object even though the cached privilege objects > are of the correct type already. > We should add a new function that returns privilege object directly to avoid > the overhead of conversion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Status: Patch Available (was: Open) > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch > > > Right now, the PolicyEngine Interface only returns the list of privileges in > the form of String. As a result, every authorization check has to convert the > privilege string to privilege object even though the cached privilege objects > are of the correct type already. > We should add a new function that returns privilege object directly to avoid > the overhead of conversion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2539) PolicyEngine should be able to return privilege directly
[ https://issues.apache.org/jira/browse/SENTRY-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2539: -- Attachment: SENTRY-2539.002.patch > PolicyEngine should be able to return privilege directly > - > > Key: SENTRY-2539 > URL: https://issues.apache.org/jira/browse/SENTRY-2539 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2539.002.patch > > > Right now, the PolicyEngine Interface only returns the list of privileges in > the form of String. As a result, every authorization check has to convert the > privilege string to privilege object even though the cached privilege objects > are of the correct type already. > We should add a new function that returns privilege object directly to avoid > the overhead of conversion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2540) Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command based on configuration
[ https://issues.apache.org/jira/browse/SENTRY-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2540: -- Attachment: SENTRY-2540.003.patch > Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command > based on configuration > --- > > Key: SENTRY-2540 > URL: https://issues.apache.org/jira/browse/SENTRY-2540 > Project: Sentry > Issue Type: Improvement >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2540.001.patch, SENTRY-2540.002.patch, > SENTRY-2540.002.patch, SENTRY-2540.003.patch, SENTRY-2540.003.patch, > SENTRY-2540.003.patch, SENTRY-2540.003.patch, SENTRY-2540.003.patch > > > When there are thousands of databases, SHOW DATABASES may take a really long > time because SENTRY checks if user has any of the following privileges on > that database for filtering out the database > DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, > DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, > DBModelAction.LOCK > To speedup the authorization checking for this case, Sentry can check only > the select privilege for SHOW DATABASES and SHOW TABLES based on > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2540) Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command based on configuration
[ https://issues.apache.org/jira/browse/SENTRY-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2540: -- Attachment: SENTRY-2540.003.patch > Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command > based on configuration > --- > > Key: SENTRY-2540 > URL: https://issues.apache.org/jira/browse/SENTRY-2540 > Project: Sentry > Issue Type: Improvement >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2540.001.patch, SENTRY-2540.002.patch, > SENTRY-2540.002.patch, SENTRY-2540.003.patch, SENTRY-2540.003.patch, > SENTRY-2540.003.patch, SENTRY-2540.003.patch > > > When there are thousands of databases, SHOW DATABASES may take a really long > time because SENTRY checks if user has any of the following privileges on > that database for filtering out the database > DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, > DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, > DBModelAction.LOCK > To speedup the authorization checking for this case, Sentry can check only > the select privilege for SHOW DATABASES and SHOW TABLES based on > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2540) Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command based on configuration
[ https://issues.apache.org/jira/browse/SENTRY-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2540: -- Attachment: SENTRY-2540.003.patch > Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command > based on configuration > --- > > Key: SENTRY-2540 > URL: https://issues.apache.org/jira/browse/SENTRY-2540 > Project: Sentry > Issue Type: Improvement >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2540.001.patch, SENTRY-2540.002.patch, > SENTRY-2540.002.patch, SENTRY-2540.003.patch, SENTRY-2540.003.patch, > SENTRY-2540.003.patch > > > When there are thousands of databases, SHOW DATABASES may take a really long > time because SENTRY checks if user has any of the following privileges on > that database for filtering out the database > DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, > DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, > DBModelAction.LOCK > To speedup the authorization checking for this case, Sentry can check only > the select privilege for SHOW DATABASES and SHOW TABLES based on > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2540) Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command based on configuration
[ https://issues.apache.org/jira/browse/SENTRY-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2540: -- Attachment: SENTRY-2540.003.patch > Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command > based on configuration > --- > > Key: SENTRY-2540 > URL: https://issues.apache.org/jira/browse/SENTRY-2540 > Project: Sentry > Issue Type: Improvement >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2540.001.patch, SENTRY-2540.002.patch, > SENTRY-2540.002.patch, SENTRY-2540.003.patch, SENTRY-2540.003.patch > > > When there are thousands of databases, SHOW DATABASES may take a really long > time because SENTRY checks if user has any of the following privileges on > that database for filtering out the database > DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, > DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, > DBModelAction.LOCK > To speedup the authorization checking for this case, Sentry can check only > the select privilege for SHOW DATABASES and SHOW TABLES based on > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2540) Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command based on configuration
[ https://issues.apache.org/jira/browse/SENTRY-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2540: -- Attachment: SENTRY-2540.003.patch > Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command > based on configuration > --- > > Key: SENTRY-2540 > URL: https://issues.apache.org/jira/browse/SENTRY-2540 > Project: Sentry > Issue Type: Improvement >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2540.001.patch, SENTRY-2540.002.patch, > SENTRY-2540.002.patch, SENTRY-2540.003.patch > > > When there are thousands of databases, SHOW DATABASES may take a really long > time because SENTRY checks if user has any of the following privileges on > that database for filtering out the database > DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, > DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, > DBModelAction.LOCK > To speedup the authorization checking for this case, Sentry can check only > the select privilege for SHOW DATABASES and SHOW TABLES based on > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2540) Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command based on configuration
[ https://issues.apache.org/jira/browse/SENTRY-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2540: -- Attachment: SENTRY-2540.002.patch > Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command > based on configuration > --- > > Key: SENTRY-2540 > URL: https://issues.apache.org/jira/browse/SENTRY-2540 > Project: Sentry > Issue Type: Improvement >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2540.001.patch, SENTRY-2540.002.patch, > SENTRY-2540.002.patch > > > When there are thousands of databases, SHOW DATABASES may take a really long > time because SENTRY checks if user has any of the following privileges on > that database for filtering out the database > DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, > DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, > DBModelAction.LOCK > To speedup the authorization checking for this case, Sentry can check only > the select privilege for SHOW DATABASES and SHOW TABLES based on > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2540) Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command based on configuration
[ https://issues.apache.org/jira/browse/SENTRY-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2540: -- Attachment: SENTRY-2540.002.patch > Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command > based on configuration > --- > > Key: SENTRY-2540 > URL: https://issues.apache.org/jira/browse/SENTRY-2540 > Project: Sentry > Issue Type: Improvement >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2540.001.patch, SENTRY-2540.002.patch > > > When there are thousands of databases, SHOW DATABASES may take a really long > time because SENTRY checks if user has any of the following privileges on > that database for filtering out the database > DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, > DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, > DBModelAction.LOCK > To speedup the authorization checking for this case, Sentry can check only > the select privilege for SHOW DATABASES and SHOW TABLES based on > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2540) Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command based on configuration
[ https://issues.apache.org/jira/browse/SENTRY-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2540: -- Status: Patch Available (was: Open) > Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command > based on configuration > --- > > Key: SENTRY-2540 > URL: https://issues.apache.org/jira/browse/SENTRY-2540 > Project: Sentry > Issue Type: Improvement >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2540.001.patch > > > When there are thousands of databases, SHOW DATABASES may take a really long > time because SENTRY checks if user has any of the following privileges on > that database for filtering out the database > DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, > DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, > DBModelAction.LOCK > To speedup the authorization checking for this case, Sentry can check only > the select privilege for SHOW DATABASES and SHOW TABLES based on > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2540) Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command based on configuration
[ https://issues.apache.org/jira/browse/SENTRY-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2540: -- Attachment: SENTRY-2540.001.patch > Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command > based on configuration > --- > > Key: SENTRY-2540 > URL: https://issues.apache.org/jira/browse/SENTRY-2540 > Project: Sentry > Issue Type: Improvement >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2540.001.patch > > > When there are thousands of databases, SHOW DATABASES may take a really long > time because SENTRY checks if user has any of the following privileges on > that database for filtering out the database > DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, > DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, > DBModelAction.LOCK > To speedup the authorization checking for this case, Sentry can check only > the select privilege for SHOW DATABASES and SHOW TABLES based on > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2540) Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command based on configuration
[ https://issues.apache.org/jira/browse/SENTRY-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2540: -- Summary: Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command based on configuration (was: Only use SELECT action for filter SHOW DATABASES command based on configuration) > Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command > based on configuration > --- > > Key: SENTRY-2540 > URL: https://issues.apache.org/jira/browse/SENTRY-2540 > Project: Sentry > Issue Type: Improvement >Reporter: Na Li >Assignee: Na Li >Priority: Major > > When there are thousands of databases, SHOW DATABASES may take a really long > time because SENTRY checks if user has any of the following privileges on > that database for filtering out the database > DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, > DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, > DBModelAction.LOCK > To speedup the authorization checking for this case, Sentry can check only > the select privilege for SHOW DATABASES based on configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2540) Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command based on configuration
[ https://issues.apache.org/jira/browse/SENTRY-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2540: -- Description: When there are thousands of databases, SHOW DATABASES may take a really long time because SENTRY checks if user has any of the following privileges on that database for filtering out the database DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, DBModelAction.LOCK To speedup the authorization checking for this case, Sentry can check only the select privilege for SHOW DATABASES and SHOW TABLES based on configuration. was: When there are thousands of databases, SHOW DATABASES may take a really long time because SENTRY checks if user has any of the following privileges on that database for filtering out the database DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, DBModelAction.LOCK To speedup the authorization checking for this case, Sentry can check only the select privilege for SHOW DATABASES based on configuration. > Only use SELECT action for filter SHOW DATABASES and SHOW TABLES command > based on configuration > --- > > Key: SENTRY-2540 > URL: https://issues.apache.org/jira/browse/SENTRY-2540 > Project: Sentry > Issue Type: Improvement >Reporter: Na Li >Assignee: Na Li >Priority: Major > > When there are thousands of databases, SHOW DATABASES may take a really long > time because SENTRY checks if user has any of the following privileges on > that database for filtering out the database > DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, > DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, > DBModelAction.LOCK > To speedup the authorization checking for this case, Sentry can check only > the select privilege for SHOW DATABASES and SHOW TABLES based on > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (SENTRY-2540) Only use SELECT action for filter SHOW DATABASES command based on configuration
[ https://issues.apache.org/jira/browse/SENTRY-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-2540: - Assignee: Na Li > Only use SELECT action for filter SHOW DATABASES command based on > configuration > --- > > Key: SENTRY-2540 > URL: https://issues.apache.org/jira/browse/SENTRY-2540 > Project: Sentry > Issue Type: Improvement >Reporter: Na Li >Assignee: Na Li >Priority: Major > > When there are thousands of databases, SHOW DATABASES may take a really long > time because SENTRY checks if user has any of the following privileges on > that database for filtering out the database > DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, > DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, > DBModelAction.LOCK > To speedup the authorization checking for this case, Sentry can check only > the select privilege for SHOW DATABASES based on configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (SENTRY-2540) Only use SELECT action for filter SHOW DATABASES command based on configuration
Na Li created SENTRY-2540: - Summary: Only use SELECT action for filter SHOW DATABASES command based on configuration Key: SENTRY-2540 URL: https://issues.apache.org/jira/browse/SENTRY-2540 Project: Sentry Issue Type: Improvement Reporter: Na Li When there are thousands of databases, SHOW DATABASES may take a really long time because SENTRY checks if user has any of the following privileges on that database for filtering out the database DBModelAction.SELECT, DBModelAction.INSERT, DBModelAction.ALTER, DBModelAction.CREATE, DBModelAction.DROP, DBModelAction.INDEX, DBModelAction.LOCK To speedup the authorization checking for this case, Sentry can check only the select privilege for SHOW DATABASES based on configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (SENTRY-2539) PolicyEngine should be able to return privilege directly
Na Li created SENTRY-2539: - Summary: PolicyEngine should be able to return privilege directly Key: SENTRY-2539 URL: https://issues.apache.org/jira/browse/SENTRY-2539 Project: Sentry Issue Type: Improvement Components: Sentry Affects Versions: 2.1 Reporter: Na Li Assignee: Na Li Right now, the PolicyEngine Interface only returns the list of privileges in the form of String. As a result, every authorization check has to convert the privilege string to privilege object even though the cached privilege objects are of the correct type already. We should add a new function that returns privilege object directly to avoid the overhead of conversion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SENTRY-2538) consecutiveUpdateFailuresCount is not reset
[
https://issues.apache.org/jira/browse/SENTRY-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2538:
--
Attachment: SENTRY-2538.001.patch
> consecutiveUpdateFailuresCount is not reset
> ---
>
> Key: SENTRY-2538
> URL: https://issues.apache.org/jira/browse/SENTRY-2538
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Reporter: László Dénes Terjéki
>Assignee: László Dénes Terjéki
>Priority: Major
> Attachments:
> 0001-SENTRY-2538-reset-consecutiveUpdateFailuresCount-on-.patch,
> SENTRY-2538.001.patch
>
>
> *consecutiveUpdateFailuresCount* reset only happens when an error occurs and
> the threshold is reached. Also as a side affect the log always contains the
> following message
> {code:java}
> ERROR org.apache.sentry.provider.db.generic.UpdatableCache: Failed to update
> roles and privileges cache for 0 times. Revoking all privileges from cache,
> which will cause all authorization requests to fail.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2538) consecutiveUpdateFailuresCount is not reset
[
https://issues.apache.org/jira/browse/SENTRY-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2538:
--
Assignee: Na Li
Status: Patch Available (was: Open)
> consecutiveUpdateFailuresCount is not reset
> ---
>
> Key: SENTRY-2538
> URL: https://issues.apache.org/jira/browse/SENTRY-2538
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Reporter: László Dénes Terjéki
>Assignee: Na Li
>Priority: Major
> Attachments:
> 0001-SENTRY-2538-reset-consecutiveUpdateFailuresCount-on-.patch
>
>
> *consecutiveUpdateFailuresCount* reset only happens when an error occurs and
> the threshold is reached. Also as a side affect the log always contains the
> following message
> {code:java}
> ERROR org.apache.sentry.provider.db.generic.UpdatableCache: Failed to update
> roles and privileges cache for 0 times. Revoking all privileges from cache,
> which will cause all authorization requests to fail.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2535) SentryKafkaAuthorizer throws Exception when describing ACLs
[
https://issues.apache.org/jira/browse/SENTRY-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2535:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
[~gwilder] Thanks for your contributions!
> SentryKafkaAuthorizer throws Exception when describing ACLs
> ---
>
> Key: SENTRY-2535
> URL: https://issues.apache.org/jira/browse/SENTRY-2535
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Reporter: Gergo Wilder
>Assignee: Gergo Wilder
>Priority: Major
> Attachments: SENTRY-2535.001.patch, SENTRY-2535.002.patch,
> SENTRY-2535.002.patch
>
>
> ConsumerGroup resource type is not handled properly in SentryKafkaAuthorizer
> causing an exception when converting from Sentry consumer group resource type
> to Kafka's consumer group resource type.
> {noformat}
> 2019-10-25 12:30:17,992 ERROR kafka.server.KafkaApis: [KafkaApi-6] Error
> when handling request: clientId=adminclient-83, correlationId=1158,
> api=DESCRIBE_ACLS,
> body={resource_type=2,resource_name=null,resource_pattern_type_filter=1,principal=null,host=null,operation=1,permission_type=1}
> kafka.common.KafkaException: CONSUMERGROUP not a valid resourceType name. The
> valid names are Topic,Group,Cluster,TransactionalId,DelegationToken
> at
> kafka.security.auth.ResourceType$$anonfun$fromString$1.apply(ResourceType.scala:64)
> at
> kafka.security.auth.ResourceType$$anonfun$fromString$1.apply(ResourceType.scala:64)
> at scala.Option.getOrElse(Option.scala:121)
> at kafka.security.auth.ResourceType$.fromString(ResourceType.scala:64)
> at
> org.apache.sentry.kafka.binding.KafkaAuthBinding.rolePrivilegesToResourceAcls(KafkaAuthBinding.java:481)
> at
> org.apache.sentry.kafka.binding.KafkaAuthBinding.getAclsForRoles(KafkaAuthBinding.java:463)
> at
> org.apache.sentry.kafka.binding.KafkaAuthBinding.getAcls(KafkaAuthBinding.java:379)
> at
> org.apache.sentry.kafka.authorizer.SentryKafkaAuthorizer.getAcls(SentryKafkaAuthorizer.java:93)
> at kafka.server.KafkaApis.handleDescribeAcls(KafkaApis.scala:1911)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:137)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:74)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SENTRY-2535) SentryKafkaAuthorizer throws Exception when describing ACLs
[
https://issues.apache.org/jira/browse/SENTRY-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2535:
--
Attachment: SENTRY-2535.002.patch
> SentryKafkaAuthorizer throws Exception when describing ACLs
> ---
>
> Key: SENTRY-2535
> URL: https://issues.apache.org/jira/browse/SENTRY-2535
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Reporter: Gergo Wilder
>Assignee: Gergo Wilder
>Priority: Major
> Attachments: SENTRY-2535.001.patch, SENTRY-2535.002.patch,
> SENTRY-2535.002.patch
>
>
> ConsumerGroup resource type is not handled properly in SentryKafkaAuthorizer
> causing an exception when converting from Sentry consumer group resource type
> to Kafka's consumer group resource type.
> {noformat}
> 2019-10-25 12:30:17,992 ERROR kafka.server.KafkaApis: [KafkaApi-6] Error
> when handling request: clientId=adminclient-83, correlationId=1158,
> api=DESCRIBE_ACLS,
> body={resource_type=2,resource_name=null,resource_pattern_type_filter=1,principal=null,host=null,operation=1,permission_type=1}
> kafka.common.KafkaException: CONSUMERGROUP not a valid resourceType name. The
> valid names are Topic,Group,Cluster,TransactionalId,DelegationToken
> at
> kafka.security.auth.ResourceType$$anonfun$fromString$1.apply(ResourceType.scala:64)
> at
> kafka.security.auth.ResourceType$$anonfun$fromString$1.apply(ResourceType.scala:64)
> at scala.Option.getOrElse(Option.scala:121)
> at kafka.security.auth.ResourceType$.fromString(ResourceType.scala:64)
> at
> org.apache.sentry.kafka.binding.KafkaAuthBinding.rolePrivilegesToResourceAcls(KafkaAuthBinding.java:481)
> at
> org.apache.sentry.kafka.binding.KafkaAuthBinding.getAclsForRoles(KafkaAuthBinding.java:463)
> at
> org.apache.sentry.kafka.binding.KafkaAuthBinding.getAcls(KafkaAuthBinding.java:379)
> at
> org.apache.sentry.kafka.authorizer.SentryKafkaAuthorizer.getAcls(SentryKafkaAuthorizer.java:93)
> at kafka.server.KafkaApis.handleDescribeAcls(KafkaApis.scala:1911)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:137)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:74)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (SENTRY-1778) Close query as soon as it is done
[ https://issues.apache.org/jira/browse/SENTRY-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-1778: - Assignee: (was: Na Li) > Close query as soon as it is done > - > > Key: SENTRY-1778 > URL: https://issues.apache.org/jira/browse/SENTRY-1778 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 1.8.0, 2.0.0 >Reporter: Na Li >Priority: Minor > Labels: bite-sized, newbie > > Make sure we close all query results after we have finished with them. > Failure to do so will result in significant memory leaks in your application. > mentioned in > http://www.datanucleus.org/products/accessplatform_4_1/jdo/performance_tuning.html > Since we always close the transaction, and closing transaction will close the > query, the memory leak without this change should not be significant. This is > why this issue is not high priority. On the other hand, closing query result > sooner should not have side effect. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-1731) java net BindException Address already in use
[ https://issues.apache.org/jira/browse/SENTRY-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-1731: - Assignee: (was: Na Li) > java net BindException Address already in use > - > > Key: SENTRY-1731 > URL: https://issues.apache.org/jira/browse/SENTRY-1731 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Priority: Major > Attachments: Sentry-1687 Patch1 Test fail build 2570 Address already > in use.zip > > > this unit test fails often. > java.net.BindException: Address already in use -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-1914) Update field size in Sentry to Match Hive definition
[ https://issues.apache.org/jira/browse/SENTRY-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-1914: - Assignee: (was: Na Li) > Update field size in Sentry to Match Hive definition > > > Key: SENTRY-1914 > URL: https://issues.apache.org/jira/browse/SENTRY-1914 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Priority: Major > Attachments: SENTRY-1914.001.patch > > > Hive defines some fields in a bigger size than sentry. When those fields have > value longer than what sentry defines, sentry will not be able to persist > those fields correctly. > Hive defined size in sql > `TABLE_NAME` VARCHAR(256) > `COLUMN_NAME` VARCHAR(767) > in > https://github.com/apache/hive/blob/master/metastore/scripts/upgrade/mysql/hive-schema-2.2.0.mysql.sql > table `TAB_COL_STATS` and `PART_COL_STATS` > Sentry defined size in sql > `TABLE_NAME` VARCHAR(128) > `COLUMN_NAME` VARCHAR(128) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-1961) Optimize memory usage for SimplePrivilegeCache
[ https://issues.apache.org/jira/browse/SENTRY-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-1961: - Assignee: (was: Na Li) > Optimize memory usage for SimplePrivilegeCache > --- > > Key: SENTRY-1961 > URL: https://issues.apache.org/jira/browse/SENTRY-1961 > Project: Sentry > Issue Type: Task > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Priority: Major > Attachments: SENTRY-1961.001.patch > > > The implementation of SimplePrivilegeCache and in particular the 2 > map> fields, cachedAuthzPrivileges and wildCardAuthz is > not memory efficient. The strings stored in these structures can be quite > long e.g. (server=server1->database=b1, > (server=server1->database=b1->action=insert)). For really big installations > with lots of entities (roles and privileges) these fields may end up taking > some non-substantial amount of memory. > We should come up with of a more efficient memory representation to implement > PrivilegeCache -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-1969) hive-authz2 makes two sentry permission requests for CREATE TABLE and CREATE/DROP FUNCTION statements
[ https://issues.apache.org/jira/browse/SENTRY-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-1969: - Assignee: (was: Na Li) > hive-authz2 makes two sentry permission requests for CREATE TABLE and > CREATE/DROP FUNCTION statements > - > > Key: SENTRY-1969 > URL: https://issues.apache.org/jira/browse/SENTRY-1969 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Sergio Peña >Priority: Major > > When Hive compiles a CREATE TABLE or CREATE/DROP FUNCTION operation, it > requests authorization to Sentry twice: > One when calling HiveAuthzBindingHookV2.postAnalyse() and another when > calling the V2 > authorization API SentryHiveAuthorizer.checkPrivileges(). > Perhaps this code was left by accident as the postAnalyse() is the old way to > request > authorization. SentryHiveAuthozer should be used only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-1971) Hive integration for auth-2 should handle creating function properly
[ https://issues.apache.org/jira/browse/SENTRY-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864351#comment-16864351 ] Na Li commented on SENTRY-1971: --- [~arjunmishra13] Does this issue still exist? > Hive integration for auth-2 should handle creating function properly > > > Key: SENTRY-1971 > URL: https://issues.apache.org/jira/browse/SENTRY-1971 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Priority: Critical > > Sergio found hive does not include UDF class name for creating function > command as input to sentry. That will break the function authorization. > Once HIVE-17544 is fixed, sentry should change code accordingly to make > authorization for creating function work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-1971) Hive integration for auth-2 should handle creating function properly
[ https://issues.apache.org/jira/browse/SENTRY-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-1971: - Assignee: (was: Na Li) > Hive integration for auth-2 should handle creating function properly > > > Key: SENTRY-1971 > URL: https://issues.apache.org/jira/browse/SENTRY-1971 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Priority: Critical > > Sergio found hive does not include UDF class name for creating function > command as input to sentry. That will break the function authorization. > Once HIVE-17544 is fixed, sentry should change code accordingly to make > authorization for creating function work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-1957) Do not parse hive sql string in Sentry
[ https://issues.apache.org/jira/browse/SENTRY-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-1957: - Assignee: (was: Na Li) > Do not parse hive sql string in Sentry > -- > > Key: SENTRY-1957 > URL: https://issues.apache.org/jira/browse/SENTRY-1957 > Project: Sentry > Issue Type: Task > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Priority: Major > Attachments: SENTRY-1957.002.patch, SENTRY-1957.003.patch > > > In current authorization 2 API, Hive does not provide parsed sql string to > Sentry. So sentry parses the sql string to get database name and table name. > Once the API changes to include the parsed sql string, sentry should remove > the code that parses the sql string in SimpleSemanticAnalyzer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-1954) Support User level privileges for Sentry HA
[ https://issues.apache.org/jira/browse/SENTRY-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-1954: - Assignee: (was: Na Li) > Support User level privileges for Sentry HA > --- > > Key: SENTRY-1954 > URL: https://issues.apache.org/jira/browse/SENTRY-1954 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Alexander Kolbasov >Priority: Major > Labels: sentry-ha > > All the Sentry HA work pretty much ignored user-level privileges and most > likely they do not work correctly in 2.0. This should be investigated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-1648) Sentry MySQL "Unique" Index
[
https://issues.apache.org/jira/browse/SENTRY-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li reassigned SENTRY-1648:
-
Assignee: (was: Na Li)
> Sentry MySQL "Unique" Index
> ---
>
> Key: SENTRY-1648
> URL: https://issues.apache.org/jira/browse/SENTRY-1648
> Project: Sentry
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: David Mollitor
>Priority: Minor
>
> {code:sql|title=sentry-mysql-1.8.0.sql}
> CREATE TABLE `SENTRY_DB_PRIVILEGE` (
> `DB_PRIVILEGE_ID` BIGINT NOT NULL,
> `PRIVILEGE_SCOPE` VARCHAR(32) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `SERVER_NAME` VARCHAR(128) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `DB_NAME` VARCHAR(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT
> '__NULL__',
> `TABLE_NAME` VARCHAR(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT
> '__NULL__',
> `COLUMN_NAME` VARCHAR(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT
> '__NULL__',
> `URI` VARCHAR(4000) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT '__NULL__',
> `ACTION` VARCHAR(128) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `CREATE_TIME` BIGINT NOT NULL,
> `WITH_GRANT_OPTION` CHAR(1) NOT NULL
> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
> ALTER TABLE `SENTRY_DB_PRIVILEGE`
> ADD UNIQUE `SENTRY_DB_PRIV_PRIV_NAME_UNIQ`
> (`SERVER_NAME`,`DB_NAME`,`TABLE_NAME`,`COLUMN_NAME`,`URI`(250),`ACTION`,`WITH_GRANT_OPTION`);
> {code}
> As you can see, only the first 250 characters of URI is considered when
> determining "uniqueness". Typically, a second column would be added
> containing the hash value (MD5/SHA) of the URI and that column would instead
> be used as part of the unique index instead of the field itself.
> Oracle would also benefit from this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (SENTRY-2091) User-based Privilege is broken by SENTRY-769
[
https://issues.apache.org/jira/browse/SENTRY-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li reassigned SENTRY-2091:
-
Assignee: (was: Na Li)
> User-based Privilege is broken by SENTRY-769
>
>
> Key: SENTRY-2091
> URL: https://issues.apache.org/jira/browse/SENTRY-2091
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Priority: Major
> Attachments: SENTRY-2091.001.patch, SENTRY-2091.002.patch,
> SENTRY-2091.003.patch, SENTRY-2091.004.patch, SENTRY-2091.004.patch,
> SENTRY-2091.006.patch
>
>
> SENTRY-769 throws exception when a user has no group. This breaks user-based
> privilege as the exception prevents getting privilege using user-based
> privilege.
> For example, in the following code
> {code}
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> authProvider.getGroupMapping().getGroups(userName),
> Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> {code}
> when user has no group, the exception causes the processing stops even when
> user has privilege.
> The solution is to catch the exception, and continue the processing.
> {code}
> try {
> Set groups = null;
> try {
> groups = authProvider.getGroupMapping().getGroups(userName)
> } catch (SentryGroupNotFoundException ex) {
> log.debug(...);
> groups = new HashSet();
> }
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> groups, Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (SENTRY-2129) User based privilege
[
https://issues.apache.org/jira/browse/SENTRY-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li reassigned SENTRY-2129:
-
Assignee: (was: Na Li)
> User based privilege
>
>
> Key: SENTRY-2129
> URL: https://issues.apache.org/jira/browse/SENTRY-2129
> Project: Sentry
> Issue Type: New Feature
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Priority: Major
> Labels: roadmap
>
> It’s standard in traditional database security to allow both groups and users
> to be assigned to roles. And hive supports to grant role to user.
> So the following command should be supported in sentry:
> GRANT role_name TO USER user
> The feature implemented in SENTRY-711 is not complete. We complete this
> feature
>
> The current user-based privilege missed some items:
>
> * Sentry policy has two service API: SentryPolicyService and
> SentryGenericPolicyService. The current implementation does not support
> user-based privilege for SentryGenericPolicyService
> * {color:#5c5c5c}Fix bug. SENTRY-2091: User-based Privilege is broken by
> SENTRY-769. The patch is available for review.{color}
> * {color:#5c5c5c}Name Node need change to generate ACL using user
> privilege.{color}
> ** The full snapshot update only contains authorization to roles mapping and
> role to group mapping. *Need to add role to user mapping in*
> SentryStore.retrieveFullRoleImageCore
> ** The delta updates are taken from table SENTRY_PERM_CHANGE, which does not
> distinguish group based permission or user based permission. No change is
> needed
> ** The user changes to a role is not included when sending delta update from
> Sentry to NN. *Need to add AddUsers and DropUsers in TRoleChanges*.
> ** Sentry only create ACL for group with ACL type as AclEntryType.GROUP.
> *Need to add code to create ACL with type as* AclEntryType.USER
> *** SentryINodeAttributesProvider.checkPermission ->
> FSPermissionChecker.checkPermission ->
> SentryINodeAttributesProvider.getAclFeature ->
> SentryAuthorizationInfo.getAclEntries -> SentryPermissions.constructAclEntry
> * {color:#5c5c5c}SentryStore.grantOptionCheck() has to be changed to find
> user level privilege. {color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (SENTRY-2132) Support assigning role to user at SentryGenericPolicyService Interface
[ https://issues.apache.org/jira/browse/SENTRY-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-2132: - Assignee: (was: Na Li) > Support assigning role to user at SentryGenericPolicyService Interface > -- > > Key: SENTRY-2132 > URL: https://issues.apache.org/jira/browse/SENTRY-2132 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Priority: Major > > Right now, Implementation of the interface SentryGenericPolicyService does > not support assigning role to user, nor find roles assigned directly to user. > Need to update to support that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-1296) Flaky test: TestPrivilegeOperatePersistence.testGrantPrivilegeExternalComponentInvalidConf
[ https://issues.apache.org/jira/browse/SENTRY-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-1296: - Assignee: (was: Na Li) > Flaky test: > TestPrivilegeOperatePersistence.testGrantPrivilegeExternalComponentInvalidConf > -- > > Key: SENTRY-1296 > URL: https://issues.apache.org/jira/browse/SENTRY-1296 > Project: Sentry > Issue Type: Test >Reporter: Sravya Tirukkovalur >Priority: Blocker > Fix For: 2.0.0 > > > Seems like above test is failing very frequently in the past few days. > output from log: > org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence.testGrantPrivilegeExternalComponentInvalidConf > Failing for the past 1 build (Since Failed#2613 ) > Took 1 sec. > Error Message > Expected exception: java.lang.Exception > Stacktrace > java.lang.AssertionError: Expected exception: java.lang.Exception > Standard Output -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-2130) Create design document for user based privilege
[ https://issues.apache.org/jira/browse/SENTRY-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-2130: - Assignee: (was: Na Li) > Create design document for user based privilege > --- > > Key: SENTRY-2130 > URL: https://issues.apache.org/jira/browse/SENTRY-2130 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Priority: Major > > The design document should list the detailed behavior for user based > privilege, the components it will update, and how to test it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-2221) Speedup DDL operation when it is blocked by fetching HMS Notification
[ https://issues.apache.org/jira/browse/SENTRY-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-2221: - Assignee: (was: Na Li) > Speedup DDL operation when it is blocked by fetching HMS Notification > - > > Key: SENTRY-2221 > URL: https://issues.apache.org/jira/browse/SENTRY-2221 > Project: Sentry > Issue Type: Improvement >Reporter: Na Li >Priority: Major > > Right now, after HMS completes a DDL operation, it calls Sentry > SentrySyncHMSNotificationsPostEventListener to make sure that sentry fetches > the notification before HMS returns. > The HMS threads blocks until HMSFollower threads fetches the notification and > wakes up the waiter, which happens once every half a second (by default). > When there is a single Sentry server, the delay could be around 0.5 second. > With Sentry HA, the delay could be up to one second because Sentry leader > takes 0.5 second to get the notification and save into sentry DB, and another > Sentry server takes 0.5 second to reads from sentry DB. > To speedup the HMS DDL and reduce the delay, Sentry can try to wake up the > waiter in HMS thread, and then wait on it in > SentryPolicyStoreProcessor.sentry_sync_notifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-1949) Old full snapshots are never cleaned up
[ https://issues.apache.org/jira/browse/SENTRY-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-1949: - Assignee: (was: Na Li) > Old full snapshots are never cleaned up > --- > > Key: SENTRY-1949 > URL: https://issues.apache.org/jira/browse/SENTRY-1949 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Alexander Kolbasov >Priority: Major > Attachments: SENTRY-1949.001.patch, SENTRY-1949.002.patch, > SENTRY-1949.003.patch, SENTRY-1949.004.patch, SENTRY-1949.1.patch > > > We never clean old snapshots and they stay forever, hurting performance as > more and more are accumulated. > [~vamsee] FYI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-2268) Review the required privileges for DDL commands
[
https://issues.apache.org/jira/browse/SENTRY-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li reassigned SENTRY-2268:
-
Assignee: (was: Na Li)
> Review the required privileges for DDL commands
> ---
>
> Key: SENTRY-2268
> URL: https://issues.apache.org/jira/browse/SENTRY-2268
> Project: Sentry
> Issue Type: Task
>Reporter: Na Li
>Priority: Major
>
> The privileges required for DDL commands are listed in
> HiveAuthzPrivilegesMap.
> {code}
> addOutputObjectPriviledge(AuthorizableType.Table,
> EnumSet.of(DBModelAction.INSERT, DBModelAction.ALTER))
> {code}
> means the required output privileges is table level insert OR alter.
> {code}
> addOutputObjectPriviledge(AuthorizableType.Table,
> EnumSet.of(DBModelAction.INSERT)).
> addOutputObjectPriviledge(AuthorizableType.Table,
> EnumSet.of(DBModelAction.ALTER))
> {code}
> means the required output privileges is table level insert AND alter.
> We need to review the privileges to see if they are defined correctly. I
> suspect multiple definitions want to have privileges with AND, but end up
> getting privileges with OR.
> We should also check if the privilege level is correct. for example, "insert"
> is table level privilege. It does not make sense to require database level
> "insert".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (SENTRY-2289) Tests in TestHDFSIntegrationAdvanced fail from time to time
[ https://issues.apache.org/jira/browse/SENTRY-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-2289: - Assignee: (was: Na Li) > Tests in TestHDFSIntegrationAdvanced fail from time to time > --- > > Key: SENTRY-2289 > URL: https://issues.apache.org/jira/browse/SENTRY-2289 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Priority: Major > Attachments: SENTRY-2289.001.patch > > > The test cases fail from time to time, but works if run individually. > It is likely due to the async nature of HDFS sync from sentry, so when next > test is running, old test clean up removes the added DB, table, role or > privileges. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-2228) Improve on how to handle unsupported Hive commands
[ https://issues.apache.org/jira/browse/SENTRY-2228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-2228: - Assignee: (was: Na Li) > Improve on how to handle unsupported Hive commands > -- > > Key: SENTRY-2228 > URL: https://issues.apache.org/jira/browse/SENTRY-2228 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Priority: Major > Attachments: SENTRY-2228.002.patch > > > Sentry should throw exception when encountering unsupported Hive command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-2239) Table rename cross database should update Full snapshot correctly
[ https://issues.apache.org/jira/browse/SENTRY-2239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-2239: - Assignee: (was: Na Li) > Table rename cross database should update Full snapshot correctly > -- > > Key: SENTRY-2239 > URL: https://issues.apache.org/jira/browse/SENTRY-2239 > Project: Sentry > Issue Type: Bug >Reporter: Na Li >Priority: Major > Attachments: SENTRY-2239.001.patch > > > This only happens if there are "alter table rename" during taking full > snapshot. > When alter table rename changes DB name of the table, FullUpdateModifier > changes all tables' DB name, not just the table whose name is changed. This > will change the table names incorrectly. > For example, when user changes table name from "db1.table1" to table name to > "db2.table2" and in db1 there are two other tables: db1.table3, db1.table4, > FullUpdateModifier changes the tables from "db1.table1, db1.table3, > db1.table4" to "db2.table2, db2.table3, db3.table4". > The desired behavior is changes the tables from "db1.table1" to "db2.table2". > and "db1.table3, db1.table4" remains the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-2340) Enable Sentry code disabled by HIVE-18031
[ https://issues.apache.org/jira/browse/SENTRY-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-2340: - Assignee: (was: Na Li) > Enable Sentry code disabled by HIVE-18031 > -- > > Key: SENTRY-2340 > URL: https://issues.apache.org/jira/browse/SENTRY-2340 > Project: Sentry > Issue Type: Task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Priority: Major > > Once Sentry integrates with Hive version that contains HIVE-18031, we should > enable the code that is disabled because HIVE-18031 is not in the hive > version integrated with Sentry right now. > some code is commented out with "TODO: Enable once HIVE-18031 is available" > or "Enable the test once HIVE-18031 is in the hiver version integrated with > Sentry". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-2338) Enable Sentry code disabled by HIVE-18762
[ https://issues.apache.org/jira/browse/SENTRY-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-2338: - Assignee: (was: Na Li) > Enable Sentry code disabled by HIVE-18762 > - > > Key: SENTRY-2338 > URL: https://issues.apache.org/jira/browse/SENTRY-2338 > Project: Sentry > Issue Type: Task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Priority: Major > > Once Sentry integrates with Hive version that contains HIVE-18762, we should > enable the code that is disabled because HIVE-18762 is not in the hive > version integrated with Sentry right now. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-2211) Review the index creation for foreign keys
[ https://issues.apache.org/jira/browse/SENTRY-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-2211: - Assignee: (was: Na Li) > Review the index creation for foreign keys > -- > > Key: SENTRY-2211 > URL: https://issues.apache.org/jira/browse/SENTRY-2211 > Project: Sentry > Issue Type: Task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Priority: Major > > It is generally recommended to create an index which leads on the foreign key > column(s), to support not only joins between the primary and foreign keys, > but also updates and deletes. > We should review the foreign keys and check if there is an index defined, if > not, should we create index. > More details on The Benefits of Indexing Foreign Keys > https://sqlperformance.com/2012/11/t-sql-queries/benefits-indexing-foreign-keys > Indexes on foreign keys > https://asktom.oracle.com/pls/asktom/f?p=100:11:0P11_QUESTION_ID:292016138754 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-2391) User without any privileges can drop a function
[ https://issues.apache.org/jira/browse/SENTRY-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-2391: - Assignee: (was: Na Li) > User without any privileges can drop a function > --- > > Key: SENTRY-2391 > URL: https://issues.apache.org/jira/browse/SENTRY-2391 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Priority: Major > > Pre-req: > 1. login as an admin. > 2. create a DB as db1 and then create a function func1 > 3. create new role and then grant role to new test user. > Steps: > 1. Login as test user. > 2. Run query : DROP FUNCTION db1.func1; > Actual : Function dropped. > Expected : should not allow drop. > DROP should be allowed only when user has ALL on SERVER or DB. > "anyone can drop a function" is not a security hole, as it does not allow > someone to gain access to something he/she should not. "This may create some > issue for admin" because a function created by admin can be dropped by > anyone, so it disrupts admin's work. Admin has to create a function (that is > dropped by someone with no privilege) again. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-2342) Update schema version verification to support multi-version clusters share same DB
[
https://issues.apache.org/jira/browse/SENTRY-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li reassigned SENTRY-2342:
-
Assignee: (was: Na Li)
> Update schema version verification to support multi-version clusters share
> same DB
> --
>
> Key: SENTRY-2342
> URL: https://issues.apache.org/jira/browse/SENTRY-2342
> Project: Sentry
> Issue Type: New Feature
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Priority: Critical
> Attachments: SENTRY-2342.001.patch, SENTRY-2342.001.patch
>
>
> Right now, sentry software has the schema version it supports hard-coded, and
> before starting its service, it gets schema version from DB and compare these
> two values. If they are not equal, exception will be thrown and sentry
> service won't start.
> To support sentry services with different versions that connect to the same
> DB at the same time, the schema version checking behavior has to be changed
> to allow sentry software with older version to work with DB with newer schema
> version.
> There are two places that verifies schema version
> 1) In SentrySchemaTool when schema is created in DB or upgraded in DB. The
> schema verification should be strict, i.e., ensure the Sentry Server version
> is exactly the same as the DB schema version.
> 2) SentryStore when Sentry service is started after creating schema or
> upgrading schema in DB. The schema verification should be loose, i.e., ensure
> the Sentry Server schema version is compatible with the DB schema version.
> When checking compatible in isSchemaVersionCompatible,
> 1) If DB version is based on schema version used by Sentry Server (DB version
> string starts with Sentry Server schema version), then it is compatible with
> Sentry server
> 2) DB version is shorter than the schema version used by Sentry Server, not
> compatible
> 3) major version number not equal, not compatible
> 4) sentry server has same major version but newer minor version than DB, not
> compatible.
> 5) sentry server has same major version but equal or smaller minor version
> than DB, compatible.
> The call stack when schema verification fails in SentryStore.
> {code}
> Exception in thread "main" java.lang.IllegalStateException: Could not create
> org.apache.sentry.provider.db.service.persistent.SentryStore
> at
> org.apache.sentry.service.thrift.SentryService.getSentryStore(SentryService.java:210)
> at
> org.apache.sentry.service.thrift.SentryService.(SentryService.java:168)
> at
> org.apache.sentry.service.thrift.SentryService$CommandImpl.run(SentryService.java:601)
> at org.apache.sentry.SentryMain.main(SentryMain.java:117)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.sentry.service.thrift.SentryService.getSentryStore(SentryService.java:194)
> ... 9 more
> Caused by: org.apache.sentry.provider.db.SentryAccessDeniedException: The
> Sentry store schema version is different from distribution
> version
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.verifySentryStoreSchema(SentryStore.java:280)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.(SentryStore.java:253)
> ... 14 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (SENTRY-2166) Privilege is not updated for table rename during full snapshot
[
https://issues.apache.org/jira/browse/SENTRY-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li reassigned SENTRY-2166:
-
Assignee: (was: Na Li)
> Privilege is not updated for table rename during full snapshot
> --
>
> Key: SENTRY-2166
> URL: https://issues.apache.org/jira/browse/SENTRY-2166
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Priority: Major
>
> It may take a while for sentry to get full snapshot from HMS. During this
> interval, table rename event could happen.
> The current behavior:
> 1) get notification ID before full snapshot: NotificationID_before
> 2) get full snapshot
> 3) get notification ID after full snapshot: NotificationID_after
> 4) If NotificationID_after != NotificationID_before, get notification events
> between, and apply them to the full snapshot through
> {color:#f79232}*FullUpdateModifier.applyEvent()*{color}.
> 5) save the updated full snapshot with
> *{color:#d04437}NotificationID_after{color}*
>
> When there is a privilege associated with the old table, and table rename
> event happens during full snapshot, the privilege is not associated with the
> renamed table.
>
> The solution is to change the behavior as following:
> 1) get notification ID before full snapshot: NotificationID_before
> 2) get full snapshot
> 3) get notification ID after full snapshot: NotificationID_after
> 4) Save the full snapshot with {color:#d04437}*NotificationID_before*{color}
> 5) When HMSFollower gets notification events from
> *{color:#d04437}NotificationID_before{color},* the notification events
> during full snapshot will be processed through
> {color:#f79232}*NotificationProcessor.processNotificationEvent()* {color}
> In this way, sentry always processes the notification events through
> {color:#f79232}*NotificationProcessor.processNotificationEvent()*{color}, and
> the privilege will be associated with renamed table.
> This issue only happens when events, such as table rename, happens during
> full snapshot.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (SENTRY-2420) TestSentryHDFSServiceProcessor.testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate is flaky
[
https://issues.apache.org/jira/browse/SENTRY-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li reassigned SENTRY-2420:
-
Assignee: (was: Na Li)
> TestSentryHDFSServiceProcessor.testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate
> is flaky
>
>
> Key: SENTRY-2420
> URL: https://issues.apache.org/jira/browse/SENTRY-2420
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Na Li
>Priority: Major
>
> This test sometimes works, and sometimes fails.
> {code}
> [ERROR] Failures:
> [ERROR]
> org.apache.sentry.hdfs.TestSentryHDFSServiceProcessor.testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate(org.apache.sentry.hdfs.TestSentryHDFSServiceProcessor)
> [ERROR] Run 1:
> TestSentryHDFSServiceProcessor.testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate:174
> expected:<1> but was:<0>
> [ERROR] Run 2:
> TestSentryHDFSServiceProcessor.testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate:174
> expected:<1> but was:<0>
> [ERROR] Run 3:
> TestSentryHDFSServiceProcessor.testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate:174
> expected:<1> but was:<0>
> [ERROR] Run 4:
> TestSentryHDFSServiceProcessor.testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate:174
> expected:<1> but was:<0>
> [INFO]
> {code}
> In another run
> {code}
> [ERROR] Tests run: 8, Failures: 4, Errors: 0, Skipped: 2, Time elapsed: 0.128
> s <<< FAILURE! - in org.apache.sentry.hdfs.TestSentryHDFSServiceProcessor
> [ERROR]
> testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate(org.apache.sentry.hdfs.TestSentryHDFSServiceProcessor)
> Time elapsed: 0.011 s <<< FAILURE!
> java.lang.AssertionError: expected:<1> but was:<0>
> at
> org.apache.sentry.hdfs.TestSentryHDFSServiceProcessor.testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate(TestSentryHDFSServiceProcessor.java:174)
> [ERROR]
> testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate(org.apache.sentry.hdfs.TestSentryHDFSServiceProcessor)
> Time elapsed: 0.01 s <<< FAILURE!
> java.lang.AssertionError: expected:<1> but was:<0>
> at
> org.apache.sentry.hdfs.TestSentryHDFSServiceProcessor.testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate(TestSentryHDFSServiceProcessor.java:174)
> [ERROR]
> testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate(org.apache.sentry.hdfs.TestSentryHDFSServiceProcessor)
> Time elapsed: 0 s <<< FAILURE!
> java.lang.AssertionError: expected:<1> but was:<0>
> at
> org.apache.sentry.hdfs.TestSentryHDFSServiceProcessor.testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate(TestSentryHDFSServiceProcessor.java:174)
> [ERROR]
> testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate(org.apache.sentry.hdfs.TestSentryHDFSServiceProcessor)
> Time elapsed: 0 s <<< FAILURE!
> java.lang.AssertionError: expected:<1> but was:<0>
> at
> org.apache.sentry.hdfs.TestSentryHDFSServiceProcessor.testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate(TestSentryHDFSServiceProcessor.java:174)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (SENTRY-1855) Improve scalability of permission delta updates
[
https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li reassigned SENTRY-1855:
-
Assignee: (was: Na Li)
> Improve scalability of permission delta updates
> ---
>
> Key: SENTRY-1855
> URL: https://issues.apache.org/jira/browse/SENTRY-1855
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Alexander Kolbasov
>Priority: Major
> Attachments: SENTRY-1855.001.patch,
> SENTRY-1855.002-sentry-ha-redesign.patch, SENTRY-1855.002.patch,
> SENTRY-1855.003-master.patch, SENTRY-1855.01-sentry-ha-redesign.patch
>
>
> Looking at the latest stress runs, we noticed that some of the transactions
> could fail to commit to the database (with Duplicate key exception) after
> exhausting all the retries.
> This problem has become more evident if we have more number of clients
> connecting to Sentry to issue the permission updates. Was able to reproduce
> consistently with 15 clients doing 100 operations each.
> In the past we introduced exponential backoff (SENTRY-1821) so as part of
> test run increased the defaults to 750ms sleep and 20 retries. But even after
> this, the cluster still shows up the transaction failures. This change also
> increases the latency of every transaction in sentry.
> We need to revisit this and come up with a better way to solve this problem.
> {code}
> 2017-07-13 13:18:14,449 ERROR
> org.apache.sentry.provider.db.service.persistent.TransactionManager: The
> transaction has reached max retry number, Exception thrown when executing
> query
> javax.jdo.JDOException: Exception thrown when executing query
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213)
> --
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
> at
> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
> ... 33 more
> 2017-07-13 13:18:14,450 ERROR
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor:
> Unknown error for request:
> TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805,
> privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1,
> dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*,
> createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The
> transaction has reached max retry number, Exception thrown when executing
> query
> java.lang.Exception: The transaction has reached max retry number, Exception
> thrown when executing query
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:255)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:209)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3330)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:593)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:633)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alte
[jira] [Assigned] (SENTRY-2158) Update notification handler to update privileges to user
[ https://issues.apache.org/jira/browse/SENTRY-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-2158: - Assignee: (was: Na Li) > Update notification handler to update privileges to user > > > Key: SENTRY-2158 > URL: https://issues.apache.org/jira/browse/SENTRY-2158 > Project: Sentry > Issue Type: New Feature > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Priority: Minor > > SentryPolicyStoreProcessor calls NotificationHandlerInvoker when processing > permission related commands. We should update notification handler in the > following files when granting privileges to user. > When authorizable changes, need to change user privileges too > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/NotificationHandler.java > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/NotificationHandlerInvoker.java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-2185) Performance Issue: Saving MAuthzPathsMapping should be done in batch for path full snapshot
[
https://issues.apache.org/jira/browse/SENTRY-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li reassigned SENTRY-2185:
-
Assignee: (was: Na Li)
> Performance Issue: Saving MAuthzPathsMapping should be done in batch for path
> full snapshot
> ---
>
> Key: SENTRY-2185
> URL: https://issues.apache.org/jira/browse/SENTRY-2185
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Priority: Major
>
> From the code below, for each MAuthzPathsMapping, there is a query to save
> each MAuthzPathsMapping instance. Need to save the changes in batch.
> {code:java}
> public void persistFullPathsImage(final Map>
> authzPaths,
> final long notificationID) throws Exception {
> tm.executeTransactionWithRetry(
> pm -> {
> pm.setDetachAllOnCommit(false); // No need to detach objects
> deleteNotificationsSince(pm, notificationID + 1);
> // persist the notidicationID
> pm.makePersistent(new MSentryHmsNotification(notificationID));
> // persist the full snapshot
> long snapshotID = getCurrentAuthzPathsSnapshotID(pm);
> long nextSnapshotID = snapshotID + 1;
> pm.makePersistent(new MAuthzPathsSnapshotId(nextSnapshotID));
> LOGGER.info("Attempting to commit new HMS snapshot with ID =
> {}", nextSnapshotID);
> for (Map.Entry> authzPath :
> authzPaths.entrySet()) {
> pm.makePersistent(new MAuthzPathsMapping(nextSnapshotID,
> authzPath.getKey(), authzPath.getValue()));
> }
> return null;
> });
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (SENTRY-2300) Move Permission Update due to DDL to HMS Post Event Listener
[
https://issues.apache.org/jira/browse/SENTRY-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li reassigned SENTRY-2300:
-
Assignee: (was: Na Li)
> Move Permission Update due to DDL to HMS Post Event Listener
>
>
> Key: SENTRY-2300
> URL: https://issues.apache.org/jira/browse/SENTRY-2300
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Na Li
>Priority: Major
> Attachments: SENTRY-2300.001.patch, SENTRY-2300.002.patch
>
>
> There was a code in MetastorePlugin that modified Sentry privileges on table
> Create/Drop and database Create/Drop. As part of Sentry HA work we moved all
> this logic from Sentry plugin to be driven by notifications which required
> the extra synchronization between HMS and Sentry.
> It should be possible to do permission changes in the post event listener
> itself to avoid blocking for Sentry. This requires some experiments though
> because it may cause strange artifacts since at the time these DDL operations
> are done Sentry may not be aware of the current state - for example you may
> try to change permissions of a table that Sentry doesn’t know about, which
> seems to be OK.
> This update will have the following benefits:
> {code}
> * HMS waits on Sentry polling HMS update takes 0.5 to 1 second. This update
> will remove this delay
> * Sentry knows every DDL update, and therefore can update permission
> correctly. In current approach using notification processing, Sentry could
> miss updates if full snapshot is fetched from HMS, and permission is not
> updated correctly. In the case of table rename, when mission DDL update event
> because of full snapshot, sentry will not move the permissions associated
> with old table to the new table. And the authorization on queries on the
> renamed table will fail.
> {code}
> The proposed approach is:
> {code}
> 1) Add the permission update in
> SentryPolicyStoreProcessor.sentry_notify_hms_event() through
> SentrySyncHMSNotificationsPostEventListener for create/drop/alter table,
> create/drop database commands. This is synchronous processing.
> 2) Remove the permission update from
> NotificationProcessor.processNotificationEvent() for create/drop/alter table,
> create/drop database commands.
> 3) Remove the blocking of HMS in SentrySyncHMSNotificationsPostEventListener
> for Sentry fetching notification events asynchronously from HMS for
> create/drop/alter table, create/drop database commands. The reason is that
> now the permission is updated through
> SentrySyncHMSNotificationsPostEventListener. Therefore, there is no need for
> HMS to wait for Sentry fetching those notification events.
> {code}
> Without this approach, the DDL operation take at least 500 to 1000
> milliseconds. It is easy for HMS to encounter SyncTimeoutException when it
> takes long time for Sentry to get full snapshot or fetch notification events.
> Many users choose to remove the post event listener in configuration (HMS not
> waiting for Sentry fetching notification events asynchronously) to get rid of
> this exception. Users cannot take this workaround if they want owner
> privilege enabled.
> This approach makes DDL operation in HMS independent from Sentry fetching
> full snapshot or notification events, and reduces the DDL operation delay
> significantly. With this approach, the DDL operation is in the range of few
> milliseconds. This approach fixes the timeout issue once for all. And users
> can get rid of timeout issue and enable owner privilege at the same time
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (SENTRY-2507) Authorization of "default" database is not controlled by "sentry.hive.restrict.defaultDB" at HMS server
[ https://issues.apache.org/jira/browse/SENTRY-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li resolved SENTRY-2507. --- Resolution: Not A Problem HMS server behaves the same as Hive server for the configuration and for default db. > Authorization of "default" database is not controlled by > "sentry.hive.restrict.defaultDB" at HMS server > --- > > Key: SENTRY-2507 > URL: https://issues.apache.org/jira/browse/SENTRY-2507 > Project: Sentry > Issue Type: Bug > Components: Sentry >Reporter: Na Li >Priority: Major > > If "sentry.hive.restrict.defaultDB" at sentry-site.xml at HMS server is set > to be false, user still has to have "SELECT", or "INSERT", or "ALL" privilege > on the "default" database in order to access it. > This behavior is not consistent with the behavior at Hive server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2507) Authorization of "default" database is not controlled by "sentry.hive.restrict.defaultDB" at HMS server
[ https://issues.apache.org/jira/browse/SENTRY-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785161#comment-16785161 ] Na Li commented on SENTRY-2507: --- >From hive beeline, user no_pri ha "ALL" privilege on default.tb1. user >no_pri_2 does not have any privilege 1) show databases; 1.1) with ALL privilege on default.tb1 default 1.2) no privilege at all default 2) describe database default; 2.1) with ALL privilege on default.tb1 Error while compiling statement: FAILED: SemanticException No valid privileges User no_pri does not have privileges for DESCDATABASE The required privileges: Server=server1->Db=default->action=select->grantOption=false;Server=server1->Db=default->action=insert->grantOption=false; 2.2) no privilege at all Error while compiling statement: FAILED: SemanticException No valid privileges User no_pri_2 does not have privileges for SWITCHDATABASE The required privileges: Server=server1->Db=*->Table=+->Column=*->action=select->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=insert->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=alter->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=create->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=drop->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=index->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=lock->grantOption=false; 3) use default; 3.1) with ALL privilege on default.tb1; succeed 3.2) no privilege at all Error while compiling statement: FAILED: SemanticException No valid privileges User no_pri_2 does not have privileges for SWITCHDATABASE The required privileges: Server=server1->Db=*->Table=+->Column=*->action=select->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=insert->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=alter->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=create->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=drop->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=index->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=lock->grantOption=false; 4) show tables; 4.1) with ALL privilege on default.tb1; tb1 4.2) no privilege at all Error while compiling statement: FAILED: SemanticException No valid privileges User no_pri_2 does not have privileges for SWITCHDATABASE The required privileges: Server=server1->Db=*->Table=+->Column=*->action=select->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=insert->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=alter->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=create->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=drop->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=index->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=lock->grantOption=false; > Authorization of "default" database is not controlled by > "sentry.hive.restrict.defaultDB" at HMS server > --- > > Key: SENTRY-2507 > URL: https://issues.apache.org/jira/browse/SENTRY-2507 > Project: Sentry > Issue Type: Bug > Components: Sentry >Reporter: Na Li >Priority: Major > > If "sentry.hive.restrict.defaultDB" at sentry-site.xml at HMS server is set > to be false, user still has to have "SELECT", or "INSERT", or "ALL" privilege > on the "default" database in order to access it. > This behavior is not consistent with the behavior at Hive server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2503) Failed to revoke the privilege from impala-shell if the privilege added from beeline cli
[ https://issues.apache.org/jira/browse/SENTRY-2503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2503: -- Resolution: Fixed Status: Resolved (was: Patch Available) [~arjunmishra13] Thanks for reviewing the code! > Failed to revoke the privilege from impala-shell if the privilege added from > beeline cli > > > Key: SENTRY-2503 > URL: https://issues.apache.org/jira/browse/SENTRY-2503 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.2.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2503.001.patch, SENTRY-2503.002.patch, > SENTRY-2503.003.patch, SENTRY-2503.004.patch, SENTRY-2503.004.patch > > > When granting all privilege on a table from impala-shell, this privilege > cannot be revoked from hiove beeline. > When granting all privilege on a table from hive beeline, this privilege > cannot be revoked from impala-shell. > The reason is that impala uses "ALL" to represent all privilege, and hive > uses "*". So getting privilege to drop it does not properly. > The solution is to find all equivalent privileges from input, and drop them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SENTRY-2507) Authorization of "default" database is not controlled by "sentry.hive.restrict.defaultDB" at HMS server
Na Li created SENTRY-2507: - Summary: Authorization of "default" database is not controlled by "sentry.hive.restrict.defaultDB" at HMS server Key: SENTRY-2507 URL: https://issues.apache.org/jira/browse/SENTRY-2507 Project: Sentry Issue Type: Bug Components: Sentry Reporter: Na Li If "sentry.hive.restrict.defaultDB" at sentry-site.xml at HMS server is set to be false, user still has to have "SELECT", or "INSERT", or "ALL" privilege on the "default" database in order to access it. This behavior is not consistent with the behavior at Hive server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2503) Failed to revoke the privilege from impala-shell if the privilege added from beeline cli
[ https://issues.apache.org/jira/browse/SENTRY-2503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2503: -- Attachment: SENTRY-2503.004.patch > Failed to revoke the privilege from impala-shell if the privilege added from > beeline cli > > > Key: SENTRY-2503 > URL: https://issues.apache.org/jira/browse/SENTRY-2503 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.2.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2503.001.patch, SENTRY-2503.002.patch, > SENTRY-2503.003.patch, SENTRY-2503.004.patch, SENTRY-2503.004.patch > > > When granting all privilege on a table from impala-shell, this privilege > cannot be revoked from hiove beeline. > When granting all privilege on a table from hive beeline, this privilege > cannot be revoked from impala-shell. > The reason is that impala uses "ALL" to represent all privilege, and hive > uses "*". So getting privilege to drop it does not properly. > The solution is to find all equivalent privileges from input, and drop them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
