[jira] [Commented] (SENTRY-1557) getRolesForGroups() does too many trips to the the DB
[ https://issues.apache.org/jira/browse/SENTRY-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727502#comment-15727502 ] Alexander Kolbasov commented on SENTRY-1557: We can optimize things a bit by using approach from SENTRY-1557 > getRolesForGroups() does too many trips to the the DB > - > > Key: SENTRY-1557 > URL: https://issues.apache.org/jira/browse/SENTRY-1557 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 1.8.0, sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: Vamsee Yarlagadda >Priority: Minor > Labels: bite-sized > Attachments: SENTRY-1557.patch > > > The function getRolesForGroups() does a SELECT query for each group in the > input set. Instead it should do one of the following: > 1) A single SELECT with OR for all groups > 2) Just fetch all groups and filter in-memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SENTRY-1557) getRolesForGroups() does too many trips to the the DB
[
https://issues.apache.org/jira/browse/SENTRY-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727504#comment-15727504
]
Hadoop QA commented on SENTRY-1557:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12842077/SENTRY-1557.patch
against master.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/2186/console
This message is automatically generated.
> getRolesForGroups() does too many trips to the the DB
> -
>
> Key: SENTRY-1557
> URL: https://issues.apache.org/jira/browse/SENTRY-1557
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 1.8.0, sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Vamsee Yarlagadda
>Priority: Minor
> Labels: bite-sized
> Attachments: SENTRY-1557.patch
>
>
> The function getRolesForGroups() does a SELECT query for each group in the
> input set. Instead it should do one of the following:
> 1) A single SELECT with OR for all groups
> 2) Just fetch all groups and filter in-memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SENTRY-1557) getRolesForGroups() does too many trips to the the DB
[ https://issues.apache.org/jira/browse/SENTRY-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727500#comment-15727500 ] Alexander Kolbasov commented on SENTRY-1557: The solution is really nice - if this works, can we do (possibly in a separate JIRA) similar thing for multiple cases of search for many roles? > getRolesForGroups() does too many trips to the the DB > - > > Key: SENTRY-1557 > URL: https://issues.apache.org/jira/browse/SENTRY-1557 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 1.8.0, sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: Vamsee Yarlagadda >Priority: Minor > Labels: bite-sized > Attachments: SENTRY-1557.patch > > > The function getRolesForGroups() does a SELECT query for each group in the > input set. Instead it should do one of the following: > 1) A single SELECT with OR for all groups > 2) Just fetch all groups and filter in-memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SENTRY-1557) getRolesForGroups() does too many trips to the the DB
[ https://issues.apache.org/jira/browse/SENTRY-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727488#comment-15727488 ] Alexander Kolbasov commented on SENTRY-1557: Do we have tests that try these with multiple groups? > getRolesForGroups() does too many trips to the the DB > - > > Key: SENTRY-1557 > URL: https://issues.apache.org/jira/browse/SENTRY-1557 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 1.8.0, sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: Vamsee Yarlagadda >Priority: Minor > Labels: bite-sized > Attachments: SENTRY-1557.patch > > > The function getRolesForGroups() does a SELECT query for each group in the > input set. Instead it should do one of the following: > 1) A single SELECT with OR for all groups > 2) Just fetch all groups and filter in-memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SENTRY-1548) Setting GrantOption to UNSET upsets Sentry
[
https://issues.apache.org/jira/browse/SENTRY-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vamsee Yarlagadda updated SENTRY-1548:
--
Attachment: (was: SENTRY-1557.patch)
> Setting GrantOption to UNSET upsets Sentry
> --
>
> Key: SENTRY-1548
> URL: https://issues.apache.org/jira/browse/SENTRY-1548
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
>Priority: Minor
> Labels: bite-sized
> Attachments: SENTRY-1548.001.patch
>
>
> If we send a Thrift request to sentry (using regular api) with GrantOption
> set to UNSET (-1) we get the following error:
> {code}
> TransactionManager.executeTransactionWithRetry(TransactionManager.java:102)]
> The transaction has reac
> hed max retry number, will not retry again.
> javax.jdo.JDODataStoreException: Insert of object
> "org.apache.sentry.provider.db.service.model.MSentryPrivilege@6bbfd4c9" using
> statement "INSERT INTO `SENTRY_DB_PRIVILEGE`
> (`DB_PRIVILEGE_ID`,`SERVER_NAME`,`WITH_GRANT_OPTION`,`CREATE_TIME`,`TABLE_NAME`,`URI`,`ACTION`,`COLUMN_NAME`,`DB_NAME`,`PRIVILEGE_SCOPE`)
> VALUES (?,?,?,?,?,?,?,?,?,?)" failed : Column 'WITH_GRANT_OPTION' cannot be
> null
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:438)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:95)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$8.execute(SentryStore.java:374)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:72)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:93)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:367)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:280)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Column 'WITH_GRANT_OPTION' cannot be null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
> at com.mysql.jdbc.Util.getInstance(Util.java:387)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3966)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3902)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2526)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
> at
>
[jira] [Updated] (SENTRY-1557) getRolesForGroups() does too many trips to the the DB
[ https://issues.apache.org/jira/browse/SENTRY-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vamsee Yarlagadda updated SENTRY-1557: -- Attachment: SENTRY-1557.patch > getRolesForGroups() does too many trips to the the DB > - > > Key: SENTRY-1557 > URL: https://issues.apache.org/jira/browse/SENTRY-1557 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 1.8.0, sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: Vamsee Yarlagadda >Priority: Minor > Labels: bite-sized > Attachments: SENTRY-1557.patch > > > The function getRolesForGroups() does a SELECT query for each group in the > input set. Instead it should do one of the following: > 1) A single SELECT with OR for all groups > 2) Just fetch all groups and filter in-memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SENTRY-1548) Setting GrantOption to UNSET upsets Sentry
[
https://issues.apache.org/jira/browse/SENTRY-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vamsee Yarlagadda updated SENTRY-1548:
--
Attachment: SENTRY-1557.patch
> Setting GrantOption to UNSET upsets Sentry
> --
>
> Key: SENTRY-1548
> URL: https://issues.apache.org/jira/browse/SENTRY-1548
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
>Priority: Minor
> Labels: bite-sized
> Attachments: SENTRY-1548.001.patch
>
>
> If we send a Thrift request to sentry (using regular api) with GrantOption
> set to UNSET (-1) we get the following error:
> {code}
> TransactionManager.executeTransactionWithRetry(TransactionManager.java:102)]
> The transaction has reac
> hed max retry number, will not retry again.
> javax.jdo.JDODataStoreException: Insert of object
> "org.apache.sentry.provider.db.service.model.MSentryPrivilege@6bbfd4c9" using
> statement "INSERT INTO `SENTRY_DB_PRIVILEGE`
> (`DB_PRIVILEGE_ID`,`SERVER_NAME`,`WITH_GRANT_OPTION`,`CREATE_TIME`,`TABLE_NAME`,`URI`,`ACTION`,`COLUMN_NAME`,`DB_NAME`,`PRIVILEGE_SCOPE`)
> VALUES (?,?,?,?,?,?,?,?,?,?)" failed : Column 'WITH_GRANT_OPTION' cannot be
> null
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:438)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:95)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$8.execute(SentryStore.java:374)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:72)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:93)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:367)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:280)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Column 'WITH_GRANT_OPTION' cannot be null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
> at com.mysql.jdbc.Util.getInstance(Util.java:387)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3966)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3902)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2526)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
> at
>
[jira] [Updated] (SENTRY-1548) Setting GrantOption to UNSET upsets Sentry
[
https://issues.apache.org/jira/browse/SENTRY-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kalyan kumar kalvagadda updated SENTRY-1548:
Status: Patch Available (was: In Progress)
> Setting GrantOption to UNSET upsets Sentry
> --
>
> Key: SENTRY-1548
> URL: https://issues.apache.org/jira/browse/SENTRY-1548
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
>Priority: Minor
> Labels: bite-sized
> Attachments: SENTRY-1548.001.patch
>
>
> If we send a Thrift request to sentry (using regular api) with GrantOption
> set to UNSET (-1) we get the following error:
> {code}
> TransactionManager.executeTransactionWithRetry(TransactionManager.java:102)]
> The transaction has reac
> hed max retry number, will not retry again.
> javax.jdo.JDODataStoreException: Insert of object
> "org.apache.sentry.provider.db.service.model.MSentryPrivilege@6bbfd4c9" using
> statement "INSERT INTO `SENTRY_DB_PRIVILEGE`
> (`DB_PRIVILEGE_ID`,`SERVER_NAME`,`WITH_GRANT_OPTION`,`CREATE_TIME`,`TABLE_NAME`,`URI`,`ACTION`,`COLUMN_NAME`,`DB_NAME`,`PRIVILEGE_SCOPE`)
> VALUES (?,?,?,?,?,?,?,?,?,?)" failed : Column 'WITH_GRANT_OPTION' cannot be
> null
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:438)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:95)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$8.execute(SentryStore.java:374)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:72)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:93)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:367)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:280)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Column 'WITH_GRANT_OPTION' cannot be null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
> at com.mysql.jdbc.Util.getInstance(Util.java:387)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3966)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3902)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2526)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
> at
>
[jira] [Comment Edited] (SENTRY-1548) Setting GrantOption to UNSET upsets Sentry
[
https://issues.apache.org/jira/browse/SENTRY-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726980#comment-15726980
]
kalyan kumar kalvagadda edited comment on SENTRY-1548 at 12/6/16 10:53 PM:
---
I have made changes assuming that grant option is either true/false removing
unset.
Also, added code so that sentry server could validate the TSentryPrivilege
object constructed from the Thrift message received client. If the validation
is failed exception is raised and appropriate error is message is sent.
was (Author: kkalyan):
I have made changes assuming that grant option is either true/false removing
unset.
> Setting GrantOption to UNSET upsets Sentry
> --
>
> Key: SENTRY-1548
> URL: https://issues.apache.org/jira/browse/SENTRY-1548
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
>Priority: Minor
> Labels: bite-sized
>
> If we send a Thrift request to sentry (using regular api) with GrantOption
> set to UNSET (-1) we get the following error:
> {code}
> TransactionManager.executeTransactionWithRetry(TransactionManager.java:102)]
> The transaction has reac
> hed max retry number, will not retry again.
> javax.jdo.JDODataStoreException: Insert of object
> "org.apache.sentry.provider.db.service.model.MSentryPrivilege@6bbfd4c9" using
> statement "INSERT INTO `SENTRY_DB_PRIVILEGE`
> (`DB_PRIVILEGE_ID`,`SERVER_NAME`,`WITH_GRANT_OPTION`,`CREATE_TIME`,`TABLE_NAME`,`URI`,`ACTION`,`COLUMN_NAME`,`DB_NAME`,`PRIVILEGE_SCOPE`)
> VALUES (?,?,?,?,?,?,?,?,?,?)" failed : Column 'WITH_GRANT_OPTION' cannot be
> null
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:438)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:95)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$8.execute(SentryStore.java:374)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:72)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:93)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:367)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:280)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Column 'WITH_GRANT_OPTION' cannot be null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
> at com.mysql.jdbc.Util.getInstance(Util.java:387)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3966)
[jira] [Commented] (SENTRY-1548) Setting GrantOption to UNSET upsets Sentry
[
https://issues.apache.org/jira/browse/SENTRY-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726980#comment-15726980
]
kalyan kumar kalvagadda commented on SENTRY-1548:
-
I have made changes assuming that grant option is either true/false removing
unset.
> Setting GrantOption to UNSET upsets Sentry
> --
>
> Key: SENTRY-1548
> URL: https://issues.apache.org/jira/browse/SENTRY-1548
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
>Priority: Minor
> Labels: bite-sized
>
> If we send a Thrift request to sentry (using regular api) with GrantOption
> set to UNSET (-1) we get the following error:
> {code}
> TransactionManager.executeTransactionWithRetry(TransactionManager.java:102)]
> The transaction has reac
> hed max retry number, will not retry again.
> javax.jdo.JDODataStoreException: Insert of object
> "org.apache.sentry.provider.db.service.model.MSentryPrivilege@6bbfd4c9" using
> statement "INSERT INTO `SENTRY_DB_PRIVILEGE`
> (`DB_PRIVILEGE_ID`,`SERVER_NAME`,`WITH_GRANT_OPTION`,`CREATE_TIME`,`TABLE_NAME`,`URI`,`ACTION`,`COLUMN_NAME`,`DB_NAME`,`PRIVILEGE_SCOPE`)
> VALUES (?,?,?,?,?,?,?,?,?,?)" failed : Column 'WITH_GRANT_OPTION' cannot be
> null
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:438)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:95)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$8.execute(SentryStore.java:374)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:72)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:93)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:367)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:280)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Column 'WITH_GRANT_OPTION' cannot be null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
> at com.mysql.jdbc.Util.getInstance(Util.java:387)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3966)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3902)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2526)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
>
[jira] [Commented] (SENTRY-1547) It is possible to create a privilege with all empty fields
[
https://issues.apache.org/jira/browse/SENTRY-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726719#comment-15726719
]
Hadoop QA commented on SENTRY-1547:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12842022/SENTRY-1547.001.patch
against master.
{color:red}Overall:{color} -1 due to 8 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.tests.e2e.dbprovider.TestDbPrivilegesAtFunctionScope
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.thrift.TestSentryServiceImportExport
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.thrift.TestSentryServiceImportExport
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.thrift.TestSentryServiceImportExport
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.thrift.TestSentryServiceImportExport
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.thrift.TestSentryServiceImportExport
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.thrift.TestSentryServiceImportExport
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/2184/console
This message is automatically generated.
> It is possible to create a privilege with all empty fields
> --
>
> Key: SENTRY-1547
> URL: https://issues.apache.org/jira/browse/SENTRY-1547
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 1.8.0, sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
>Priority: Minor
> Labels: bite-sized
> Attachments: SENTRY-1547.001.patch
>
>
> It is possible (at least by sending Thrift message) to create privilege with
> everything set to __NULL__ which is a pretty useless privilege. We should
> check that at least some fields are set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SENTRY-1547) It is possible to create a privilege with all empty fields
[ https://issues.apache.org/jira/browse/SENTRY-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-1547: Status: Patch Available (was: In Progress) > It is possible to create a privilege with all empty fields > -- > > Key: SENTRY-1547 > URL: https://issues.apache.org/jira/browse/SENTRY-1547 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 1.8.0, sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda >Priority: Minor > Labels: bite-sized > Attachments: SENTRY-1547.001.patch > > > It is possible (at least by sending Thrift message) to create privilege with > everything set to __NULL__ which is a pretty useless privilege. We should > check that at least some fields are set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SENTRY-1547) It is possible to create a privilege with all empty fields
[ https://issues.apache.org/jira/browse/SENTRY-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-1547: Attachment: SENTRY-1547.001.patch > It is possible to create a privilege with all empty fields > -- > > Key: SENTRY-1547 > URL: https://issues.apache.org/jira/browse/SENTRY-1547 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 1.8.0, sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda >Priority: Minor > Labels: bite-sized > Attachments: SENTRY-1547.001.patch > > > It is possible (at least by sending Thrift message) to create privilege with > everything set to __NULL__ which is a pretty useless privilege. We should > check that at least some fields are set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (SENTRY-1547) It is possible to create a privilege with all empty fields
[ https://issues.apache.org/jira/browse/SENTRY-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726616#comment-15726616 ] kalyan kumar kalvagadda edited comment on SENTRY-1547 at 12/6/16 8:30 PM: -- SENTRY-1547.001.patch I have added code so that sentry server could validate the TSentryPrivilege object constructed from the Thrift message received client. If the validation is failed exception is raised and appropriate error is message is sent. was (Author: kkalyan): I have added code so that sentry server could validate the TSentryPrivilege object constructed from the Thrift message received client. If the validation is failed exception is raised and appropriate error is message is sent. > It is possible to create a privilege with all empty fields > -- > > Key: SENTRY-1547 > URL: https://issues.apache.org/jira/browse/SENTRY-1547 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 1.8.0, sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda >Priority: Minor > Labels: bite-sized > > It is possible (at least by sending Thrift message) to create privilege with > everything set to __NULL__ which is a pretty useless privilege. We should > check that at least some fields are set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SENTRY-1547) It is possible to create a privilege with all empty fields
[ https://issues.apache.org/jira/browse/SENTRY-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726580#comment-15726580 ] kalyan kumar kalvagadda commented on SENTRY-1547: - Mandatory fields in that are expected in a privilege are server_name and action. If any of these are not provided sentry server should return error with appropriate description. > It is possible to create a privilege with all empty fields > -- > > Key: SENTRY-1547 > URL: https://issues.apache.org/jira/browse/SENTRY-1547 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 1.8.0, sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda >Priority: Minor > Labels: bite-sized > > It is possible (at least by sending Thrift message) to create privilege with > everything set to __NULL__ which is a pretty useless privilege. We should > check that at least some fields are set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SENTRY-1556) Simplify privilege cleaning
[ https://issues.apache.org/jira/browse/SENTRY-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726294#comment-15726294 ] kalyan kumar kalvagadda commented on SENTRY-1556: - I strongly agree with the approach-1 mentioned in the description unless there is a reason not to take that approach. > Simplify privilege cleaning > --- > > Key: SENTRY-1556 > URL: https://issues.apache.org/jira/browse/SENTRY-1556 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 1.8.0, sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda >Priority: Minor > > The SentryStore class has a privCleaner that cleans up orphaned privileges. > Currently cleaning is happening after 50 notification requests are sent and > it uses locking to synchronize. > I think the whole thing can be simplified: > 1) We should consider whether it is possible to clean up a privilege simply > when we see that there are no roles associated with it. In this case we do > not need this at all. > 2) We can simply run a periodic job to clean up orphaned privileges and > groups (which are not cleaned up at all now). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SENTRY-1556) Simplify privilege cleaning
[ https://issues.apache.org/jira/browse/SENTRY-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726123#comment-15726123 ] Alexander Kolbasov commented on SENTRY-1556: I also think that SentryStore class shouldn't manage priv cleaner thread - it should be up-leveled. SentryStore can provide necessary methods to do that, but shouldn't be in the business of job management. > Simplify privilege cleaning > --- > > Key: SENTRY-1556 > URL: https://issues.apache.org/jira/browse/SENTRY-1556 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 1.8.0, sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda >Priority: Minor > > The SentryStore class has a privCleaner that cleans up orphaned privileges. > Currently cleaning is happening after 50 notification requests are sent and > it uses locking to synchronize. > I think the whole thing can be simplified: > 1) We should consider whether it is possible to clean up a privilege simply > when we see that there are no roles associated with it. In this case we do > not need this at all. > 2) We can simply run a periodic job to clean up orphaned privileges and > groups (which are not cleaned up at all now). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
