[jira] [Commented] (SENTRY-1684) FullUpdateInitializer has a race condition in handling results list
[
https://issues.apache.org/jira/browse/SENTRY-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951953#comment-15951953
]
Hadoop QA commented on SENTRY-1684:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12861552/SENTRY-1684.001-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/2469/console
This message is automatically generated.
> FullUpdateInitializer has a race condition in handling results list
> ---
>
> Key: SENTRY-1684
> URL: https://issues.apache.org/jira/browse/SENTRY-1684
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: SENTRY-1684.001-sentry-ha-redesign.patch
>
>
> The FullUpdateInitializer class has exactly the same race condition as the
> one described in SENTRY-1683.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (SENTRY-1688) Apache fails to build Sentry package Sentry-jdk-1.7-v2
Alexander Kolbasov created SENTRY-1688:
--
Summary: Apache fails to build Sentry package Sentry-jdk-1.7-v2
Key: SENTRY-1688
URL: https://issues.apache.org/jira/browse/SENTRY-1688
Project: Sentry
Issue Type: Bug
Components: Sentry
Reporter: Alexander Kolbasov
Priority: Minor
Fix For: 1.8.0
For some time Apache job that builds Sentry-jdk-1.7-v2 fails with a rat
exception in sentry-service-server:
{code}
[INFO]
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Sentry SUCCESS [16.973s]
[INFO] Sentry Core ... SUCCESS [2.369s]
[INFO] Sentry Core Common SUCCESS [1:18.513s]
[INFO] Sentry Core Model DB .. SUCCESS [7.649s]
[INFO] Sentry Core Model Indexer . SUCCESS [5.596s]
[INFO] Sentry Core Model Search .. SUCCESS [5.282s]
[INFO] Sentry Core Model Sqoop ... SUCCESS [5.448s]
[INFO] Sentry Core Model Kafka ... SUCCESS [5.676s]
[INFO] Sentry Service SUCCESS [2.295s]
[INFO] Sentry Service Common . SUCCESS [14.661s]
[INFO] Sentry Providers .. SUCCESS [2.751s]
[INFO] Sentry Provider Common SUCCESS [4.940s]
[INFO] Sentry Provider File .. SUCCESS [6.453s]
[INFO] Sentry Service Client . SUCCESS [5.534s]
[INFO] Sentry Policies ... SUCCESS [2.150s]
[INFO] Sentry Policy Common .. SUCCESS [10.257s]
[INFO] Sentry Authorization Provider . SUCCESS [9.241s]
[INFO] Sentry Service Server . FAILURE [10:06.319s]
[INFO] Sentry Provider DB SUCCESS [5.009s]
[INFO] Sentry Policy Engine .. SUCCESS [5.334s]
[INFO] Sentry Bindings ... SUCCESS [2.437s]
[INFO] Sentry Binding for Kafka .. SUCCESS [20.473s]
[INFO] Sentry Provider Cache . SUCCESS [7.156s]
[INFO] Sentry Hive Binding Common SUCCESS [10.143s]
[INFO] Sentry Binding for Solr ... SUCCESS [19.103s]
[INFO] Sentry Binding for Sqoop .. SUCCESS [15.041s]
[INFO] Sentry Binding v2 for Hive SUCCESS [16.203s]
[INFO] Sentry Policy for Indexer . SUCCESS [17.663s]
[INFO] Sentry Solr ... SUCCESS [3.083s]
[INFO] Solr Sentry Core .. SUCCESS [7.723s]
[INFO] Solr Sentry handler ... SUCCESS [27.111s]
[INFO] Sentry Tests .. SUCCESS [2.865s]
[INFO] Sentry Solr Tests . SKIPPED
[INFO] Sentry Sqoop Tests SKIPPED
[INFO] Sentry Kafka Tests SKIPPED
[INFO] Sentry HDFS ... SUCCESS [3.260s]
[INFO] Sentry HDFS Common SKIPPED
[INFO] Sentry HDFS Service ... SKIPPED
[INFO] Sentry HDFS Namenode Plugin ... SKIPPED
[INFO] Sentry Hive Tests v2 .. SKIPPED
[INFO] Sentry HDFS Dist .. SKIPPED
[INFO] Sentry Distribution ... SKIPPED
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 17:17.124s
[INFO] Finished at: Sat Apr 01 00:32:51 UTC 2017
[INFO] Final Memory: 113M/694M
[INFO]
Waiting for Jenkins to finish collecting data
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.10:check
(header-check) on project sentry-service-server: Too many files with unapproved
license: 1 See RAT report in:
/home/jenkins/jenkins-slave/workspace/Sentry-jdk-1.7-v2/sentry-service/sentry-service-server/target/rat.txt
-> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal
org.apache.rat:apache-rat-plugin:0.10:check (header-check) on project
sentry-service-server: Too many files with unapproved license: 1 See RAT report
in:
/home/jenkins/jenkins-slave/workspace/Sentry-jdk-1.7-v2/sentry-service/sentry-service-server/target/rat.txt
{code}
But when I build it manually I
[jira] [Updated] (SENTRY-1687) FullUpdateInitializer can be more efficient
[
https://issues.apache.org/jira/browse/SENTRY-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-1687:
---
Issue Type: Sub-task (was: Improvement)
Parent: SENTRY-872
> FullUpdateInitializer can be more efficient
> ---
>
> Key: SENTRY-1687
> URL: https://issues.apache.org/jira/browse/SENTRY-1687
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Minor
>
> The FullUpdateInitializer follows the {{MetastoreCacheInitializer}}. It reads
> a bunch of information from HMS and uses Thrift structures to pass around,
> but in the end it just constructs {{Map}}. It uses
> concurrent fetches from HMS, but synchronizes a lot on common data structures
> to update them.
> I think that we can refactor all this code to make it faster and consume less
> memory. The idea is the following:
> Use background threads to collect Thrift results from HMS calls (database,
> table and partition data). Then we can use a single thread to construct the
> resulting update and return it without using intermediate Thrift methods.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SENTRY-1530) Sentry should use to io.dropwizard.metrics
[
https://issues.apache.org/jira/browse/SENTRY-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951877#comment-15951877
]
Hadoop QA commented on SENTRY-1530:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12839481/SENTRY-1530.002.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/2468/console
This message is automatically generated.
> Sentry should use to io.dropwizard.metrics
> --
>
> Key: SENTRY-1530
> URL: https://issues.apache.org/jira/browse/SENTRY-1530
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 1.8.0, sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Priority: Minor
> Attachments: SENTRY-1530.001.patch, SENTRY-1530.002.patch
>
>
> The com.codahale.metrics is now io.dropwizard.metrics with current version
> being 3.1.0.
> We should consider moving Sentry to this version.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (SENTRY-1687) FullUpdateInitializer can be more efficient
Alexander Kolbasov created SENTRY-1687:
--
Summary: FullUpdateInitializer can be more efficient
Key: SENTRY-1687
URL: https://issues.apache.org/jira/browse/SENTRY-1687
Project: Sentry
Issue Type: Improvement
Components: Sentry
Affects Versions: sentry-ha-redesign
Reporter: Alexander Kolbasov
Assignee: Alexander Kolbasov
Priority: Minor
The FullUpdateInitializer follows the {{MetastoreCacheInitializer}}. It reads a
bunch of information from HMS and uses Thrift structures to pass around, but in
the end it just constructs {{Map}}. It uses concurrent
fetches from HMS, but synchronizes a lot on common data structures to update
them.
I think that we can refactor all this code to make it faster and consume less
memory. The idea is the following:
Use background threads to collect Thrift results from HMS calls (database,
table and partition data). Then we can use a single thread to construct the
resulting update and return it without using intermediate Thrift methods.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (SENTRY-1684) FullUpdateInitializer has a race condition in handling results list
[ https://issues.apache.org/jira/browse/SENTRY-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-1684: --- Status: Patch Available (was: In Progress) > FullUpdateInitializer has a race condition in handling results list > --- > > Key: SENTRY-1684 > URL: https://issues.apache.org/jira/browse/SENTRY-1684 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Attachments: SENTRY-1684.001-sentry-ha-redesign.patch > > > The FullUpdateInitializer class has exactly the same race condition as the > one described in SENTRY-1683. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SENTRY-1684) FullUpdateInitializer has a race condition in handling results list
[ https://issues.apache.org/jira/browse/SENTRY-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-1684: --- Attachment: SENTRY-1684.001-sentry-ha-redesign.patch > FullUpdateInitializer has a race condition in handling results list > --- > > Key: SENTRY-1684 > URL: https://issues.apache.org/jira/browse/SENTRY-1684 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Attachments: SENTRY-1684.001-sentry-ha-redesign.patch > > > The FullUpdateInitializer class has exactly the same race condition as the > one described in SENTRY-1683. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SENTRY-1684) FullUpdateInitializer has a race condition in handling results list
[ https://issues.apache.org/jira/browse/SENTRY-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-1684: --- Attachment: SENTRY-1684.001-sentry-ha-redesign.patch > FullUpdateInitializer has a race condition in handling results list > --- > > Key: SENTRY-1684 > URL: https://issues.apache.org/jira/browse/SENTRY-1684 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Attachments: SENTRY-1684.001-sentry-ha-redesign.patch > > > The FullUpdateInitializer class has exactly the same race condition as the > one described in SENTRY-1683. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1683) MetastoreCacheInitializer has a race condition in handling results list
[
https://issues.apache.org/jira/browse/SENTRY-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951841#comment-15951841
]
Hadoop QA commented on SENTRY-1683:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12861532/SENTRY-1683.001.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/2467/console
This message is automatically generated.
> MetastoreCacheInitializer has a race condition in handling results list
> ---
>
> Key: SENTRY-1683
> URL: https://issues.apache.org/jira/browse/SENTRY-1683
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 1.8.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: SENTRY-1683.001.patch, SENTRY-1683.001.patch
>
>
> The {{MetastoreCacheInitializer}} has the following logic:
> 1) It creates a list of futures to hold results of executors ({{results}})
> 2) {{CreateInitialUpdate() }}does this:
> {code}
> UpdateableAuthzPaths createInitialUpdate() throws
> Exception {
> PathsUpdate tempUpdate = new PathsUpdate(-1, false);
> List allDbStr = hmsHandler.get_all_databases();
> for (String dbName : allDbStr) {
> Callable dbTask = new DbTask(tempUpdate, dbName);
> results.add(threadPool.submit(dbTask));
> }
> while (taskCounter.get() > 0) {
> Thread.sleep(1000);
> // Wait until no more tasks remain
> }
> for (Future result : results) {
> CallResult callResult = result.get();
> // Fail the HMS startup if tasks are not all successful and
> // fail on partial updates flag is set in the config.
> if (!callResult.getSuccessStatus() && failOnRetry) {
> throw callResult.getFailure();
> }
> }
> authzPaths.updatePartial(Lists.newArrayList(tempUpdate),
> new ReentrantReadWriteLock());
> return authzPaths;
> }
> {code}
> Note that in both cases the results list is accessed without holding any
> locks.
> But the list is also updated from tasks themselves:
> {code}
> class DbTask extends BaseTask {
> @Override
> public void doTask() throws TException, SentryMalformedPathException {
> ...
> List allTblStr = hmsHandler.get_all_tables(dbName);
> for (int i = 0; i < allTblStr.size(); i += maxTablesPerCall) {
> synchronized (results) {
> results.add(threadPool.submit(tableTask));
> }
> }
> {code}
> There is some synchronization which uses taskCounter so the second walk of
> the list happens when all tasks are complete, but the first walk isn't
> thread-safe - the list may be updated while initial tasks are created there.
> Note that some access are synchronized and some are not.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SENTRY-1677) Add metrics to measure how much time to get Delta Path/Perm Updates
[
https://issues.apache.org/jira/browse/SENTRY-1677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951814#comment-15951814
]
Hadoop QA commented on SENTRY-1677:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12861521/SENTRY-1677.001-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/2464/console
This message is automatically generated.
> Add metrics to measure how much time to get Delta Path/Perm Updates
> ---
>
> Key: SENTRY-1677
> URL: https://issues.apache.org/jira/browse/SENTRY-1677
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Affects Versions: sentry-ha-redesign
>Reporter: Hao Hao
>Assignee: Alexander Kolbasov
>Priority: Minor
> Labels: bite-sized, metrics, newbie, observability
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1677.001-sentry-ha-redesign.patch
>
>
> It is a follow up of SENTRY-1613. It would be nice to have metrics for
> measuring how much time it takes to get delta perm/path updates at
> PermDeltaRetriever and PathDeltaRetriever.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SENTRY-1548) Setting GrantOption to UNSET upsets Sentry
[
https://issues.apache.org/jira/browse/SENTRY-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951808#comment-15951808
]
Hadoop QA commented on SENTRY-1548:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12861514/SENTRY-1548.001-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.tests.e2e.dbprovider.TestPrivilegeWithGrantOption
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/2466/console
This message is automatically generated.
> Setting GrantOption to UNSET upsets Sentry
> --
>
> Key: SENTRY-1548
> URL: https://issues.apache.org/jira/browse/SENTRY-1548
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 1.8.0, sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
>Priority: Minor
> Labels: bite-sized
> Fix For: 1.8.0
>
> Attachments: SENTRY-1548.001.patch,
> SENTRY-1548.001-sentry-ha-redesign.patch, SENTRY-1548.002.patch,
> SENTRY-1548.003.patch, SENTRY-1548.005.patch, SENTRY-1548.006.patch,
> SENTRY-1548.007.patch, SENTRY-1548.008.patch, SENTRY-1548.009.patch
>
>
> If we send a Thrift request to sentry (using regular api) with GrantOption
> set to UNSET (-1) we get the following error:
> {code}
> TransactionManager.executeTransactionWithRetry(TransactionManager.java:102)]
> The transaction has reac
> hed max retry number, will not retry again.
> javax.jdo.JDODataStoreException: Insert of object
> "org.apache.sentry.provider.db.service.model.MSentryPrivilege@6bbfd4c9" using
> statement "INSERT INTO `SENTRY_DB_PRIVILEGE`
> (`DB_PRIVILEGE_ID`,`SERVER_NAME`,`WITH_GRANT_OPTION`,`CREATE_TIME`,`TABLE_NAME`,`URI`,`ACTION`,`COLUMN_NAME`,`DB_NAME`,`PRIVILEGE_SCOPE`)
> VALUES (?,?,?,?,?,?,?,?,?,?)" failed : Column 'WITH_GRANT_OPTION' cannot be
> null
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:438)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:95)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$8.execute(SentryStore.java:374)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:72)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:93)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:367)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:280)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Column 'WITH_GRANT_OPTION' cannot be null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
[jira] [Commented] (SENTRY-1556) Simplify privilege cleaning
[
https://issues.apache.org/jira/browse/SENTRY-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951802#comment-15951802
]
Hadoop QA commented on SENTRY-1556:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12861507/SENTRY-1556.001-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:red}Overall:{color} -1 due to an error
{color:red}ERROR:{color} mvn test exited 1
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/2465/console
This message is automatically generated.
> Simplify privilege cleaning
> ---
>
> Key: SENTRY-1556
> URL: https://issues.apache.org/jira/browse/SENTRY-1556
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 1.8.0, sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
>Priority: Minor
> Fix For: 1.8.0
>
> Attachments: SENTRY-1556.001-sentry-ha-redesign.patch,
> SENTRY-1556.002.patch, SENTRY-1556.003.patch, SENTRY-1556.004.patch,
> SENTRY-1556.005.patch, SENTRY-1556.006.patch, SENTRY-1556.007.patch,
> SENTRY-1556.008.patch
>
>
> The SentryStore class has a privCleaner that cleans up orphaned privileges.
> Currently cleaning is happening after 50 notification requests are sent and
> it uses locking to synchronize.
> I think the whole thing can be simplified:
> 1) We should consider whether it is possible to clean up a privilege simply
> when we see that there are no roles associated with it. In this case we do
> not need this at all.
> 2) We can simply run a periodic job to clean up orphaned privileges and
> groups (which are not cleaned up at all now).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (SENTRY-1670) Expose current HMS notification ID as a Sentry gauge (metric)
[ https://issues.apache.org/jira/browse/SENTRY-1670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-1670: --- Priority: Minor (was: Major) > Expose current HMS notification ID as a Sentry gauge (metric) > - > > Key: SENTRY-1670 > URL: https://issues.apache.org/jira/browse/SENTRY-1670 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Minor > Labels: bite-sized, metrics, newbie, observability > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1670.001-sentry-ha-redesign.patch > > > The current HMS notification ID is an important value to observe, so we > should expose it via metrics (probably as a gauge) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SENTRY-1610) list_sentry_roles_by_group() semantics for Generic model is broken
[
https://issues.apache.org/jira/browse/SENTRY-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov resolved SENTRY-1610.
Resolution: Not A Problem
Apparently this isn't a bug, although this is rather strange behavior.
> list_sentry_roles_by_group() semantics for Generic model is broken
> --
>
> Key: SENTRY-1610
> URL: https://issues.apache.org/jira/browse/SENTRY-1610
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 1.8.0, sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>
> Looking at SentryGenericPolicyProcessor:
> {code}
> @Override
> public TListSentryRolesResponse list_sentry_roles_by_group(
> final TListSentryRolesRequest request) throws TException {
> Response respose = requestHandle(new
> RequestHandler() {
> @Override
> public Response handle() throws Exception {
> validateClientVersion(request.getProtocol_version());
> // Here we assign groups to the requestor's Unix groups!
> Set groups = getRequestorGroups(conf,
> request.getRequestorUserName());
> if (!AccessConstants.ALL.equalsIgnoreCase(request.getGroupName())) {
> boolean admin = inAdminGroups(groups);
> // Only admin users can list all roles in the system ( groupname =
> null)
> // Non admin users are only allowed to list only groups which they
> belong to
> if(!admin && (request.getGroupName() == null ||
> !groups.contains(request.getGroupName( {
> throw new SentryAccessDeniedException(ACCESS_DENIAL_MESSAGE +
> request.getRequestorUserName());
> }
> groups.clear();
> groups.add(request.getGroupName());
> }
> // And here we use Unix groups if the group is "*"
> Set roleNames =
> store.getRolesByGroups(request.getComponent(), groups);
> ...
> {code}
> What happens here is weird - when the group in the request is ALL ("*"), we
> attempt to return roles for Unix groups that the requestor belongs to, not
> Sentry groups. The problem is that Sentry groups and User groups have nothing
> in common, so this is completely wrong.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (SENTRY-1677) Add metrics to measure how much time to get Delta Path/Perm Updates
[ https://issues.apache.org/jira/browse/SENTRY-1677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-1677: --- Priority: Minor (was: Major) > Add metrics to measure how much time to get Delta Path/Perm Updates > --- > > Key: SENTRY-1677 > URL: https://issues.apache.org/jira/browse/SENTRY-1677 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov >Priority: Minor > Labels: bite-sized, metrics, newbie, observability > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1677.001-sentry-ha-redesign.patch > > > It is a follow up of SENTRY-1613. It would be nice to have metrics for > measuring how much time it takes to get delta perm/path updates at > PermDeltaRetriever and PathDeltaRetriever. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (SENTRY-1579) Sentry should avoid using Proxy
[
https://issues.apache.org/jira/browse/SENTRY-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov reassigned SENTRY-1579:
--
Assignee: (was: Alexander Kolbasov)
> Sentry should avoid using Proxy
> ---
>
> Key: SENTRY-1579
> URL: https://issues.apache.org/jira/browse/SENTRY-1579
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 1.8.0, sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Priority: Minor
>
> Sentry code uses Java java.lang.Reflect.Proxy class in several places:
> {code}SentryHDFSServiceClientFactory{code} uses this to modify behavior in HA
> case.
> {code}SentryServiceClientFactory{code} uses it to modify behavior in
> connection pool or Ha case:
> {code}
> public static SentryPolicyServiceClient create(Configuration conf) throws
> Exception {
> boolean haEnabled = conf.getBoolean(ClientConfig.SERVER_HA_ENABLED,
> false);
> boolean pooled = conf.getBoolean(ClientConfig.SENTRY_POOL_ENABLED, false);
> if (pooled) {
> return (SentryPolicyServiceClient) Proxy
>
> .newProxyInstance(SentryPolicyServiceClientDefaultImpl.class.getClassLoader(),
> SentryPolicyServiceClientDefaultImpl.class.getInterfaces(),
> new PoolClientInvocationHandler(conf));
> } else if (haEnabled) {
> return (SentryPolicyServiceClient) Proxy
>
> .newProxyInstance(SentryPolicyServiceClientDefaultImpl.class.getClassLoader(),
> SentryPolicyServiceClientDefaultImpl.class.getInterfaces(),
> new HAClientInvocationHandler(conf));
> } else {
> return new SentryPolicyServiceClientDefaultImpl(conf);
> }
> }
> {code}
> {code}SentryServiceClientPoolFactory{code} uses it to modify behavior in HA
> case.
> This seems to be an abuse of the Proxy interface and some simpler solution
> should be possible. We should investigate.
> Note that in sentry-ha-redesign branch there is only one use in
> {code}SentryPolicyServiceClient{code} that is used to adjust behavior for
> pooled vs retry behaviors:
> {code}
> public static SentryPolicyServiceClient create(Configuration conf) throws
> Exception {
> boolean pooled = conf.getBoolean(
> ClientConfig.SENTRY_POOL_ENABLED,
> ClientConfig.SENTRY_POOL_ENABLED_DEFAULT);
> if (pooled) {
> return (SentryPolicyServiceClient) Proxy
>
> .newProxyInstance(SentryPolicyServiceClientDefaultImpl.class.getClassLoader(),
> SentryPolicyServiceClientDefaultImpl.class.getInterfaces(),
> new PoolClientInvocationHandler(conf));
> } else {
> return (SentryPolicyServiceClient) Proxy
>
> .newProxyInstance(SentryPolicyServiceClientDefaultImpl.class.getClassLoader(),
> SentryPolicyServiceClientDefaultImpl.class.getInterfaces(),
> new RetryClientInvocationHandler(conf));
> }
> }
> {code}
> Again, a more straightforward solution should be possible.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (SENTRY-1530) Sentry should use to io.dropwizard.metrics
[ https://issues.apache.org/jira/browse/SENTRY-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov reassigned SENTRY-1530: -- Assignee: (was: Alexander Kolbasov) > Sentry should use to io.dropwizard.metrics > -- > > Key: SENTRY-1530 > URL: https://issues.apache.org/jira/browse/SENTRY-1530 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 1.8.0, sentry-ha-redesign >Reporter: Alexander Kolbasov >Priority: Minor > Attachments: SENTRY-1530.001.patch, SENTRY-1530.002.patch > > > The com.codahale.metrics is now io.dropwizard.metrics with current version > being 3.1.0. > We should consider moving Sentry to this version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1626) Tables contain createTime should either remove it or update the time using SQL
[
https://issues.apache.org/jira/browse/SENTRY-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951777#comment-15951777
]
Alexander Kolbasov commented on SENTRY-1626:
One thing to consider is whether it is worth changing the date handling for
*new* tables that are introduced with Sentry HA.
> Tables contain createTime should either remove it or update the time using SQL
> --
>
> Key: SENTRY-1626
> URL: https://issues.apache.org/jira/browse/SENTRY-1626
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Affects Versions: sentry-ha-redesign
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1626.000-sentry-ha-redesign.patch
>
>
> Most of the table in Sentry contains createTime, e.g {{MSentryRole}},
> {{MSentryGroup}}, {{MSentryPermChange}}, {{MSentryPrivilege}},
> {{MAuthzPathsMapping}} and {{MSentryPathChange}}. Even though it is not used,
> it may be error-prone for future.
> The time between leader and standby servers might be different, so that after
> a failover, the newly elected leader might write changes with older
> {{createdTime}}.
> We can either remove this field, or update the time based in the database.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (SENTRY-1683) MetastoreCacheInitializer has a race condition in handling results list
[
https://issues.apache.org/jira/browse/SENTRY-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-1683:
---
Attachment: SENTRY-1683.001.patch
> MetastoreCacheInitializer has a race condition in handling results list
> ---
>
> Key: SENTRY-1683
> URL: https://issues.apache.org/jira/browse/SENTRY-1683
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 1.8.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: SENTRY-1683.001.patch, SENTRY-1683.001.patch
>
>
> The {{MetastoreCacheInitializer}} has the following logic:
> 1) It creates a list of futures to hold results of executors ({{results}})
> 2) {{CreateInitialUpdate() }}does this:
> {code}
> UpdateableAuthzPaths createInitialUpdate() throws
> Exception {
> PathsUpdate tempUpdate = new PathsUpdate(-1, false);
> List allDbStr = hmsHandler.get_all_databases();
> for (String dbName : allDbStr) {
> Callable dbTask = new DbTask(tempUpdate, dbName);
> results.add(threadPool.submit(dbTask));
> }
> while (taskCounter.get() > 0) {
> Thread.sleep(1000);
> // Wait until no more tasks remain
> }
> for (Future result : results) {
> CallResult callResult = result.get();
> // Fail the HMS startup if tasks are not all successful and
> // fail on partial updates flag is set in the config.
> if (!callResult.getSuccessStatus() && failOnRetry) {
> throw callResult.getFailure();
> }
> }
> authzPaths.updatePartial(Lists.newArrayList(tempUpdate),
> new ReentrantReadWriteLock());
> return authzPaths;
> }
> {code}
> Note that in both cases the results list is accessed without holding any
> locks.
> But the list is also updated from tasks themselves:
> {code}
> class DbTask extends BaseTask {
> @Override
> public void doTask() throws TException, SentryMalformedPathException {
> ...
> List allTblStr = hmsHandler.get_all_tables(dbName);
> for (int i = 0; i < allTblStr.size(); i += maxTablesPerCall) {
> synchronized (results) {
> results.add(threadPool.submit(tableTask));
> }
> }
> {code}
> There is some synchronization which uses taskCounter so the second walk of
> the list happens when all tasks are complete, but the first walk isn't
> thread-safe - the list may be updated while initial tasks are created there.
> Note that some access are synchronized and some are not.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (SENTRY-1683) MetastoreCacheInitializer has a race condition in handling results list
[
https://issues.apache.org/jira/browse/SENTRY-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-1683:
---
Status: Open (was: Patch Available)
> MetastoreCacheInitializer has a race condition in handling results list
> ---
>
> Key: SENTRY-1683
> URL: https://issues.apache.org/jira/browse/SENTRY-1683
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 1.8.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: SENTRY-1683.001.patch, SENTRY-1683.001.patch
>
>
> The {{MetastoreCacheInitializer}} has the following logic:
> 1) It creates a list of futures to hold results of executors ({{results}})
> 2) {{CreateInitialUpdate() }}does this:
> {code}
> UpdateableAuthzPaths createInitialUpdate() throws
> Exception {
> PathsUpdate tempUpdate = new PathsUpdate(-1, false);
> List allDbStr = hmsHandler.get_all_databases();
> for (String dbName : allDbStr) {
> Callable dbTask = new DbTask(tempUpdate, dbName);
> results.add(threadPool.submit(dbTask));
> }
> while (taskCounter.get() > 0) {
> Thread.sleep(1000);
> // Wait until no more tasks remain
> }
> for (Future result : results) {
> CallResult callResult = result.get();
> // Fail the HMS startup if tasks are not all successful and
> // fail on partial updates flag is set in the config.
> if (!callResult.getSuccessStatus() && failOnRetry) {
> throw callResult.getFailure();
> }
> }
> authzPaths.updatePartial(Lists.newArrayList(tempUpdate),
> new ReentrantReadWriteLock());
> return authzPaths;
> }
> {code}
> Note that in both cases the results list is accessed without holding any
> locks.
> But the list is also updated from tasks themselves:
> {code}
> class DbTask extends BaseTask {
> @Override
> public void doTask() throws TException, SentryMalformedPathException {
> ...
> List allTblStr = hmsHandler.get_all_tables(dbName);
> for (int i = 0; i < allTblStr.size(); i += maxTablesPerCall) {
> synchronized (results) {
> results.add(threadPool.submit(tableTask));
> }
> }
> {code}
> There is some synchronization which uses taskCounter so the second walk of
> the list happens when all tasks are complete, but the first walk isn't
> thread-safe - the list may be updated while initial tasks are created there.
> Note that some access are synchronized and some are not.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (SENTRY-1683) MetastoreCacheInitializer has a race condition in handling results list
[
https://issues.apache.org/jira/browse/SENTRY-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-1683:
---
Status: Patch Available (was: Open)
> MetastoreCacheInitializer has a race condition in handling results list
> ---
>
> Key: SENTRY-1683
> URL: https://issues.apache.org/jira/browse/SENTRY-1683
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 1.8.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: SENTRY-1683.001.patch, SENTRY-1683.001.patch
>
>
> The {{MetastoreCacheInitializer}} has the following logic:
> 1) It creates a list of futures to hold results of executors ({{results}})
> 2) {{CreateInitialUpdate() }}does this:
> {code}
> UpdateableAuthzPaths createInitialUpdate() throws
> Exception {
> PathsUpdate tempUpdate = new PathsUpdate(-1, false);
> List allDbStr = hmsHandler.get_all_databases();
> for (String dbName : allDbStr) {
> Callable dbTask = new DbTask(tempUpdate, dbName);
> results.add(threadPool.submit(dbTask));
> }
> while (taskCounter.get() > 0) {
> Thread.sleep(1000);
> // Wait until no more tasks remain
> }
> for (Future result : results) {
> CallResult callResult = result.get();
> // Fail the HMS startup if tasks are not all successful and
> // fail on partial updates flag is set in the config.
> if (!callResult.getSuccessStatus() && failOnRetry) {
> throw callResult.getFailure();
> }
> }
> authzPaths.updatePartial(Lists.newArrayList(tempUpdate),
> new ReentrantReadWriteLock());
> return authzPaths;
> }
> {code}
> Note that in both cases the results list is accessed without holding any
> locks.
> But the list is also updated from tasks themselves:
> {code}
> class DbTask extends BaseTask {
> @Override
> public void doTask() throws TException, SentryMalformedPathException {
> ...
> List allTblStr = hmsHandler.get_all_tables(dbName);
> for (int i = 0; i < allTblStr.size(); i += maxTablesPerCall) {
> synchronized (results) {
> results.add(threadPool.submit(tableTask));
> }
> }
> {code}
> There is some synchronization which uses taskCounter so the second walk of
> the list happens when all tasks are complete, but the first walk isn't
> thread-safe - the list may be updated while initial tasks are created there.
> Note that some access are synchronized and some are not.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Reopened] (SENTRY-1548) Setting GrantOption to UNSET upsets Sentry
[
https://issues.apache.org/jira/browse/SENTRY-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kalyan kumar kalvagadda reopened SENTRY-1548:
-
> Setting GrantOption to UNSET upsets Sentry
> --
>
> Key: SENTRY-1548
> URL: https://issues.apache.org/jira/browse/SENTRY-1548
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 1.8.0, sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
>Priority: Minor
> Labels: bite-sized
> Fix For: 1.8.0
>
> Attachments: SENTRY-1548.001.patch,
> SENTRY-1548.001-sentry-ha-redesign.patch, SENTRY-1548.002.patch,
> SENTRY-1548.003.patch, SENTRY-1548.005.patch, SENTRY-1548.006.patch,
> SENTRY-1548.007.patch, SENTRY-1548.008.patch, SENTRY-1548.009.patch
>
>
> If we send a Thrift request to sentry (using regular api) with GrantOption
> set to UNSET (-1) we get the following error:
> {code}
> TransactionManager.executeTransactionWithRetry(TransactionManager.java:102)]
> The transaction has reac
> hed max retry number, will not retry again.
> javax.jdo.JDODataStoreException: Insert of object
> "org.apache.sentry.provider.db.service.model.MSentryPrivilege@6bbfd4c9" using
> statement "INSERT INTO `SENTRY_DB_PRIVILEGE`
> (`DB_PRIVILEGE_ID`,`SERVER_NAME`,`WITH_GRANT_OPTION`,`CREATE_TIME`,`TABLE_NAME`,`URI`,`ACTION`,`COLUMN_NAME`,`DB_NAME`,`PRIVILEGE_SCOPE`)
> VALUES (?,?,?,?,?,?,?,?,?,?)" failed : Column 'WITH_GRANT_OPTION' cannot be
> null
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:438)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:95)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$8.execute(SentryStore.java:374)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:72)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:93)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:367)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:280)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Column 'WITH_GRANT_OPTION' cannot be null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
> at com.mysql.jdbc.Util.getInstance(Util.java:387)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3966)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3902)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2526)
> at
[jira] [Updated] (SENTRY-1548) Setting GrantOption to UNSET upsets Sentry
[
https://issues.apache.org/jira/browse/SENTRY-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kalyan kumar kalvagadda updated SENTRY-1548:
Status: Patch Available (was: Reopened)
> Setting GrantOption to UNSET upsets Sentry
> --
>
> Key: SENTRY-1548
> URL: https://issues.apache.org/jira/browse/SENTRY-1548
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 1.8.0, sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
>Priority: Minor
> Labels: bite-sized
> Fix For: 1.8.0
>
> Attachments: SENTRY-1548.001.patch,
> SENTRY-1548.001-sentry-ha-redesign.patch, SENTRY-1548.002.patch,
> SENTRY-1548.003.patch, SENTRY-1548.005.patch, SENTRY-1548.006.patch,
> SENTRY-1548.007.patch, SENTRY-1548.008.patch, SENTRY-1548.009.patch
>
>
> If we send a Thrift request to sentry (using regular api) with GrantOption
> set to UNSET (-1) we get the following error:
> {code}
> TransactionManager.executeTransactionWithRetry(TransactionManager.java:102)]
> The transaction has reac
> hed max retry number, will not retry again.
> javax.jdo.JDODataStoreException: Insert of object
> "org.apache.sentry.provider.db.service.model.MSentryPrivilege@6bbfd4c9" using
> statement "INSERT INTO `SENTRY_DB_PRIVILEGE`
> (`DB_PRIVILEGE_ID`,`SERVER_NAME`,`WITH_GRANT_OPTION`,`CREATE_TIME`,`TABLE_NAME`,`URI`,`ACTION`,`COLUMN_NAME`,`DB_NAME`,`PRIVILEGE_SCOPE`)
> VALUES (?,?,?,?,?,?,?,?,?,?)" failed : Column 'WITH_GRANT_OPTION' cannot be
> null
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:438)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:95)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$8.execute(SentryStore.java:374)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:72)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:93)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:367)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:280)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Column 'WITH_GRANT_OPTION' cannot be null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
> at com.mysql.jdbc.Util.getInstance(Util.java:387)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3966)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3902)
> at
[jira] [Issue Comment Deleted] (SENTRY-1593) Implement client failover for Generic and NN clients
[ https://issues.apache.org/jira/browse/SENTRY-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-1593: Comment: was deleted (was: I ran the test TestDbCrossDbOps manually and it passed.) > Implement client failover for Generic and NN clients > > > Key: SENTRY-1593 > URL: https://issues.apache.org/jira/browse/SENTRY-1593 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda > Labels: HA > Fix For: sentry-ha-redesign > > Attachments: old_service_client_class_diagram.png, > SENTRY-1593.001-sentry-ha-redesign.patch, > SENTRY-1593.002-sentry-ha-redesign.patch, > SENTRY-1593.003-sentry-ha-redesign.patch, > SENTRY-1593.004-sentry-ha-redesign.patch, > SENTRY-1593.005-sentry-ha-redesign.patch, > SENTRY-1593.006-sentry-ha-redesign.patch, service_client_class_diagram.png > > > We need to have client failover logic for Generic service clients and Name > Node clients. Currently only db policy clients have it implemented. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (SENTRY-1593) Implement client failover for Generic and NN clients
[ https://issues.apache.org/jira/browse/SENTRY-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951706#comment-15951706 ] kalyan kumar kalvagadda edited comment on SENTRY-1593 at 3/31/17 10:05 PM: --- I ran the test TestDbCrossDbOps manually and it passed. was (Author: kkalyan): I ran the test TestDbCrossDbOps and is the test passed. > Implement client failover for Generic and NN clients > > > Key: SENTRY-1593 > URL: https://issues.apache.org/jira/browse/SENTRY-1593 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda > Labels: HA > Fix For: sentry-ha-redesign > > Attachments: old_service_client_class_diagram.png, > SENTRY-1593.001-sentry-ha-redesign.patch, > SENTRY-1593.002-sentry-ha-redesign.patch, > SENTRY-1593.003-sentry-ha-redesign.patch, > SENTRY-1593.004-sentry-ha-redesign.patch, > SENTRY-1593.005-sentry-ha-redesign.patch, > SENTRY-1593.006-sentry-ha-redesign.patch, service_client_class_diagram.png > > > We need to have client failover logic for Generic service clients and Name > Node clients. Currently only db policy clients have it implemented. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1593) Implement client failover for Generic and NN clients
[ https://issues.apache.org/jira/browse/SENTRY-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951706#comment-15951706 ] kalyan kumar kalvagadda commented on SENTRY-1593: - I ran the test TestDbCrossDbOps and is the test passed. > Implement client failover for Generic and NN clients > > > Key: SENTRY-1593 > URL: https://issues.apache.org/jira/browse/SENTRY-1593 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda > Labels: HA > Fix For: sentry-ha-redesign > > Attachments: old_service_client_class_diagram.png, > SENTRY-1593.001-sentry-ha-redesign.patch, > SENTRY-1593.002-sentry-ha-redesign.patch, > SENTRY-1593.003-sentry-ha-redesign.patch, > SENTRY-1593.004-sentry-ha-redesign.patch, > SENTRY-1593.005-sentry-ha-redesign.patch, > SENTRY-1593.006-sentry-ha-redesign.patch, service_client_class_diagram.png > > > We need to have client failover logic for Generic service clients and Name > Node clients. Currently only db policy clients have it implemented. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SENTRY-1677) Add metrics to measure how much time to get Delta Path/Perm Updates
[ https://issues.apache.org/jira/browse/SENTRY-1677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-1677: --- Status: Patch Available (was: In Progress) > Add metrics to measure how much time to get Delta Path/Perm Updates > --- > > Key: SENTRY-1677 > URL: https://issues.apache.org/jira/browse/SENTRY-1677 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov > Labels: bite-sized, metrics, newbie, observability > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1677.001-sentry-ha-redesign.patch > > > It is a follow up of SENTRY-1613. It would be nice to have metrics for > measuring how much time it takes to get delta perm/path updates at > PermDeltaRetriever and PathDeltaRetriever. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SENTRY-1677) Add metrics to measure how much time to get Delta Path/Perm Updates
[ https://issues.apache.org/jira/browse/SENTRY-1677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-1677: --- Attachment: SENTRY-1677.001-sentry-ha-redesign.patch > Add metrics to measure how much time to get Delta Path/Perm Updates > --- > > Key: SENTRY-1677 > URL: https://issues.apache.org/jira/browse/SENTRY-1677 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov > Labels: bite-sized, metrics, newbie, observability > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1677.001-sentry-ha-redesign.patch > > > It is a follow up of SENTRY-1613. It would be nice to have metrics for > measuring how much time it takes to get delta perm/path updates at > PermDeltaRetriever and PathDeltaRetriever. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SENTRY-1548) Setting GrantOption to UNSET upsets Sentry
[
https://issues.apache.org/jira/browse/SENTRY-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kalyan kumar kalvagadda updated SENTRY-1548:
Attachment: SENTRY-1548.001-sentry-ha-redesign.patch
> Setting GrantOption to UNSET upsets Sentry
> --
>
> Key: SENTRY-1548
> URL: https://issues.apache.org/jira/browse/SENTRY-1548
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 1.8.0, sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
>Priority: Minor
> Labels: bite-sized
> Fix For: 1.8.0
>
> Attachments: SENTRY-1548.001.patch,
> SENTRY-1548.001-sentry-ha-redesign.patch, SENTRY-1548.002.patch,
> SENTRY-1548.003.patch, SENTRY-1548.005.patch, SENTRY-1548.006.patch,
> SENTRY-1548.007.patch, SENTRY-1548.008.patch, SENTRY-1548.009.patch
>
>
> If we send a Thrift request to sentry (using regular api) with GrantOption
> set to UNSET (-1) we get the following error:
> {code}
> TransactionManager.executeTransactionWithRetry(TransactionManager.java:102)]
> The transaction has reac
> hed max retry number, will not retry again.
> javax.jdo.JDODataStoreException: Insert of object
> "org.apache.sentry.provider.db.service.model.MSentryPrivilege@6bbfd4c9" using
> statement "INSERT INTO `SENTRY_DB_PRIVILEGE`
> (`DB_PRIVILEGE_ID`,`SERVER_NAME`,`WITH_GRANT_OPTION`,`CREATE_TIME`,`TABLE_NAME`,`URI`,`ACTION`,`COLUMN_NAME`,`DB_NAME`,`PRIVILEGE_SCOPE`)
> VALUES (?,?,?,?,?,?,?,?,?,?)" failed : Column 'WITH_GRANT_OPTION' cannot be
> null
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:438)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:95)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$8.execute(SentryStore.java:374)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:72)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:93)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:367)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:280)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Column 'WITH_GRANT_OPTION' cannot be null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
> at com.mysql.jdbc.Util.getInstance(Util.java:387)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3966)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3902)
> at
[jira] [Updated] (SENTRY-1548) Setting GrantOption to UNSET upsets Sentry
[
https://issues.apache.org/jira/browse/SENTRY-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kalyan kumar kalvagadda updated SENTRY-1548:
Attachment: (was: SENTRY-1548.001-sentry-ha-redesign.patch)
> Setting GrantOption to UNSET upsets Sentry
> --
>
> Key: SENTRY-1548
> URL: https://issues.apache.org/jira/browse/SENTRY-1548
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 1.8.0, sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
>Priority: Minor
> Labels: bite-sized
> Fix For: 1.8.0
>
> Attachments: SENTRY-1548.001.patch, SENTRY-1548.002.patch,
> SENTRY-1548.003.patch, SENTRY-1548.005.patch, SENTRY-1548.006.patch,
> SENTRY-1548.007.patch, SENTRY-1548.008.patch, SENTRY-1548.009.patch
>
>
> If we send a Thrift request to sentry (using regular api) with GrantOption
> set to UNSET (-1) we get the following error:
> {code}
> TransactionManager.executeTransactionWithRetry(TransactionManager.java:102)]
> The transaction has reac
> hed max retry number, will not retry again.
> javax.jdo.JDODataStoreException: Insert of object
> "org.apache.sentry.provider.db.service.model.MSentryPrivilege@6bbfd4c9" using
> statement "INSERT INTO `SENTRY_DB_PRIVILEGE`
> (`DB_PRIVILEGE_ID`,`SERVER_NAME`,`WITH_GRANT_OPTION`,`CREATE_TIME`,`TABLE_NAME`,`URI`,`ACTION`,`COLUMN_NAME`,`DB_NAME`,`PRIVILEGE_SCOPE`)
> VALUES (?,?,?,?,?,?,?,?,?,?)" failed : Column 'WITH_GRANT_OPTION' cannot be
> null
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:438)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:95)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$8.execute(SentryStore.java:374)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:72)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:93)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:367)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:280)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Column 'WITH_GRANT_OPTION' cannot be null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
> at com.mysql.jdbc.Util.getInstance(Util.java:387)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3966)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3902)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2526)
> at
[jira] [Updated] (SENTRY-1556) Simplify privilege cleaning
[ https://issues.apache.org/jira/browse/SENTRY-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-1556: Attachment: SENTRY-1556.001-sentry-ha-redesign.patch > Simplify privilege cleaning > --- > > Key: SENTRY-1556 > URL: https://issues.apache.org/jira/browse/SENTRY-1556 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 1.8.0, sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda >Priority: Minor > Fix For: 1.8.0 > > Attachments: SENTRY-1556.001-sentry-ha-redesign.patch, > SENTRY-1556.002.patch, SENTRY-1556.003.patch, SENTRY-1556.004.patch, > SENTRY-1556.005.patch, SENTRY-1556.006.patch, SENTRY-1556.007.patch, > SENTRY-1556.008.patch > > > The SentryStore class has a privCleaner that cleans up orphaned privileges. > Currently cleaning is happening after 50 notification requests are sent and > it uses locking to synchronize. > I think the whole thing can be simplified: > 1) We should consider whether it is possible to clean up a privilege simply > when we see that there are no roles associated with it. In this case we do > not need this at all. > 2) We can simply run a periodic job to clean up orphaned privileges and > groups (which are not cleaned up at all now). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (SENTRY-1680) MetastoreCacheInitializer is lo longer used and should be removed
[
https://issues.apache.org/jira/browse/SENTRY-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Hentschel reassigned SENTRY-1680:
-
Assignee: Jan Hentschel
> MetastoreCacheInitializer is lo longer used and should be removed
> -
>
> Key: SENTRY-1680
> URL: https://issues.apache.org/jira/browse/SENTRY-1680
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Jan Hentschel
> Labels: bite-sized, newbie
> Fix For: sentry-ha-redesign
>
>
> The {{MetastoreCacheInitializer}} class is not used since SENTRY-1613 and
> should be removed, together with its tests.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SENTRY-1593) Implement client failover for Generic and NN clients
[
https://issues.apache.org/jira/browse/SENTRY-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951457#comment-15951457
]
Hadoop QA commented on SENTRY-1593:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12861474/SENTRY-1593.006-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.tests.e2e.dbprovider.TestDbCrossDbOps
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/2463/console
This message is automatically generated.
> Implement client failover for Generic and NN clients
>
>
> Key: SENTRY-1593
> URL: https://issues.apache.org/jira/browse/SENTRY-1593
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
> Labels: HA
> Fix For: sentry-ha-redesign
>
> Attachments: old_service_client_class_diagram.png,
> SENTRY-1593.001-sentry-ha-redesign.patch,
> SENTRY-1593.002-sentry-ha-redesign.patch,
> SENTRY-1593.003-sentry-ha-redesign.patch,
> SENTRY-1593.004-sentry-ha-redesign.patch,
> SENTRY-1593.005-sentry-ha-redesign.patch,
> SENTRY-1593.006-sentry-ha-redesign.patch, service_client_class_diagram.png
>
>
> We need to have client failover logic for Generic service clients and Name
> Node clients. Currently only db policy clients have it implemented.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SENTRY-1626) Tables contain createTime should either remove it or update the time using SQL
[
https://issues.apache.org/jira/browse/SENTRY-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951368#comment-15951368
]
Lei (Eddy) Xu commented on SENTRY-1626:
---
[~akolb] yea, I agree that we should clean this when the things are settled.
> Tables contain createTime should either remove it or update the time using SQL
> --
>
> Key: SENTRY-1626
> URL: https://issues.apache.org/jira/browse/SENTRY-1626
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Affects Versions: sentry-ha-redesign
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1626.000-sentry-ha-redesign.patch
>
>
> Most of the table in Sentry contains createTime, e.g {{MSentryRole}},
> {{MSentryGroup}}, {{MSentryPermChange}}, {{MSentryPrivilege}},
> {{MAuthzPathsMapping}} and {{MSentryPathChange}}. Even though it is not used,
> it may be error-prone for future.
> The time between leader and standby servers might be different, so that after
> a failover, the newly elected leader might write changes with older
> {{createdTime}}.
> We can either remove this field, or update the time based in the database.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (SENTRY-1682) Investigate use of EXPORT for replication for initial HMS snapshot
[
https://issues.apache.org/jira/browse/SENTRY-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña reassigned SENTRY-1682:
---
Assignee: Sergio Peña
> Investigate use of EXPORT for replication for initial HMS snapshot
> --
>
> Key: SENTRY-1682
> URL: https://issues.apache.org/jira/browse/SENTRY-1682
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
>Reporter: Alexander Kolbasov
>Assignee: Sergio Peña
> Fix For: sentry-ha-redesign
>
>
> [~mohitsabharwal] mentioned that Hive supports
> {code}
> EXPORT ... for replication('somerandomstring')
> {code}
> command,
> It will dump the last notification id that EXPORT sees (in the export
> metadata json).
> i.e. if this works as documented, then EXPORT is telling you what is the last
> notification that EXPORT saw before executing.
> This can be used for ontaining consistent initial snapshot instead of the
> current conservative stop-the0world scheme.
> We should explore this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (SENTRY-1686) Port SENTRY-1548 to sentry-ha-redesign branch
Alexander Kolbasov created SENTRY-1686: -- Summary: Port SENTRY-1548 to sentry-ha-redesign branch Key: SENTRY-1686 URL: https://issues.apache.org/jira/browse/SENTRY-1686 Project: Sentry Issue Type: Sub-task Components: Sentry Reporter: Alexander Kolbasov Assignee: kalyan kumar kalvagadda Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SENTRY-1685) Port SENTRY-1489 to sentry-ha-redesign branch
Alexander Kolbasov created SENTRY-1685: -- Summary: Port SENTRY-1489 to sentry-ha-redesign branch Key: SENTRY-1685 URL: https://issues.apache.org/jira/browse/SENTRY-1685 Project: Sentry Issue Type: Sub-task Components: Sentry Affects Versions: sentry-ha-redesign Reporter: Alexander Kolbasov Priority: Minor We need to port SENTRY-1489 to sentry-ha-redesign branch -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SENTRY-1593) Implement client failover for Generic and NN clients
[ https://issues.apache.org/jira/browse/SENTRY-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-1593: Attachment: SENTRY-1593.006-sentry-ha-redesign.patch > Implement client failover for Generic and NN clients > > > Key: SENTRY-1593 > URL: https://issues.apache.org/jira/browse/SENTRY-1593 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda > Labels: HA > Fix For: sentry-ha-redesign > > Attachments: old_service_client_class_diagram.png, > SENTRY-1593.001-sentry-ha-redesign.patch, > SENTRY-1593.002-sentry-ha-redesign.patch, > SENTRY-1593.003-sentry-ha-redesign.patch, > SENTRY-1593.004-sentry-ha-redesign.patch, > SENTRY-1593.005-sentry-ha-redesign.patch, > SENTRY-1593.006-sentry-ha-redesign.patch, service_client_class_diagram.png > > > We need to have client failover logic for Generic service clients and Name > Node clients. Currently only db policy clients have it implemented. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1593) Implement client failover for Generic and NN clients
[ https://issues.apache.org/jira/browse/SENTRY-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951294#comment-15951294 ] kalyan kumar kalvagadda commented on SENTRY-1593: - At a high level the change set submitted does following. 1. Extends the capability of sentry name-node client and generic policy client to connect to multiple servers and failover to the server that is available 2. Transport handling of the clients is abstracted to another class so that the client's just have to extend the class that abstracts it. Implementations of the clients don't have to worry about it. 3. Changed the behavior of the clients so that they connect to one of the configured servers when the client is actually instantiated. > Implement client failover for Generic and NN clients > > > Key: SENTRY-1593 > URL: https://issues.apache.org/jira/browse/SENTRY-1593 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda > Labels: HA > Fix For: sentry-ha-redesign > > Attachments: old_service_client_class_diagram.png, > SENTRY-1593.001-sentry-ha-redesign.patch, > SENTRY-1593.002-sentry-ha-redesign.patch, > SENTRY-1593.003-sentry-ha-redesign.patch, > SENTRY-1593.004-sentry-ha-redesign.patch, > SENTRY-1593.005-sentry-ha-redesign.patch, service_client_class_diagram.png > > > We need to have client failover logic for Generic service clients and Name > Node clients. Currently only db policy clients have it implemented. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SENTRY-1677) Add metrics to measure how much time to get Delta Path/Perm Updates
[ https://issues.apache.org/jira/browse/SENTRY-1677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-1677: --- Description: It is a follow up of SENTRY-1613. It would be nice to have metrics for measuring how much time it takes to get delta perm/path updates at PermDeltaRetriever and PathDeltaRetriever. (was: It is a follow up of SENTEY-1613. It would be nice to have metrics for measuring how much time it takes to get delta perm/path updates at PermDeltaRetriever and PathDeltaRetriever.) > Add metrics to measure how much time to get Delta Path/Perm Updates > --- > > Key: SENTRY-1677 > URL: https://issues.apache.org/jira/browse/SENTRY-1677 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao > Labels: bite-sized, metrics, newbie, observability > Fix For: sentry-ha-redesign > > > It is a follow up of SENTRY-1613. It would be nice to have metrics for > measuring how much time it takes to get delta perm/path updates at > PermDeltaRetriever and PathDeltaRetriever. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (SENTRY-1677) Add metrics to measure how much time to get Delta Path/Perm Updates
[ https://issues.apache.org/jira/browse/SENTRY-1677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov reassigned SENTRY-1677: -- Assignee: Alexander Kolbasov > Add metrics to measure how much time to get Delta Path/Perm Updates > --- > > Key: SENTRY-1677 > URL: https://issues.apache.org/jira/browse/SENTRY-1677 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov > Labels: bite-sized, metrics, newbie, observability > Fix For: sentry-ha-redesign > > > It is a follow up of SENTRY-1613. It would be nice to have metrics for > measuring how much time it takes to get delta perm/path updates at > PermDeltaRetriever and PathDeltaRetriever. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1626) Tables contain createTime should either remove it or update the time using SQL
[
https://issues.apache.org/jira/browse/SENTRY-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951265#comment-15951265
]
Alexander Kolbasov commented on SENTRY-1626:
I think that the whole date thing is a bit of a mess but it doesn't look like
we *have* to do it now - this can be delayed for the future work.
> Tables contain createTime should either remove it or update the time using SQL
> --
>
> Key: SENTRY-1626
> URL: https://issues.apache.org/jira/browse/SENTRY-1626
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Affects Versions: sentry-ha-redesign
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1626.000-sentry-ha-redesign.patch
>
>
> Most of the table in Sentry contains createTime, e.g {{MSentryRole}},
> {{MSentryGroup}}, {{MSentryPermChange}}, {{MSentryPrivilege}},
> {{MAuthzPathsMapping}} and {{MSentryPathChange}}. Even though it is not used,
> it may be error-prone for future.
> The time between leader and standby servers might be different, so that after
> a failover, the newly elected leader might write changes with older
> {{createdTime}}.
> We can either remove this field, or update the time based in the database.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SENTRY-1626) Tables contain createTime should either remove it or update the time using SQL
[
https://issues.apache.org/jira/browse/SENTRY-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951262#comment-15951262
]
Alexander Kolbasov commented on SENTRY-1626:
How would this work for upgrades? Are we going to change the schema for
existing columns? This will make things backwards incompatible - for example we
wouldn't be able to downgrade the version.
> Tables contain createTime should either remove it or update the time using SQL
> --
>
> Key: SENTRY-1626
> URL: https://issues.apache.org/jira/browse/SENTRY-1626
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Affects Versions: sentry-ha-redesign
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1626.000-sentry-ha-redesign.patch
>
>
> Most of the table in Sentry contains createTime, e.g {{MSentryRole}},
> {{MSentryGroup}}, {{MSentryPermChange}}, {{MSentryPrivilege}},
> {{MAuthzPathsMapping}} and {{MSentryPathChange}}. Even though it is not used,
> it may be error-prone for future.
> The time between leader and standby servers might be different, so that after
> a failover, the newly elected leader might write changes with older
> {{createdTime}}.
> We can either remove this field, or update the time based in the database.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (SENTRY-1592) Implement NN client failover for Sentry HA
[ https://issues.apache.org/jira/browse/SENTRY-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov resolved SENTRY-1592. Resolution: Duplicate The work is done as part of SENTRY-1593. > Implement NN client failover for Sentry HA > -- > > Key: SENTRY-1592 > URL: https://issues.apache.org/jira/browse/SENTRY-1592 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: sentry-ha-redesign >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda > Fix For: sentry-ha-redesign > > Attachments: old_service_client_class_diagram.png, > service_client_class_diagram.png > > > We have failover logic for regular clients but nor for NN client which should > be able to failover to new Sentry server. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1679) HDFS tests configure MetastorePlugin which is gone
[
https://issues.apache.org/jira/browse/SENTRY-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951195#comment-15951195
]
Alexander Kolbasov commented on SENTRY-1679:
The test is likely to be still valid.
> HDFS tests configure MetastorePlugin which is gone
> --
>
> Key: SENTRY-1679
> URL: https://issues.apache.org/jira/browse/SENTRY-1679
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Mathew Crocker
> Labels: bite-sized, newbie
> Fix For: sentry-ha-redesign
>
>
> In TestHDFSIntegrationBase#startHiveAndMetastore() there is:
> {code}
> hiveConf.set("sentry.metastore.plugins",
> "org.apache.sentry.hdfs.MetastorePlugin");
> {code}
> But we no longer have {{MetastorePlugin}}, so this should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (SENTRY-1679) HDFS tests configure MetastorePlugin which is gone
[
https://issues.apache.org/jira/browse/SENTRY-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mathew Crocker reassigned SENTRY-1679:
--
Assignee: Mathew Crocker
> HDFS tests configure MetastorePlugin which is gone
> --
>
> Key: SENTRY-1679
> URL: https://issues.apache.org/jira/browse/SENTRY-1679
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Mathew Crocker
> Labels: bite-sized, newbie
> Fix For: sentry-ha-redesign
>
>
> In TestHDFSIntegrationBase#startHiveAndMetastore() there is:
> {code}
> hiveConf.set("sentry.metastore.plugins",
> "org.apache.sentry.hdfs.MetastorePlugin");
> {code}
> But we no longer have {{MetastorePlugin}}, so this should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SENTRY-1679) HDFS tests configure MetastorePlugin which is gone
[
https://issues.apache.org/jira/browse/SENTRY-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950876#comment-15950876
]
Mathew Crocker commented on SENTRY-1679:
[~akolb] should the whole test be removed ?
> HDFS tests configure MetastorePlugin which is gone
> --
>
> Key: SENTRY-1679
> URL: https://issues.apache.org/jira/browse/SENTRY-1679
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
> Labels: bite-sized, newbie
> Fix For: sentry-ha-redesign
>
>
> In TestHDFSIntegrationBase#startHiveAndMetastore() there is:
> {code}
> hiveConf.set("sentry.metastore.plugins",
> "org.apache.sentry.hdfs.MetastorePlugin");
> {code}
> But we no longer have {{MetastorePlugin}}, so this should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SENTRY-1683) MetastoreCacheInitializer has a race condition in handling results list
[
https://issues.apache.org/jira/browse/SENTRY-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950389#comment-15950389
]
Hadoop QA commented on SENTRY-1683:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12861378/SENTRY-1683.001.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/2462/console
This message is automatically generated.
> MetastoreCacheInitializer has a race condition in handling results list
> ---
>
> Key: SENTRY-1683
> URL: https://issues.apache.org/jira/browse/SENTRY-1683
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 1.8.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: SENTRY-1683.001.patch
>
>
> The {{MetastoreCacheInitializer}} has the following logic:
> 1) It creates a list of futures to hold results of executors ({{results}})
> 2) {{CreateInitialUpdate() }}does this:
> {code}
> UpdateableAuthzPaths createInitialUpdate() throws
> Exception {
> PathsUpdate tempUpdate = new PathsUpdate(-1, false);
> List allDbStr = hmsHandler.get_all_databases();
> for (String dbName : allDbStr) {
> Callable dbTask = new DbTask(tempUpdate, dbName);
> results.add(threadPool.submit(dbTask));
> }
> while (taskCounter.get() > 0) {
> Thread.sleep(1000);
> // Wait until no more tasks remain
> }
> for (Future result : results) {
> CallResult callResult = result.get();
> // Fail the HMS startup if tasks are not all successful and
> // fail on partial updates flag is set in the config.
> if (!callResult.getSuccessStatus() && failOnRetry) {
> throw callResult.getFailure();
> }
> }
> authzPaths.updatePartial(Lists.newArrayList(tempUpdate),
> new ReentrantReadWriteLock());
> return authzPaths;
> }
> {code}
> Note that in both cases the results list is accessed without holding any
> locks.
> But the list is also updated from tasks themselves:
> {code}
> class DbTask extends BaseTask {
> @Override
> public void doTask() throws TException, SentryMalformedPathException {
> ...
> List allTblStr = hmsHandler.get_all_tables(dbName);
> for (int i = 0; i < allTblStr.size(); i += maxTablesPerCall) {
> synchronized (results) {
> results.add(threadPool.submit(tableTask));
> }
> }
> {code}
> There is some synchronization which uses taskCounter so the second walk of
> the list happens when all tasks are complete, but the first walk isn't
> thread-safe - the list may be updated while initial tasks are created there.
> Note that some access are synchronized and some are not.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
