[jira] [Commented] (SENTRY-1876) Filter the result of show databases and show tables in hive

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121012#comment-16121012
 ] 

Na Li commented on SENTRY-1876:
---

[~dongyifeng] Can you create a review at review board 
"https://reviews.apache.org/dashboard/";, select " New Review Request" , and 
attach your patch file. It is much easier to review code there. Thanks!

> Filter the result of show databases and show tables in hive
> ---
>
> Key: SENTRY-1876
> URL: https://issues.apache.org/jira/browse/SENTRY-1876
> Project: Sentry
>  Issue Type: Improvement
>  Components: Hive Binding
>Affects Versions: 1.7.0, 1.8.0, 2.0.0
>Reporter: Yifeng Dong
> Fix For: 2.0.0
>
> Attachments: SENTRY-1876.002.patch, SENTRY-1876.patch
>
>
> Since now sentry only filter the result of show columns.It's very necessary 
> to filter the result of show databases and show tables in hive.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1855) PERM/PATH transactions can fail to commit to the sentry database under load

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1855:
--
Attachment: SENTRY-1855.001.patch

v1 on master. Get perm delta changes using timestamp, not changeID

> PERM/PATH transactions can fail to commit to the sentry database under load
> ---
>
> Key: SENTRY-1855
> URL: https://issues.apache.org/jira/browse/SENTRY-1855
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Alexander Kolbasov
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1855.001.patch, 
> SENTRY-1855.002-sentry-ha-redesign.patch, SENTRY-1855.003-master.patch, 
> SENTRY-1855.01-sentry-ha-redesign.patch
>
>
> Looking at the latest stress runs, we noticed that some of the transactions 
> could fail to commit to the database (with Duplicate key exception) after 
> exhausting all the retries.
> This problem has become more evident if we have more number of clients 
> connecting to Sentry to issue the permission updates. Was able to reproduce 
> consistently with 15 clients doing 100 operations each.
> In the past we introduced exponential backoff (SENTRY-1821) so as part of 
> test run increased the defaults to 750ms sleep and 20 retries. But even after 
> this, the cluster still shows up the transaction failures. This change also 
> increases the latency of every transaction in sentry.
> We need to revisit this and come up with a better way to solve this problem.
> {code}
> 2017-07-13 13:18:14,449 ERROR 
> org.apache.sentry.provider.db.service.persistent.TransactionManager: The 
> transaction has reached max retry number, Exception thrown when executing 
> query
> javax.jdo.JDOException: Exception thrown when executing query
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
>   at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213)
> --
>   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
>   at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>   at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
>   at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
>   at 
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
>   at 
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
>   at 
> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
>   ... 33 more
> 2017-07-13 13:18:14,450 ERROR 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: 
> Unknown error for request: 
> TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805, 
> privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1, 
> dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*, 
> createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The 
> transaction has reached max retry number, Exception thrown when executing 
> query
> java.lang.Exception: The transaction has reached max retry number, Exception 
> thrown when executing query
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:255)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:209)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3330)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:593)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRol

[jira] [Assigned] (SENTRY-1856) Persisting HMS snapshot and the notification-id to database in same transaction

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li reassigned SENTRY-1856:
-

Assignee: Na Li

> Persisting HMS snapshot and the notification-id to database in same 
> transaction
> ---
>
> Key: SENTRY-1856
> URL: https://issues.apache.org/jira/browse/SENTRY-1856
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: kalyan kumar kalvagadda
>Assignee: Na Li
> Fix For: 2.0.0
>
>
> Currently notification-id associated with the snapshot  and the actual HMS 
> snapshot are persisted into database in a separate transactions.
> Ideally they should be updated in same transaction.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1855) PERM/PATH transactions can fail to commit to the sentry database under load

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118537#comment-16118537
 ] 

Na Li commented on SENTRY-1855:
---

Sasha mentioned: Oracle 12c added something called "identity" columns which may 
be usable, but we are not guaranteed to be on Oracle 12c. Also it looks like 
Postgres doesn't support the semantics that we need in their SEQUENCE columns.

> PERM/PATH transactions can fail to commit to the sentry database under load
> ---
>
> Key: SENTRY-1855
> URL: https://issues.apache.org/jira/browse/SENTRY-1855
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Alexander Kolbasov
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1855.002-sentry-ha-redesign.patch, 
> SENTRY-1855.003-master.patch, SENTRY-1855.01-sentry-ha-redesign.patch
>
>
> Looking at the latest stress runs, we noticed that some of the transactions 
> could fail to commit to the database (with Duplicate key exception) after 
> exhausting all the retries.
> This problem has become more evident if we have more number of clients 
> connecting to Sentry to issue the permission updates. Was able to reproduce 
> consistently with 15 clients doing 100 operations each.
> In the past we introduced exponential backoff (SENTRY-1821) so as part of 
> test run increased the defaults to 750ms sleep and 20 retries. But even after 
> this, the cluster still shows up the transaction failures. This change also 
> increases the latency of every transaction in sentry.
> We need to revisit this and come up with a better way to solve this problem.
> {code}
> 2017-07-13 13:18:14,449 ERROR 
> org.apache.sentry.provider.db.service.persistent.TransactionManager: The 
> transaction has reached max retry number, Exception thrown when executing 
> query
> javax.jdo.JDOException: Exception thrown when executing query
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
>   at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213)
> --
>   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
>   at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>   at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
>   at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
>   at 
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
>   at 
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
>   at 
> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
>   ... 33 more
> 2017-07-13 13:18:14,450 ERROR 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: 
> Unknown error for request: 
> TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805, 
> privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1, 
> dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*, 
> createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The 
> transaction has reached max retry number, Exception thrown when executing 
> query
> java.lang.Exception: The transaction has reached max retry number, Exception 
> thrown when executing query
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:255)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:209)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3330)
>   at 
> org.apache.sentry.provider.db.service.

[jira] [Commented] (SENTRY-1855) PERM/PATH transactions can fail to commit to the sentry database under load

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117056#comment-16117056
 ] 

Na Li commented on SENTRY-1855:
---

Hello,

Below is the latest proposed approach for permission change. Path change will 
use the existing approach because there is only one thread to get HMS changes 
and save into DB.

===
Requirements:

+ R1.) from sentry 1.8 to sentry 2.0 "do no harm", such as what's our upgrade 
experience. Performance comparison
+ R2.) Is the approach scalable in terms of supporting multiple Sentry servers 
and concurrent updates
+ R3.) Are the results consistent between Sentry and NN
+ R4.) Does it impact HMS-HA

Constrains from Requirements:

+ C1.) Normal operations should not generate exceptions. The performance of 
Sentry 2.0 should be similar to or better than Sentry 2.0
+ C2.) The system should work reasonable well when having more Sentry servers 
adding into the system and handling concurrent updates
+ C3.) The order Sentry applies the changes should be the same order NN applies 
the changes, so NN will end up with the same result as Sentry and they behave 
consistently

Current Approach for Permission Change:
+ The changeID is primary key and manually updated in Sentry application. The 
current max(changeID) is read from DB, increased by 1, and then used as 
primarykey for new change entry. If more than one threads are saving new change 
entry with the same changeID, the transaction in onethread succeeds, such 
transactions from other transactions fail, and go through retry with 
exponential retry interval.

Benefits of the current approach
+ Satisfies R3.). The changeID is continuously increased. There is no hole in 
changeID. Guaranteed consistency for both deltas and full snapshots.
+ Satisfied R4.). It should not affect HMS-HA
 
Issues with current approach
+ Violates R1.) because normal concurrent transaction could cause transaction 
failure due to key conflict. When transaction fails, exponential retry interval 
causes additional delay for transaction. When load reaches to certain point, 
transaction fails to commit
+ Violates R2.) and it is not scalable. In our endurance test, we see 
transaction failed to commit after max retry with two sentry servers

Proposed Approach for Permission Change:
+ The changeID is primary key and auto-incremented in DataStore. When a 
transaction fails, it creates permanent hole in changeID. When a transaction 
starts early but committed after transactions with larger changeID, it forms 
temporary hole in changeID. Once this transaction is committed, the temporary 
hole disappears. How long a temporary hole can exist is limited by how long a 
transaction can be pending and still be committed successfully.

0) read  from configuration and set to datastore to timeout 
transactions that take longer than that value. In this way, we can limit  how 
long a transaction can be pending and still be committed successfully. 
 =  *  , and its default 
value is 2. This is how long we go back of time to capture temporary holes in 
changeID.
1.) Change the MSentryPermChange timestamp to be assigned by DB, not from 
sentry server. In this way, we have a single source of truth for the timestamp 
of the permission change entries. 
1.a) Need to make sure the timestamp (using @CreateTimestamp annotation in 
datanucleus) is set by DB, not be set by datanucleus using server time. This 
can be done without using @CreateTimestamp if necessary by just doing a 
default value of CURRENT_TIMESTAMP(6) on the database table and not setting 
anything on the object.  If the annotation can handle it, great, but we have to 
make sure of the semantics.
2.) When Sentry sends changes to NN, it also includes the "current time in DB" 
(Now).  
3.) When NN asks for changes, it sends to Sentry the "last received 'current 
time in DB'" (referred as Last_Now) together with the last processed changeID 
for permission and [imageID, changeID] for path changes
4.) When Sentry receives the request, it sends all entries that are newer than 
the time = Last_Now - . NN applies the changes in 
increasing order of changeID

Benefits of the proposed approach:
+ Satisfies requirements R1.), R2.), R3.) and R4.). It should not affect HMS-HA

Issues with Proposed Approach:
+ More complicated to implement than current approach is. 
+ Require change protocol API between NN and Sentry to pass  "current time in 
DB"
+ Need to re-apply some old updates. May cause more overhead, but should be 
reasonable. We can optimize how to reapply old updates. It will be done in next 
release as the change will be local to NN.

Background on R3.)
There are several types of transaction orders: TO1) The order of transaction 
starting time; TO2) The order of permission SQL execution time; TO3) The order 
of transaction commit time; What order affects the result? NN should apply 

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.007-master.patch

v7 again. Trigger build

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-master.patch, 
> SENTRY-1854.001-sentry-ha-redesign.patch, SENTRY-1854.002-master.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, SENTRY-1854.003-master.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, SENTRY-1854.004-master.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, SENTRY-1854.005-master.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, SENTRY-1854.006-master.patch, 
> SENTRY-1854.006-sentry-ha-redesign.patch, SENTRY-1854.007-master.patch, 
> SENTRY-1854.007-master.patch, SENTRY-1854.007-sentry-ha-redesign.patch, 
> SENTRY-1854.008-sentry-ha-redesign.patch, 
> SENTRY-1854.009-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.007-master.patch

v7 on master. no change. just to trigger build.

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-master.patch, 
> SENTRY-1854.001-sentry-ha-redesign.patch, SENTRY-1854.002-master.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, SENTRY-1854.003-master.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, SENTRY-1854.004-master.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, SENTRY-1854.005-master.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, SENTRY-1854.006-master.patch, 
> SENTRY-1854.006-sentry-ha-redesign.patch, SENTRY-1854.007-master.patch, 
> SENTRY-1854.007-sentry-ha-redesign.patch, 
> SENTRY-1854.008-sentry-ha-redesign.patch, 
> SENTRY-1854.009-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.006-master.patch

v6 on master branch. fix test failure

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-master.patch, 
> SENTRY-1854.001-sentry-ha-redesign.patch, SENTRY-1854.002-master.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, SENTRY-1854.003-master.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, SENTRY-1854.004-master.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, SENTRY-1854.005-master.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, SENTRY-1854.006-master.patch, 
> SENTRY-1854.006-sentry-ha-redesign.patch, 
> SENTRY-1854.007-sentry-ha-redesign.patch, 
> SENTRY-1854.008-sentry-ha-redesign.patch, 
> SENTRY-1854.009-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.005-master.patch

v5 on master branch. rebase on latest code

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-master.patch, 
> SENTRY-1854.001-sentry-ha-redesign.patch, SENTRY-1854.002-master.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, SENTRY-1854.003-master.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, SENTRY-1854.004-master.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, SENTRY-1854.005-master.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, 
> SENTRY-1854.006-sentry-ha-redesign.patch, 
> SENTRY-1854.007-sentry-ha-redesign.patch, 
> SENTRY-1854.008-sentry-ha-redesign.patch, 
> SENTRY-1854.009-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.004-master.patch

v4. no change. just to trigger build.

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-master.patch, 
> SENTRY-1854.001-sentry-ha-redesign.patch, SENTRY-1854.002-master.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, SENTRY-1854.003-master.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, SENTRY-1854.004-master.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, 
> SENTRY-1854.006-sentry-ha-redesign.patch, 
> SENTRY-1854.007-sentry-ha-redesign.patch, 
> SENTRY-1854.008-sentry-ha-redesign.patch, 
> SENTRY-1854.009-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109825#comment-16109825
 ] 

Na Li commented on SENTRY-1854:
---

The failure for v2 on master branch is due to out of memory at building machine.
"# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources."

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-master.patch, 
> SENTRY-1854.001-sentry-ha-redesign.patch, SENTRY-1854.002-master.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, SENTRY-1854.003-master.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, 
> SENTRY-1854.006-sentry-ha-redesign.patch, 
> SENTRY-1854.007-sentry-ha-redesign.patch, 
> SENTRY-1854.008-sentry-ha-redesign.patch, 
> SENTRY-1854.009-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.003-master.patch

v3. no change to v2. just to trigger build.

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-master.patch, 
> SENTRY-1854.001-sentry-ha-redesign.patch, SENTRY-1854.002-master.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, SENTRY-1854.003-master.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, 
> SENTRY-1854.006-sentry-ha-redesign.patch, 
> SENTRY-1854.007-sentry-ha-redesign.patch, 
> SENTRY-1854.008-sentry-ha-redesign.patch, 
> SENTRY-1854.009-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.002-master.patch

v2 on master. Update based on Kalyan comments

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-master.patch, 
> SENTRY-1854.001-sentry-ha-redesign.patch, SENTRY-1854.002-master.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, 
> SENTRY-1854.006-sentry-ha-redesign.patch, 
> SENTRY-1854.007-sentry-ha-redesign.patch, 
> SENTRY-1854.008-sentry-ha-redesign.patch, 
> SENTRY-1854.009-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Status: Patch Available  (was: Open)

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-master.patch, 
> SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, 
> SENTRY-1854.006-sentry-ha-redesign.patch, 
> SENTRY-1854.007-sentry-ha-redesign.patch, 
> SENTRY-1854.008-sentry-ha-redesign.patch, 
> SENTRY-1854.009-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.001-master.patch

v1 for master branch. 

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-master.patch, 
> SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, 
> SENTRY-1854.006-sentry-ha-redesign.patch, 
> SENTRY-1854.007-sentry-ha-redesign.patch, 
> SENTRY-1854.008-sentry-ha-redesign.patch, 
> SENTRY-1854.009-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1855) PERM/PATH transactions can fail to commit to the sentry database under load

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1855:
--
Attachment: SENTRY-1855.003-master.patch

v3. approach 3. for hole at front of list, return one entry to indicate there 
is a hole to NN. NN will retry. Once retry exceeds max, skip the missing 
change, move to next one.

> PERM/PATH transactions can fail to commit to the sentry database under load
> ---
>
> Key: SENTRY-1855
> URL: https://issues.apache.org/jira/browse/SENTRY-1855
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Alexander Kolbasov
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1855.002-sentry-ha-redesign.patch, 
> SENTRY-1855.003-master.patch, SENTRY-1855.01-sentry-ha-redesign.patch
>
>
> Looking at the latest stress runs, we noticed that some of the transactions 
> could fail to commit to the database (with Duplicate key exception) after 
> exhausting all the retries.
> This problem has become more evident if we have more number of clients 
> connecting to Sentry to issue the permission updates. Was able to reproduce 
> consistently with 15 clients doing 100 operations each.
> In the past we introduced exponential backoff (SENTRY-1821) so as part of 
> test run increased the defaults to 750ms sleep and 20 retries. But even after 
> this, the cluster still shows up the transaction failures. This change also 
> increases the latency of every transaction in sentry.
> We need to revisit this and come up with a better way to solve this problem.
> {code}
> 2017-07-13 13:18:14,449 ERROR 
> org.apache.sentry.provider.db.service.persistent.TransactionManager: The 
> transaction has reached max retry number, Exception thrown when executing 
> query
> javax.jdo.JDOException: Exception thrown when executing query
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
>   at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213)
> --
>   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
>   at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>   at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
>   at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
>   at 
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
>   at 
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
>   at 
> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
>   ... 33 more
> 2017-07-13 13:18:14,450 ERROR 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: 
> Unknown error for request: 
> TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805, 
> privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1, 
> dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*, 
> createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The 
> transaction has reached max retry number, Exception thrown when executing 
> query
> java.lang.Exception: The transaction has reached max retry number, Exception 
> thrown when executing query
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:255)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:209)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3330)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:5

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.009-sentry-ha-redesign.patch

v9. fix failued tests

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, 
> SENTRY-1854.006-sentry-ha-redesign.patch, 
> SENTRY-1854.007-sentry-ha-redesign.patch, 
> SENTRY-1854.008-sentry-ha-redesign.patch, 
> SENTRY-1854.009-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.008-sentry-ha-redesign.patch

v8. Do not persist path if no HDFS sync

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, 
> SENTRY-1854.006-sentry-ha-redesign.patch, 
> SENTRY-1854.007-sentry-ha-redesign.patch, 
> SENTRY-1854.008-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1855) PERM/PATH transactions can fail to commit to the sentry database under load

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1855:
--
Attachment: SENTRY-1855.002-sentry-ha-redesign.patch

v2. 
1. Make perm changeID to be auto-increased for performance improvement
2. revert change in sentry-1824 to synchronize the perm changes in SentryStore
3. Return continuous delta list starting from requested changeID from 
SentryStore even when the delta list has hole (for example, the list is 
1,2,3,5,6, return 1,2,3). If there is a hole at the front of the list, return 
full snapshot. 

Note: this only applies to perm changes. path changeID is manually increased, 
so has no hole. 
The changeID sent to NN in the full snapshot is in the same transaction of 
getting the full snapshot. 

> PERM/PATH transactions can fail to commit to the sentry database under load
> ---
>
> Key: SENTRY-1855
> URL: https://issues.apache.org/jira/browse/SENTRY-1855
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Alexander Kolbasov
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1855.002-sentry-ha-redesign.patch, 
> SENTRY-1855.01-sentry-ha-redesign.patch
>
>
> Looking at the latest stress runs, we noticed that some of the transactions 
> could fail to commit to the database (with Duplicate key exception) after 
> exhausting all the retries.
> This problem has become more evident if we have more number of clients 
> connecting to Sentry to issue the permission updates. Was able to reproduce 
> consistently with 15 clients doing 100 operations each.
> In the past we introduced exponential backoff (SENTRY-1821) so as part of 
> test run increased the defaults to 750ms sleep and 20 retries. But even after 
> this, the cluster still shows up the transaction failures. This change also 
> increases the latency of every transaction in sentry.
> We need to revisit this and come up with a better way to solve this problem.
> {code}
> 2017-07-13 13:18:14,449 ERROR 
> org.apache.sentry.provider.db.service.persistent.TransactionManager: The 
> transaction has reached max retry number, Exception thrown when executing 
> query
> javax.jdo.JDOException: Exception thrown when executing query
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
>   at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213)
> --
>   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
>   at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>   at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
>   at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
>   at 
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
>   at 
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
>   at 
> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
>   ... 33 more
> 2017-07-13 13:18:14,450 ERROR 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: 
> Unknown error for request: 
> TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805, 
> privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1, 
> dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*, 
> createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The 
> transaction has reached max retry number, Exception thrown when executing 
> query
> java.lang.Exception: The transaction has reached max retry number, Exception 
> thrown when executing query
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(Trans

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.007-sentry-ha-redesign.patch

v7. Check if policy store sync in SentryServiceUnil

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, 
> SENTRY-1854.006-sentry-ha-redesign.patch, 
> SENTRY-1854.007-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (SENTRY-1855) PERM/PATH transactions can fail to commit to the sentry database under load

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li reassigned SENTRY-1855:
-

Assignee: Na Li  (was: Vamsee Yarlagadda)

> PERM/PATH transactions can fail to commit to the sentry database under load
> ---
>
> Key: SENTRY-1855
> URL: https://issues.apache.org/jira/browse/SENTRY-1855
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Alexander Kolbasov
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1855.01-sentry-ha-redesign.patch
>
>
> Looking at the latest stress runs, we noticed that some of the transactions 
> could fail to commit to the database (with Duplicate key exception) after 
> exhausting all the retries.
> This problem has become more evident if we have more number of clients 
> connecting to Sentry to issue the permission updates. Was able to reproduce 
> consistently with 15 clients doing 100 operations each.
> In the past we introduced exponential backoff (SENTRY-1821) so as part of 
> test run increased the defaults to 750ms sleep and 20 retries. But even after 
> this, the cluster still shows up the transaction failures. This change also 
> increases the latency of every transaction in sentry.
> We need to revisit this and come up with a better way to solve this problem.
> {code}
> 2017-07-13 13:18:14,449 ERROR 
> org.apache.sentry.provider.db.service.persistent.TransactionManager: The 
> transaction has reached max retry number, Exception thrown when executing 
> query
> javax.jdo.JDOException: Exception thrown when executing query
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
>   at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213)
> --
>   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
>   at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>   at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
>   at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
>   at 
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
>   at 
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
>   at 
> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
>   ... 33 more
> 2017-07-13 13:18:14,450 ERROR 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: 
> Unknown error for request: 
> TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805, 
> privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1, 
> dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*, 
> createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The 
> transaction has reached max retry number, Exception thrown when executing 
> query
> java.lang.Exception: The transaction has reached max retry number, Exception 
> thrown when executing query
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:255)
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:209)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3330)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:593)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:633)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privi

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.006-sentry-ha-redesign.patch

v6. Same as v5. just to trigger build

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, 
> SENTRY-1854.006-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Status: Patch Available  (was: Open)

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Status: Open  (was: Patch Available)

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: (was: SENTRY-1854.005-sentry-ha-redesign.patch)

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.005-sentry-ha-redesign.patch

v5. Reattch to trigger the build

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: (was: SENTRY-1854.005-sentry-ha-redesign.patch)

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100672#comment-16100672
 ] 

Na Li commented on SENTRY-1854:
---

test failed due to "org.apache.thrift.transport.TTransportException: Could not 
create ServerSocket on address localhost/127.0.0.1:8038." should not be related 
to this code change.

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.005-sentry-ha-redesign.patch

v5. re-attach the patch for build

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1731) java net BindException Address already in use

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100669#comment-16100669
 ] 

Na Li commented on SENTRY-1731:
---

see an exception "Could not create ServerSocket on address 
localhost/127.0.0.1:8038" in apache build 
https://builds.apache.org/job/PreCommit-SENTRY-Build/3093/testReport/org.apache.sentry.tests.e2e.solr.db.integration/TestSolrAdminOperations/org_apache_sentry_tests_e2e_solr_db_integration_TestSolrAdminOperations/

2017-07-25 19:16:51,384 (sentry-service) [ERROR - 
org.apache.sentry.service.thrift.SentryService.call(SentryService.java:221)] 
Error starting server
org.apache.thrift.transport.TTransportException: Could not create ServerSocket 
on address localhost/127.0.0.1:8038.
at 
org.apache.thrift.transport.TServerSocket.(TServerSocket.java:109)
at 
org.apache.thrift.transport.TServerSocket.(TServerSocket.java:91)
at 
org.apache.thrift.transport.TServerSocket.(TServerSocket.java:87)
at 
org.apache.sentry.service.thrift.SentryService.runServer(SentryService.java:270)
at 
org.apache.sentry.service.thrift.SentryService.call(SentryService.java:218)
at 
org.apache.sentry.service.thrift.SentryService.call(SentryService.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

> java net BindException Address already in use
> -
>
> Key: SENTRY-1731
> URL: https://issues.apache.org/jira/browse/SENTRY-1731
> Project: Sentry
>  Issue Type: Bug
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: Sentry-1687 Patch1 Test fail build 2570 Address already 
> in use.zip
>
>
> this unit test fails often. 
> java.net.BindException: Address already in use



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1755) Add HMSFollower per-operation metrics

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100441#comment-16100441
 ] 

Na Li commented on SENTRY-1755:
---

I did not realize it is moved to  SentryHMSClient. So it is not missing.

> Add HMSFollower per-operation metrics
> -
>
> Key: SENTRY-1755
> URL: https://issues.apache.org/jira/browse/SENTRY-1755
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Fix For: 2.0.0
>
> Attachments: SENTRY-1755.01-sentry-ha-redesign.patch
>
>
> We need to get an idea of the time it takes to process HMS notifications on 
> the Sentry side using metrics.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1755) Add HMSFollower per-operation metrics

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100334#comment-16100334
 ] 

Na Li commented on SENTRY-1755:
---

[~akolb] I think some of your code change in SENTRY-1696 ( 
https://reviews.apache.org/r/60806/diff/1#1) is lost  in HMSFollower due to HMS 
follower refector. We don't measure how long does it take to get full snapshot 
any more. Can you add it back to this Jira?



> Add HMSFollower per-operation metrics
> -
>
> Key: SENTRY-1755
> URL: https://issues.apache.org/jira/browse/SENTRY-1755
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Fix For: 2.0.0
>
> Attachments: SENTRY-1755.01-sentry-ha-redesign.patch
>
>
> We need to get an idea of the time it takes to process HMS notifications on 
> the Sentry side using metrics.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.005-sentry-ha-redesign.patch

v5. refactor code to reduce lines.

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch, 
> SENTRY-1854.005-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.004-sentry-ha-redesign.patch

v4. Rebase on latest code

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch, 
> SENTRY-1854.004-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.003-sentry-ha-redesign.patch

v3. update based on Sasha comments

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch, 
> SENTRY-1854.003-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1855) PERM/PATH transactions can fail to commit to the sentry database under load

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096961#comment-16096961
 ] 

Na Li commented on SENTRY-1855:
---

1) Summary

The current approach updates changeID (primary key for permission change and 
path change) manually. In a single transaction, the code reads the max changeID 
from DB, increases it by one, and save the value in new change entry. If two 
threads are adding changes into DB at the same time, collision happens (primary 
key does not allow multiple entries having the same value), and one transaction 
fails. Then the failed transaction goes through multiple re-tries. If retry 
count reaches max value, the transaction fails.

In our stress testing on a single sentry server, with 15 clients doing 
grant/revoke operations concurrently, we saw multiple transaction failures, and 
the exponential-back-off retry increases the latency of every transaction in 
sentry. We have serious performance issue on saving permission and path updates.

2) Potential solutions

2.1) Find out why we have collision on a single sentry server with 
synchronization on saving updates. Once find the cause, fix it.
+ Follow existing approach. Does not introduce big change to the code base. 
- Need time to investigate why synchronization at application level on a single 
sentry server does not prevent key collision.
- Does not scale. All updates are serialized, not much concurrency. 
- Still have key collision exception and transaction failure when more than one 
Sentry servers are deployed. 
- Transaction failure at collision increases time to execute a transaction. 
- It is confusing to customer that there are transaction failure in normal 
operation. Increase support cases

2.2) Auto-increment changeID and send delta changes as much as possible
The patch that achieves 5 times or more performance increase than the current 
approach.
It contains the following changes
revert sentry-1795 (so the changeID is auto-increment. This avoids key 
collision. This is main reason we have performance improvement)
revert sentry-1824 (no need to synchronize when changeID is auto-increment)
get continuous delta list from SentryStore even when the delta list has hole 
(for example, the list is 1,2,3,5,6, return 1,2,3. If the hole is at the front 
of list, return full snapshot)
+ Relative small changes. Verified working with good performance when there is 
no transaction failure
+ When the hole in delta list is temporary (transaction in flight), return the 
continuous delta list is effective to deal with the hole. Most likely, the hole 
will disappear next time HDFS requests for changes.
- When there is transaction failure (the hole in changeID is permanent), sends 
back full snapshot, which is expensive. If we can detect permanent hole, then 
we don't need to send full snapshot, which is very expensive, and may exhaust 
memory for big customer
2.3) Use timestamp to sort the changes

a) use timestamp, such as MSentryPermChange.createTimeMs or 
MSentryPathChange.createTimeMs to sort the entries. If there are more than one 
entry having same timestamp, use changeID to break the tie.
b) HDFS asks for updates using these timestamp values instead of changeID. 
Sentry server sends back changes at and after this timestamp. HDFS keeps the 
list of changeIDs associated with the requesting timestamp and skip entries 
already processed. This is to handle the situation when more than one entry 
having same timestamp, and some are sent in previous request, and some need to 
be send in next request.
c) changeID is primary key, only used to uniquely identify the entry, and not 
required to be sequential nor consecutive.
d) Purge the change entries in DB using timestamp instead of changeID. For 
example, keep 3 polling intervals of entries to allow HDFS getting the changes 
before they are purged.
+ Sentry only sends full snapshot to HDFS at the first time when HDFS starts, 
and then always sends delta changes to HDFS
+ High concurrency. Scale well with large number of clients
- Relative big code change for API between sentry server and sentry plugin at 
HDFS. 
- No easy way to detect that HDFS has received and processed all updates

3) Decisions

My suggestion is that we take approach 2.2) for short term and take the hit of 
full snapshot when there is transaction failure. And take approach 2.3) as long 
term solution.

> PERM/PATH transactions can fail to commit to the sentry database under load
> ---
>
> Key: SENTRY-1855
> URL: https://issues.apache.org/jira/browse/SENTRY-1855
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
> Fix For: sentry-ha-redesign
>
>

[jira] [Updated] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1825:
--
Attachment: SENTRY-1825.008-sentry-ha-redesign.patch

v8. Update based on Sasha comments

> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Na Li
>Priority: Critical
>  Labels: sentry-ha
> Attachments: SENTRY-1825.001-sentry-ha-redesign.patch, 
> SENTRY-1825.002-sentry-ha-redesign.patch, 
> SENTRY-1825.003-sentry-ha-redesign.patch, 
> SENTRY-1825.004-sentry-ha-redesign.patch, 
> SENTRY-1825.005-sentry-ha-redesign.patch, 
> SENTRY-1825.005-sentry-ha-redesign.patch, 
> SENTRY-1825.006-sentry-ha-redesign.patch, 
> SENTRY-1825.007-sentry-ha-redesign.patch, 
> SENTRY-1825.008-sentry-ha-redesign.patch
>
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.002-sentry-ha-redesign.patch

v2. fix test failure in TestSentryStore and add new test

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Na Li
>Assignee: Na Li
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch, 
> SENTRY-1854.002-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Status: Patch Available  (was: Open)

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Na Li
>Assignee: Na Li
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1854) HMSFollower should handle notifications even if HDFS sync is disabled.

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1854:
--
Attachment: SENTRY-1854.001-sentry-ha-redesign.patch

v1.

> HMSFollower should handle notifications even if HDFS sync is disabled.
> --
>
> Key: SENTRY-1854
> URL: https://issues.apache.org/jira/browse/SENTRY-1854
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Na Li
>Assignee: Na Li
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1854.001-sentry-ha-redesign.patch
>
>
> Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
> breaks PERM sync. For example, when a table/database is dropped, its 
> permission should be removed when perm sync is enabled. Without HMSFollower, 
> Sentry does not know when table/database is dropped, and therefore, cannot 
> remove the permission accordingly.
> Sentry needs to make updates to have the follow behavior
> 1) When HDFS sync is disabled + PERM sync is disabled, do not start 
> HMSFollower
> 2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower 
> (Sentry change).
> 3) when HDFS sync is enabled, we save path change and perm change. nothing 
> more. 
> 4) when perm sync is enabled, we update perm when table/database is created, 
> dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1825:
--
Attachment: SENTRY-1825.007-sentry-ha-redesign.patch

v7. rebase on additional commits.

> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Na Li
>Priority: Critical
>  Labels: sentry-ha
> Attachments: SENTRY-1825.001-sentry-ha-redesign.patch, 
> SENTRY-1825.002-sentry-ha-redesign.patch, 
> SENTRY-1825.003-sentry-ha-redesign.patch, 
> SENTRY-1825.004-sentry-ha-redesign.patch, 
> SENTRY-1825.005-sentry-ha-redesign.patch, 
> SENTRY-1825.005-sentry-ha-redesign.patch, 
> SENTRY-1825.006-sentry-ha-redesign.patch, 
> SENTRY-1825.007-sentry-ha-redesign.patch
>
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1825:
--
Attachment: SENTRY-1825.006-sentry-ha-redesign.patch

v6. rebase on latest code

> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Na Li
>Priority: Critical
>  Labels: sentry-ha
> Attachments: SENTRY-1825.001-sentry-ha-redesign.patch, 
> SENTRY-1825.002-sentry-ha-redesign.patch, 
> SENTRY-1825.003-sentry-ha-redesign.patch, 
> SENTRY-1825.004-sentry-ha-redesign.patch, 
> SENTRY-1825.005-sentry-ha-redesign.patch, 
> SENTRY-1825.005-sentry-ha-redesign.patch, 
> SENTRY-1825.006-sentry-ha-redesign.patch
>
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1825:
--
Attachment: SENTRY-1825.005-sentry-ha-redesign.patch

v5 again. 

> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Na Li
>Priority: Critical
>  Labels: sentry-ha
> Attachments: SENTRY-1825.001-sentry-ha-redesign.patch, 
> SENTRY-1825.002-sentry-ha-redesign.patch, 
> SENTRY-1825.003-sentry-ha-redesign.patch, 
> SENTRY-1825.004-sentry-ha-redesign.patch, 
> SENTRY-1825.005-sentry-ha-redesign.patch, 
> SENTRY-1825.005-sentry-ha-redesign.patch
>
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1825:
--
Attachment: SENTRY-1825.005-sentry-ha-redesign.patch

v5. update base on Kalyan's code review

> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Na Li
>Priority: Critical
>  Labels: sentry-ha
> Attachments: SENTRY-1825.001-sentry-ha-redesign.patch, 
> SENTRY-1825.002-sentry-ha-redesign.patch, 
> SENTRY-1825.003-sentry-ha-redesign.patch, 
> SENTRY-1825.004-sentry-ha-redesign.patch, 
> SENTRY-1825.005-sentry-ha-redesign.patch
>
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1760) HMSFollower should detect when a full snapshot from HMS is required

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093577#comment-16093577
 ] 

Na Li commented on SENTRY-1760:
---

I agree. I mentioned the similar idea to Sergio.

> HMSFollower should detect when a full snapshot from HMS is required
> ---
>
> Key: SENTRY-1760
> URL: https://issues.apache.org/jira/browse/SENTRY-1760
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Sergio Peña
>Priority: Blocker
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1760.1-sentry-ha-redesign.patch, 
> SENTRY-1760.2-sentry-ha-redesign.patch, 
> SENTRY-1760.3-sentry-ha-redesign.patch, SENTRY-1760.4-sentry-ha-redesign.patch
>
>
> The current criteria for getting a full snapshot from HMS is insufficient. We 
> need to get full snapshot every time when there is a gap between last stored 
> notification and the smallest notification available from HMS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (SENTRY-1854) HMSFollower should behave based on configuration

[Na Li (JIRA)]

Na Li created SENTRY-1854:
-

 Summary: HMSFollower should behave based on configuration
 Key: SENTRY-1854
 URL: https://issues.apache.org/jira/browse/SENTRY-1854
 Project: Sentry
  Issue Type: Sub-task
  Components: Sentry
Affects Versions: sentry-ha-redesign
Reporter: Na Li
Assignee: Na Li


Currently, Sentry does not start HMSFollower when HDFS sync is disabled. This 
breaks PERM sync. For example, when a table/database is dropped, its permission 
should be removed when perm sync is enabled. Without HMSFollower, Sentry does 
not know when table/database is dropped, and therefore, cannot remove the 
permission accordingly.

Sentry needs to make updates to have the follow behavior
1) When HDFS sync is disabled + PERM sync is disabled, do not start HMSFollower
2) when HDFS sync is enabled or PERM sync is enabled, start HMSFollower (Sentry 
change).
3) when HDFS sync is enabled, we save path change and perm change. nothing 
more. 
4) when perm sync is enabled, we update perm when table/database is created, 
dropped or altered, nothing more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-872) Uber jira for HMS HA + Sentry HA redesign

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-872:
-
Priority: Critical  (was: Major)

> Uber jira for HMS HA + Sentry HA redesign
> -
>
> Key: SENTRY-872
> URL: https://issues.apache.org/jira/browse/SENTRY-872
> Project: Sentry
>  Issue Type: Improvement
>  Components: Hdfs Plugin
>Affects Versions: 1.5.0
>Reporter: Sravya Tirukkovalur
>Assignee: Alexander Kolbasov
>Priority: Critical
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-872.0.patch, SENTRY-872_design.pdf, 
> Sentry-872_design_v2_1.pdf, SENTRY-872_design_v2.pdf, 
> SENTRY-872_dessign-v2.1.1.pdf, SENTRY-872.pdf
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1825:
--
Status: Patch Available  (was: Open)

> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Na Li
>Priority: Critical
>  Labels: sentry-ha
> Attachments: SENTRY-1825.001-sentry-ha-redesign.patch, 
> SENTRY-1825.002-sentry-ha-redesign.patch, 
> SENTRY-1825.003-sentry-ha-redesign.patch, 
> SENTRY-1825.004-sentry-ha-redesign.patch
>
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1825:
--
Attachment: SENTRY-1825.004-sentry-ha-redesign.patch

v4. rebase on latest code

> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Na Li
>Priority: Critical
>  Labels: sentry-ha
> Attachments: SENTRY-1825.001-sentry-ha-redesign.patch, 
> SENTRY-1825.002-sentry-ha-redesign.patch, 
> SENTRY-1825.003-sentry-ha-redesign.patch, 
> SENTRY-1825.004-sentry-ha-redesign.patch
>
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1853) Add the log level access mechanism

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091751#comment-16091751
 ] 

Na Li commented on SENTRY-1853:
---

Donghui,

Can you add the code review link to the Jira case page
https://issues.apache.org/jira/browse/SENTRY-1853
,
so we can review?

Usually, we use review board for apache probject. such as
https://reviews.apache.org/r/59566/.

Thanks,

Lina




> Add the log level access mechanism
> --
>
> Key: SENTRY-1853
> URL: https://issues.apache.org/jira/browse/SENTRY-1853
> Project: Sentry
>  Issue Type: New Feature
>Reporter: Donghui Xu
>Assignee: Donghui Xu
> Attachments: SENTRY-1853.001.patch
>
>
> Add the log level setting and reading mechanism, through which users can 
> control the log content.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1825:
--
Attachment: SENTRY-1825.003-sentry-ha-redesign.patch

v3. update based on Brian's comment

> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Na Li
>Priority: Critical
>  Labels: sentry-ha
> Attachments: SENTRY-1825.001-sentry-ha-redesign.patch, 
> SENTRY-1825.002-sentry-ha-redesign.patch, 
> SENTRY-1825.003-sentry-ha-redesign.patch
>
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1812) Provide interactive Sentry CLI

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090443#comment-16090443
 ] 

Na Li commented on SENTRY-1812:
---

Colm,

Alexander is on vacation right now. He will be back this Wednesday.

Thanks,

Lina

On Mon, Jul 17, 2017 at 5:54 AM, Colm O hEigeartaigh (JIRA)  Provide interactive Sentry CLI
> --
>
> Key: SENTRY-1812
> URL: https://issues.apache.org/jira/browse/SENTRY-1812
> Project: Sentry
>  Issue Type: Improvement
>  Components: Sentry
>Affects Versions: 1.8.0, sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Fix For: 1.8.0
>
> Attachments: SENTRY-1812.01-sentry-ha-redesign.patch
>
>
> It would be quyite useful to have interactive CLI that can be used to access 
> and manipulate Sentry roles, groups and privileges.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (SENTRY-1848) Separate legacy sentry configs from sentry ha configs for api compatibility

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li resolved SENTRY-1848.
---
Resolution: Won't Fix

sentry-ha-redesign will be sentry 2.0. We don't need to maintain the 
compatibility.

> Separate legacy sentry configs from sentry ha configs for api compatibility
> ---
>
> Key: SENTRY-1848
> URL: https://issues.apache.org/jira/browse/SENTRY-1848
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Na Li
>Assignee: Na Li
> Fix For: sentry-ha-redesign
>
>
> sentry.service.client.server.rpc-address has been overloaded with ha 
> awareness which breaks legacy compatibility.
> we'll need to keep the legacy format, and add a new ha setting. The new HA 
> aware sentry client should read the new ha setting



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (SENTRY-1848) Separate legacy sentry configs from sentry ha configs for api compatibility

[Na Li (JIRA)]

Na Li created SENTRY-1848:
-

 Summary: Separate legacy sentry configs from sentry ha configs for 
api compatibility
 Key: SENTRY-1848
 URL: https://issues.apache.org/jira/browse/SENTRY-1848
 Project: Sentry
  Issue Type: Sub-task
  Components: Sentry
Affects Versions: sentry-ha-redesign
Reporter: Na Li
Assignee: Na Li
 Fix For: sentry-ha-redesign


sentry.service.client.server.rpc-address has been overloaded with ha awareness 
which breaks legacy compatibility.
we'll need to keep the legacy format, and add a new ha setting. The new HA 
aware sentry client should read the new ha setting



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1825:
--
Attachment: SENTRY-1825.002-sentry-ha-redesign.patch

v2. Fix failed test case in TestHiveAuthzConf

> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Na Li
>Priority: Critical
>  Labels: sentry-ha
> Attachments: SENTRY-1825.001-sentry-ha-redesign.patch, 
> SENTRY-1825.002-sentry-ha-redesign.patch
>
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1825:
--
Attachment: SENTRY-1825.001-sentry-ha-redesign.patch

v1. get server name from sentry config and add test

> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Na Li
>Priority: Critical
>  Labels: sentry-ha
> Attachments: SENTRY-1825.001-sentry-ha-redesign.patch
>
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1825:
--
Status: Patch Available  (was: In Progress)

> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Na Li
>Priority: Critical
>  Labels: sentry-ha
> Attachments: SENTRY-1825.001-sentry-ha-redesign.patch
>
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1767) Multiple followers should not create full snapshot

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084788#comment-16084788
 ] 

Na Li commented on SENTRY-1767:
---

[~akolb] I have not fully figure out how to achieve that. 

One way is to have a new table in DB whose entry contains 
[CurrentAuthzPathsSnapshotID, time it starts, sentry server ID or name]. Once 
full snapshot is done and persisted, the CurrentAuthzPathsSnapshotID increases, 
and that entry does not take effect and won't stop another sentry server from 
getting a full snapshot.

When a sentry server wants to get full snapshot, if such entry exist for 
CurrentAuthzPathsSnapshotID, not itself, and not old enough ("current time" - 
"time it starts"), it will skip and wait for next run. 

> Multiple followers should not create full snapshot
> --
>
> Key: SENTRY-1767
> URL: https://issues.apache.org/jira/browse/SENTRY-1767
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: kalyan kumar kalvagadda
>Assignee: Na Li
> Fix For: sentry-ha-redesign
>
>
> Suppose that we have two HMSFollower starting up and for some reason they 
> both decided they are masters. They both have initial value for the 
> notification ID and both try to get full snapshot. 
> This should be avoided.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1767) Multiple followers should not create full snapshot

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084154#comment-16084154
 ] 

Na Li commented on SENTRY-1767:
---

[~kkalyan] [~akolb] Is this issue about avoiding two sentry servers get full 
snapshot (avoid the overhead of getting it twice unnecessarily)? Sentry-1815 
only avoid using the same generation ID.


> Multiple followers should not create full snapshot
> --
>
> Key: SENTRY-1767
> URL: https://issues.apache.org/jira/browse/SENTRY-1767
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: kalyan kumar kalvagadda
>Assignee: Na Li
> Fix For: sentry-ha-redesign
>
>
> Suppose that we have two HMSFollower starting up and for some reason they 
> both decided they are masters. They both have initial value for the 
> notification ID and both try to get full snapshot. 
> This should be avoided.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1817) Deprecate SENTRY_HA_ENABLED and all tests that use it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1817:
--
Attachment: SENTRY-1817.007-sentry-ha-redesign.patch

v7. Remove zkServer in InternalSentrySrv

> Deprecate SENTRY_HA_ENABLED and all tests that use it
> -
>
> Key: SENTRY-1817
> URL: https://issues.apache.org/jira/browse/SENTRY-1817
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1817.001-sentry-ha-redesign.patch, 
> SENTRY-1817.002-sentry-ha-redesign.patch, 
> SENTRY-1817.003-sentry-ha-redesign.patch, 
> SENTRY-1817.004-sentry-ha-redesign.patch, 
> SENTRY-1817.005-sentry-ha-redesign.patch, 
> SENTRY-1817.006-sentry-ha-redesign.patch, 
> SENTRY-1817.007-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We need to deprecate old Sentry HA and remove all uses of SENTRY_HA_ENABLED



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1817) Deprecate SENTRY_HA_ENABLED and all tests that use it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1817:
--
Attachment: SENTRY-1817.006-sentry-ha-redesign.patch

v6. Remove zkServer in more places

> Deprecate SENTRY_HA_ENABLED and all tests that use it
> -
>
> Key: SENTRY-1817
> URL: https://issues.apache.org/jira/browse/SENTRY-1817
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1817.001-sentry-ha-redesign.patch, 
> SENTRY-1817.002-sentry-ha-redesign.patch, 
> SENTRY-1817.003-sentry-ha-redesign.patch, 
> SENTRY-1817.004-sentry-ha-redesign.patch, 
> SENTRY-1817.005-sentry-ha-redesign.patch, 
> SENTRY-1817.006-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We need to deprecate old Sentry HA and remove all uses of SENTRY_HA_ENABLED



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (SENTRY-1767) Multiple followers should not create full snapshot

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li reassigned SENTRY-1767:
-

Assignee: Na Li  (was: kalyan kumar kalvagadda)

> Multiple followers should not create full snapshot
> --
>
> Key: SENTRY-1767
> URL: https://issues.apache.org/jira/browse/SENTRY-1767
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: kalyan kumar kalvagadda
>Assignee: Na Li
> Fix For: sentry-ha-redesign
>
>
> Suppose that we have two HMSFollower starting up and for some reason they 
> both decided they are masters. They both have initial value for the 
> notification ID and both try to get full snapshot. 
> This should be avoided.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li reassigned SENTRY-1825:
-

Assignee: Na Li  (was: kalyan kumar kalvagadda)

> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Na Li
>Priority: Critical
>  Labels: sentry-ha
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-854) Refactor SentryGenericProviderBackend to use the client pool for creating client instance

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081125#comment-16081125
 ] 

Na Li commented on SENTRY-854:
--

[~akolb] Is this implemented in SENTRY-1580?

> Refactor SentryGenericProviderBackend to use the client pool for creating 
> client instance
> -
>
> Key: SENTRY-854
> URL: https://issues.apache.org/jira/browse/SENTRY-854
> Project: Sentry
>  Issue Type: Improvement
>Affects Versions: 1.5.0
>Reporter: shenguoquan
>Assignee: shenguoquan
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-627) Extend Sentry HA to work with Hive Metastore HA and HDFS

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081116#comment-16081116
 ] 

Na Li commented on SENTRY-627:
--

[~akolb] Can we close this one? the new HA can work with Hive Metastore HA and 
HDFS. We can have another Jira for adding HA related tests

> Extend Sentry HA to work with Hive Metastore HA and HDFS
> 
>
> Key: SENTRY-627
> URL: https://issues.apache.org/jira/browse/SENTRY-627
> Project: Sentry
>  Issue Type: Improvement
>Affects Versions: 1.5.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Fix For: 2.0.0
>
>
>  Currently the Sentry HA can only work with Single Hive metastore when HDFS 
> integration is enabled. This is the umbrella ticket make that integration 
> work.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1817) Deprecate SENTRY_HA_ENABLED and all tests that use it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1817:
--
Attachment: SENTRY-1817.005-sentry-ha-redesign.patch

v5. Restore two deleted files required by sentry-1708

> Deprecate SENTRY_HA_ENABLED and all tests that use it
> -
>
> Key: SENTRY-1817
> URL: https://issues.apache.org/jira/browse/SENTRY-1817
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1817.001-sentry-ha-redesign.patch, 
> SENTRY-1817.002-sentry-ha-redesign.patch, 
> SENTRY-1817.003-sentry-ha-redesign.patch, 
> SENTRY-1817.004-sentry-ha-redesign.patch, 
> SENTRY-1817.005-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We need to deprecate old Sentry HA and remove all uses of SENTRY_HA_ENABLED



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1731) java net BindException Address already in use

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078628#comment-16078628
 ] 

Na Li commented on SENTRY-1731:
---

This error is no longer observed in recent builds. 
for example from build 2956-2960 
https://builds.apache.org/job/PreCommit-SENTRY-Build/2958/console

> java net BindException Address already in use
> -
>
> Key: SENTRY-1731
> URL: https://issues.apache.org/jira/browse/SENTRY-1731
> Project: Sentry
>  Issue Type: Bug
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Na Li
>Assignee: Na Li
> Attachments: Sentry-1687 Patch1 Test fail build 2570 Address already 
> in use.zip
>
>
> this unit test fails often. 
> java.net.BindException: Address already in use



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1817) Deprecate SENTRY_HA_ENABLED and all tests that use it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1817:
--
Attachment: SENTRY-1817.004-sentry-ha-redesign.patch

v4. remove unused variables and functions.

> Deprecate SENTRY_HA_ENABLED and all tests that use it
> -
>
> Key: SENTRY-1817
> URL: https://issues.apache.org/jira/browse/SENTRY-1817
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1817.001-sentry-ha-redesign.patch, 
> SENTRY-1817.002-sentry-ha-redesign.patch, 
> SENTRY-1817.003-sentry-ha-redesign.patch, 
> SENTRY-1817.004-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We need to deprecate old Sentry HA and remove all uses of SENTRY_HA_ENABLED



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1712) Add capability to force Sentry to send full snapshot to HDFS

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078534#comment-16078534
 ] 

Na Li commented on SENTRY-1712:
---

[~akolb] We can have a function func_A to force HMSFollower to get full 
snapshot from HMS, and another function func_B to force Sentry sending back 
full snapshot to HDFS when HDFS requests for update. We can have a third 
function func_C that calls the func_A. Once Sentry obtains the full snapshot 
from HMS, it calls func_B, so the HDFS can get the latest full snapshot.  

UI calls func_C to accomplish this. 

> Add capability to force Sentry to send full snapshot to HDFS
> 
>
> Key: SENTRY-1712
> URL: https://issues.apache.org/jira/browse/SENTRY-1712
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Na Li
>Assignee: Na Li
>Priority: Minor
> Fix For: sentry-ha-redesign
>
>
> Right now, there is no way to manually force Sentry to get full snapshot. 
> We need to add this capability. That gives us more power to handle error 
> condition and enable testing. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1817) Deprecate SENTRY_HA_ENABLED and all tests that use it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1817:
--
Attachment: SENTRY-1817.003-sentry-ha-redesign.patch

v3. Remove HA related test code.

> Deprecate SENTRY_HA_ENABLED and all tests that use it
> -
>
> Key: SENTRY-1817
> URL: https://issues.apache.org/jira/browse/SENTRY-1817
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1817.001-sentry-ha-redesign.patch, 
> SENTRY-1817.002-sentry-ha-redesign.patch, 
> SENTRY-1817.003-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We need to deprecate old Sentry HA and remove all uses of SENTRY_HA_ENABLED



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li reassigned SENTRY-1825:
-

Assignee: kalyan kumar kalvagadda  (was: Na Li)

> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Critical
>  Labels: sentry-ha
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1817) Deprecate SENTRY_HA_ENABLED and all tests that use it

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076888#comment-16076888
 ] 

Na Li commented on SENTRY-1817:
---

[~akolb]  1) Most of the tests have {{haEnabled == false}}, should I remove the 
code block when HA is enabled in their base class? That will prevent someone 
wants to test when HA is enabled using that base class. 2) 
{{TestPrivilegeWithHAGrantOption}} sets {{haEnabled == true}}. I will keep the 
HA enabled block in its base class {{AbstractTestWithDbProvider}}. Are you OK 
with this approach?


> Deprecate SENTRY_HA_ENABLED and all tests that use it
> -
>
> Key: SENTRY-1817
> URL: https://issues.apache.org/jira/browse/SENTRY-1817
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1817.001-sentry-ha-redesign.patch, 
> SENTRY-1817.002-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We need to deprecate old Sentry HA and remove all uses of SENTRY_HA_ENABLED



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076841#comment-16076841
 ] 

Na Li commented on SENTRY-1825:
---

the issues 1) and 2) will be in sentry-1769 when refectoring HMSFollower.

> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Na Li
>Priority: Critical
>  Labels: sentry-ha
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1825) Dropping a Hive database/table doesn't cleanup the permissions associated with it

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076834#comment-16076834
 ] 

Na Li commented on SENTRY-1825:
---

There are several problems:

1) HMSFollower should read sentry-site.xml to get the server name, not from 
hive-site.xml through hiveConf. The input configuration for HMSFollower 
constructor is from sentry-site.xm. We should use that configuration to get 
server name.
2) There are two variable names that could hold the value of the server. 
{quote}hive.sentry.server{quote} is deprecated and is in 
{quote}HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME_DEPRECATED{quote}. The new 
name is {quote}sentry.hive.server{quote} in 
{quote}HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME{quote}
We should check the {quote}sentry.hive.server{quote} first. If it is not set, 
check the deprecated {quote}hive.sentry.server{quote} to be backward compatible.
3) Whoever generates the sentry-site.xml for hive and sentry needs to make sure 
the variable is set for both host hive and sentry and of the same value. 

[~kkalyan] will make update for items 1) and 2) as he is refectoring 
HMSFollower. I will make sure item 3) is handled by proper team. And add e2e 
test to show the issue and verify the fix. In e2e test, the sentry-site.xml 
will be manually generated in test setup



> Dropping a Hive database/table doesn't cleanup the permissions associated 
> with it 
> --
>
> Key: SENTRY-1825
> URL: https://issues.apache.org/jira/browse/SENTRY-1825
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Na Li
>Priority: Critical
>  Labels: sentry-ha
>
> Sasha helped in finding this bug. Looks like dropping a database/table does 
> no longer clean up the privileges associated with it.
> This problem is because of:
> https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java#L126-L127
> {code}
> final HiveConf hiveConf = new HiveConf();
> hiveInstance = 
> hiveConf.get(HiveAuthzConf.AuthzConfVars.AUTHZ_SERVER_NAME.getVar());
> {code}
> With the latest redesign, we are only setting this property on Hive's 
> (sentry-site.xml) and not on Sentry's (sentry-site.xml).
> So during permission grants, Hive ensures to supply the *server1* for 
> permission updates. But when we drop the table/database that has the perms 
> attached, it goes through HMSFollower and this code sets the property as NULL 
> as sentry-site.xml doesn't have this set. So it attempts to remove 
> permissions with NULL server setting and this always returns without deleting 
> anything. 
> We need to ensure that the corresponding property is set on both (Sentry, 
> Hive) sentry-site.xml to ensure referring to proper privileges. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1817) Deprecate SENTRY_HA_ENABLED and all tests that use it

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076685#comment-16076685
 ] 

Na Li commented on SENTRY-1817:
---

The reason those tests fail is because when I remove "haEnabled", which was 
false in those tests, we are testing new scenarios that "haEnabled = true". I 
need to investigate why those tests fail when sentry HA is enabled.

> Deprecate SENTRY_HA_ENABLED and all tests that use it
> -
>
> Key: SENTRY-1817
> URL: https://issues.apache.org/jira/browse/SENTRY-1817
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1817.001-sentry-ha-redesign.patch, 
> SENTRY-1817.002-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We need to deprecate old Sentry HA and remove all uses of SENTRY_HA_ENABLED



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (SENTRY-1764) HMSFollower should check for leader status after each event processed

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li reassigned SENTRY-1764:
-

Assignee: kalyan kumar kalvagadda  (was: Na Li)

> HMSFollower should check for leader status after each event processed
> -
>
> Key: SENTRY-1764
> URL: https://issues.apache.org/jira/browse/SENTRY-1764
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
>Priority: Critical
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1764.001-sentry-ha-redesign.patch
>
>
> Currently HMSFollower checks the leader status only at the beginning of the 
> run method. This could increase the probability that it will continue 
> processing events after re-election. So it should check after every event is 
> processed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1817) Deprecate SENTRY_HA_ENABLED and all tests that use it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1817:
--
Attachment: SENTRY-1817.002-sentry-ha-redesign.patch

v2. remove more old code related to HA

> Deprecate SENTRY_HA_ENABLED and all tests that use it
> -
>
> Key: SENTRY-1817
> URL: https://issues.apache.org/jira/browse/SENTRY-1817
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1817.001-sentry-ha-redesign.patch, 
> SENTRY-1817.002-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We need to deprecate old Sentry HA and remove all uses of SENTRY_HA_ENABLED



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (SENTRY-1712) Add capability to force Sentry to send full snapshot to HDFS

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li reassigned SENTRY-1712:
-

Assignee: Na Li

> Add capability to force Sentry to send full snapshot to HDFS
> 
>
> Key: SENTRY-1712
> URL: https://issues.apache.org/jira/browse/SENTRY-1712
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Na Li
>Assignee: Na Li
>Priority: Minor
> Fix For: sentry-ha-redesign
>
>
> Right now, there is no way to manually force Sentry to get full snapshot. 
> We need to add this capability. That gives us more power to handle error 
> condition and enable testing. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1824) SentryStore may serialize transactions that rely on unique key

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1824:
--
Attachment: SENTRY-1824.004-sentry-ha-redesign.patch

v4. update comment based on Sasha input.

> SentryStore may serialize transactions that rely on unique key
> --
>
> Key: SENTRY-1824
> URL: https://issues.apache.org/jira/browse/SENTRY-1824
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1824.001-sentry-ha-redesign.patch, 
> SENTRY-1824.002-sentry-ha-redesign.patch, 
> SENTRY-1824.003-sentry-ha-redesign.patch, 
> SENTRY-1824.004-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> The suggestion came from [~lina.li] during the review for SENTRY-1821.
> We have several places where we rely on transactions to support 
> read/modify/write semantics for incrementing IDs. This works but using DB 
> support is rather expensive and we can user in-core serializations to help 
> with this at least within a single node and rely on DB for multi-node 
> synchronization. 
> This isn't much of a problem for path updates since they are driven by 
> HMSFollower which usually runs on a single leader node, but permission 
> updates originate from clients directly and may be highly concurrent.
> We are internally serializing all permissions update anyway, so doing partial 
> serialization on every node helps.
> The patch serializes all calls that affect permission updates.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1824) SentryStore may serialize transactions that rely on unique key

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1824:
--
Attachment: SENTRY-1824.003-sentry-ha-redesign.patch

v3. update comment format

> SentryStore may serialize transactions that rely on unique key
> --
>
> Key: SENTRY-1824
> URL: https://issues.apache.org/jira/browse/SENTRY-1824
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1824.001-sentry-ha-redesign.patch, 
> SENTRY-1824.002-sentry-ha-redesign.patch, 
> SENTRY-1824.003-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> The suggestion came from [~lina.li] during the review for SENTRY-1821.
> We have several places where we rely on transactions to support 
> read/modify/write semantics for incrementing IDs. This works but using DB 
> support is rather expensive and we can user in-core serializations to help 
> with this at least within a single node and rely on DB for multi-node 
> synchronization.
> 1) The DeltaTransactionBlock gets latest changeID and then increases it for 
> this update.
>   Since changeID is the primary key of persisted update, when two or more 
> transactions are
>   updating changeID at the same time, only one transaction succeeds, and the 
> other transactions
>   will fail, and go through retry. It is quite expensive, and cause 
> performance issue.
>  When more clients are making updates, transaction may fail to commit after 
> retries.
>  2)Permission updates are the main issue, getting worse with more clients.
>   Path update is from one thread in HMSFollower, independent on the number of 
> clients, so it is not a big issue.
>  3) Adding synchronization to serilize the updates can minimize the chance a 
> transaction fails
>  4) Therefore, synchronizing all permission updates. In this way, when more 
> than one threads are updating the 
>  permission, only one thread will update changeID. When it finishes, the next 
> thread on the same Sentry server
>  will make update. As a result, no transaction will fail if there is only one 
> Sentry server. When there are more
>  Sentry servers, the collision will happen among the Sentry servers, not 
> depending on how many clients are
> updating Sentry, so it is more scalable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1824) SentryStore may serialize transactions that rely on unique key

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1824:
--
Description: 
The suggestion came from [~lina.li] during the review for SENTRY-1821.

We have several places where we rely on transactions to support 
read/modify/write semantics for incrementing IDs. This works but using DB 
support is rather expensive and we can user in-core serializations to help with 
this at least within a single node and rely on DB for multi-node 
synchronization.

1) The DeltaTransactionBlock gets latest changeID and then increases it for 
this update.
  Since changeID is the primary key of persisted update, when two or more 
transactions are
  updating changeID at the same time, only one transaction succeeds, and the 
other transactions
  will fail, and go through retry. It is quite expensive, and cause performance 
issue.
 When more clients are making updates, transaction may fail to commit after 
retries.
 2)Permission updates are the main issue, getting worse with more clients.
  Path update is from one thread in HMSFollower, independent on the number of 
clients, so it is not a big issue.
 3) Adding synchronization to serilize the updates can minimize the chance a 
transaction fails
 4) Therefore, synchronizing all permission updates. In this way, when more 
than one threads are updating the 
 permission, only one thread will update changeID. When it finishes, the next 
thread on the same Sentry server
 will make update. As a result, no transaction will fail if there is only one 
Sentry server. When there are more
 Sentry servers, the collision will happen among the Sentry servers, not 
depending on how many clients are
updating Sentry, so it is more scalable.

  was:
The suggestion came from [~lina.li] during the review for SENTRY-1821.

We have several places where we rely on transactions to support 
read/modify/write semantics for incrementing IDs. This works but using DB 
support is rather expensive and we can user in-core serializations to help with 
this at least within a single node and rely on DB for multi-node 
synchronization.


> SentryStore may serialize transactions that rely on unique key
> --
>
> Key: SENTRY-1824
> URL: https://issues.apache.org/jira/browse/SENTRY-1824
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1824.001-sentry-ha-redesign.patch, 
> SENTRY-1824.002-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> The suggestion came from [~lina.li] during the review for SENTRY-1821.
> We have several places where we rely on transactions to support 
> read/modify/write semantics for incrementing IDs. This works but using DB 
> support is rather expensive and we can user in-core serializations to help 
> with this at least within a single node and rely on DB for multi-node 
> synchronization.
> 1) The DeltaTransactionBlock gets latest changeID and then increases it for 
> this update.
>   Since changeID is the primary key of persisted update, when two or more 
> transactions are
>   updating changeID at the same time, only one transaction succeeds, and the 
> other transactions
>   will fail, and go through retry. It is quite expensive, and cause 
> performance issue.
>  When more clients are making updates, transaction may fail to commit after 
> retries.
>  2)Permission updates are the main issue, getting worse with more clients.
>   Path update is from one thread in HMSFollower, independent on the number of 
> clients, so it is not a big issue.
>  3) Adding synchronization to serilize the updates can minimize the chance a 
> transaction fails
>  4) Therefore, synchronizing all permission updates. In this way, when more 
> than one threads are updating the 
>  permission, only one thread will update changeID. When it finishes, the next 
> thread on the same Sentry server
>  will make update. As a result, no transaction will fail if there is only one 
> Sentry server. When there are more
>  Sentry servers, the collision will happen among the Sentry servers, not 
> depending on how many clients are
> updating Sentry, so it is more scalable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1824) SentryStore may serialize transactions that rely on unique key

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1824:
--
Attachment: SENTRY-1824.002-sentry-ha-redesign.patch

v2. Add comment to explain the synchronization

> SentryStore may serialize transactions that rely on unique key
> --
>
> Key: SENTRY-1824
> URL: https://issues.apache.org/jira/browse/SENTRY-1824
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1824.001-sentry-ha-redesign.patch, 
> SENTRY-1824.002-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> The suggestion came from [~lina.li] during the review for SENTRY-1821.
> We have several places where we rely on transactions to support 
> read/modify/write semantics for incrementing IDs. This works but using DB 
> support is rather expensive and we can user in-core serializations to help 
> with this at least within a single node and rely on DB for multi-node 
> synchronization.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1824) SentryStore may serialize transactions that rely on unique key

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1824:
--
Status: Patch Available  (was: In Progress)

> SentryStore may serialize transactions that rely on unique key
> --
>
> Key: SENTRY-1824
> URL: https://issues.apache.org/jira/browse/SENTRY-1824
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1824.001-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> The suggestion came from [~lina.li] during the review for SENTRY-1821.
> We have several places where we rely on transactions to support 
> read/modify/write semantics for incrementing IDs. This works but using DB 
> support is rather expensive and we can user in-core serializations to help 
> with this at least within a single node and rely on DB for multi-node 
> synchronization.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1824) SentryStore may serialize transactions that rely on unique key

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1824:
--
Attachment: SENTRY-1824.001-sentry-ha-redesign.patch

v1. add "synchronized" for each SentryStore function that update permissions.

> SentryStore may serialize transactions that rely on unique key
> --
>
> Key: SENTRY-1824
> URL: https://issues.apache.org/jira/browse/SENTRY-1824
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1824.001-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> The suggestion came from [~lina.li] during the review for SENTRY-1821.
> We have several places where we rely on transactions to support 
> read/modify/write semantics for incrementing IDs. This works but using DB 
> support is rather expensive and we can user in-core serializations to help 
> with this at least within a single node and rely on DB for multi-node 
> synchronization.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1787) Remove use of unnecessary SentryMetastorePostEventListenerNotificationLog class

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066844#comment-16066844
 ] 

Na Li commented on SENTRY-1787:
---

The failure is because you did not sync the latest code, so the patch cannot be 
applied. You need to get the latest code, rebase your code change, and then 
generate the patch file to trigger the build.

ERROR: failed to apply patch (exit code 1):
The patch does not appear to apply with p0, p1, or p2

> Remove use of unnecessary SentryMetastorePostEventListenerNotificationLog 
> class
> ---
>
> Key: SENTRY-1787
> URL: https://issues.apache.org/jira/browse/SENTRY-1787
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Sergio Peña
>Assignee: Venkatesh Ramasamy
>Priority: Minor
>  Labels: bite-sized, newbie
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1787.001.patch, 
> SENTRY-1787.01-sentry-ha-redesign.patch
>
>
> The SentryMetastorePostEventListenerNotificationLog class is not used anymore 
> on Sentry. We should remove this from the codebase.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1821) Transactions could fail to commit to the database under load

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066773#comment-16066773
 ] 

Na Li commented on SENTRY-1821:
---

Even with fuzzy logic at client retry, when there are many clients, the 
collision still happens. It gets worse when more clients are updating sentry. 
The solution is not scalable.

It would be the best if we can minimize transaction failure when two or more 
transactions are making updates. Permission updates are the main issue, getting 
worse with more clients. Path update is from one thread in HMSFollower, 
independent on the number of clients, so it is not a big issue.

Can we add synchronization in DeltaTransactionBlock.persistUpdate? In this way, 
when more than one threads are updating the permission, only one thread will 
update changeID, when it finishes, the next thread on the same Sentry server 
will make update. In this way, no transaction will fail if there is only one 
Sentry server. When there are more Sentry servers, the collision will happen 
among the Sentry servers, not depending on how many clients are updating 
Sentry, so it is more scalable. 

> Transactions could fail to commit to the database under load
> 
>
> Key: SENTRY-1821
> URL: https://issues.apache.org/jira/browse/SENTRY-1821
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Vamsee Yarlagadda
>Assignee: Alexander Kolbasov
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1821.01-sentry-ha-redesign.patch
>
>
> I notice this problem when trying to issue some GRANT commands where the 
> transactions fail after exhausting the number of retries to commit to the 
> database. 
> Current load conditions:
> * 20 parallel clients each issuing a GRANT command on MariaDB backed Sentry 
> server.
> Even with this relatively small load, we came across this issue which could 
> get become a major problem under high load. Looking at the retry logic, we do 
> have static wait before retrying but most probably all the clients are waking 
> up at the same time which keeps hitting the same problem and thus exhausts 
> all the retries. 
> Instead of a static wait, we can think of implementing an exponential backoff 
> as a way to avoid this.
> {code}
> Caused by: java.lang.RuntimeException: Unknown error for request: 
> TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:hive, roleName:gr_role_2017_06_27_16_48_10_0_0_8, 
> privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1, 
> dbName:gr_db_2017_06_27_16_48_10, tableName:, URI:, action:*, 
> createTime:1498607304833, grantOption:FALSE, columnName:)]), message: The 
> transaction has reached max retry number, Exception thrown when executing 
> query. Server Stacktrace: java.lang.Exception: The transaction has reached 
> max retry number, Exception thrown when executing query
>   at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:224)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3272)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:589)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:629)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:256)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:997)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:982)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:36)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: javax.jdo.JDOException: Exception thrown when executing query
> NestedThrowables:
> java.sql.BatchUpdateException: Duplicate entry 

[jira] [Updated] (SENTRY-1817) Deprecate SENTRY_HA_ENABLED and all tests that use it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1817:
--
Attachment: SENTRY-1817.001-sentry-ha-redesign.patch

v1. remove usage of SENTRY_HA_ENABLED

> Deprecate SENTRY_HA_ENABLED and all tests that use it
> -
>
> Key: SENTRY-1817
> URL: https://issues.apache.org/jira/browse/SENTRY-1817
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1817.001-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We need to deprecate old Sentry HA and remove all uses of SENTRY_HA_ENABLED



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1817) Deprecate SENTRY_HA_ENABLED and all tests that use it

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1817:
--
Status: Patch Available  (was: In Progress)

> Deprecate SENTRY_HA_ENABLED and all tests that use it
> -
>
> Key: SENTRY-1817
> URL: https://issues.apache.org/jira/browse/SENTRY-1817
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1817.001-sentry-ha-redesign.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We need to deprecate old Sentry HA and remove all uses of SENTRY_HA_ENABLED



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1400) [Flaky test] TestSentryWebServerWithSSL times out

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1400:
--
   Resolution: Cannot Reproduce
Fix Version/s: sentry-ha-redesign
   Status: Resolved  (was: Patch Available)

I have checked the test results of past available builds (build 2913-2920, for 
example https://builds.apache.org/job/PreCommit-SENTRY-Build/2918/console), and 
I don't see this error at all. It seems this issue does not exist anymore.

> [Flaky test] TestSentryWebServerWithSSL times out
> -
>
> Key: SENTRY-1400
> URL: https://issues.apache.org/jira/browse/SENTRY-1400
> Project: Sentry
>  Issue Type: Test
>Affects Versions: sentry-ha-redesign
>Reporter: Sravya Tirukkovalur
>Assignee: Na Li
>  Labels: test
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1400.001-sentry-ha-redesign.patch, 
> SENTRY-1400.001-sentry-ha-redesign.patch, Sentry-1649 Patch30 Test fail build 
> 2569 a.zip
>
>
> I atleast saw two precommit jobs which failed due to this:
> https://builds.apache.org/job/PreCommit-SENTRY-Build/1872/console
> https://builds.apache.org/job/PreCommit-SENTRY-Build/1874/console
> [~lili] any idea whats going on?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1764) HMSFollower should check for leader status after each event processed

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1764:
--
Status: Patch Available  (was: In Progress)

> HMSFollower should check for leader status after each event processed
> -
>
> Key: SENTRY-1764
> URL: https://issues.apache.org/jira/browse/SENTRY-1764
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>Priority: Critical
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1764.001-sentry-ha-redesign.patch
>
>
> Currently HMSFollower checks the leader status only at the beginning of the 
> run method. This could increase the probability that it will continue 
> processing events after re-election. So it should check after every event is 
> processed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SENTRY-1764) HMSFollower should check for leader status after each event processed

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li updated SENTRY-1764:
--
Attachment: SENTRY-1764.001-sentry-ha-redesign.patch

v1. add check after each event processed

> HMSFollower should check for leader status after each event processed
> -
>
> Key: SENTRY-1764
> URL: https://issues.apache.org/jira/browse/SENTRY-1764
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>Priority: Critical
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1764.001-sentry-ha-redesign.patch
>
>
> Currently HMSFollower checks the leader status only at the beginning of the 
> run method. This could increase the probability that it will continue 
> processing events after re-election. So it should check after every event is 
> processed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1750) HMSFollower does not handle view update correctly

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065158#comment-16065158
 ] 

Na Li commented on SENTRY-1750:
---

[~akolb] Sentry does not need to do anything for a view event. Sentry-1669 made 
the code change that does not throw exception when location of an event is null 
(a possible view event). So the behavior is correct now. 

Before, I have disabled the test "TestHDFSIntegrationEnd2End.testViews" because 
it causes other tests to fail (the view event caused exception and then 
following events are not processed, and this issue is tracked in SENTRY-1752). 

Since the problem is fixed in Sentry-1669, the only thing left is to enable the 
test, which is enabled in SENTRY-1793. 

Therefore, I close this issue without any code change.

> HMSFollower does not handle view update correctly
> -
>
> Key: SENTRY-1750
> URL: https://issues.apache.org/jira/browse/SENTRY-1750
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Na Li
>Assignee: Na Li
>Priority: Minor
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
>
> The location of a view is null, and it should be accepted by Sentry. However, 
> when there is view update such "create view testView as select * from test" 
> in TestHDFSIntegrationEnd2End.testViews, the location is null, and 
> HMSFollower throws exception when processing this notification.
> The notification uses HCatEventMessage.EventType. Should we add types for 
> view? So sentry can process view and table differently.
> The call stack of exception
> 2017-05-04 23:24:00,198 (pool-5-thread-1) [ERROR - 
> org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:300)] 
> Encounter SentryInvalidInputException|SentryInvalidHMSEventException while 
> processing notification log
> org.apache.sentry.core.common.exception.SentryInvalidHMSEventException: 
> Create table event has incomplete information. dbName = default, tableName = 
> testView, location = null
>   at 
> org.apache.sentry.service.thrift.HMSFollower.processNotificationEvents(HMSFollower.java:415)
>   at 
> org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:287)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (SENTRY-1296) Flaky test: TestPrivilegeOperatePersistence.testGrantPrivilegeExternalComponentInvalidConf

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li resolved SENTRY-1296.
---
   Resolution: Cannot Reproduce
Fix Version/s: sentry-ha-redesign

 I have checked the test results of past available builds (build 2913-2918, for 
example https://builds.apache.org/job/PreCommit-SENTRY-Build/2918/console), and 
theTestPrivilegeOperatePersistence.testGrantPrivilegeExternalComponentInvalidConf
 did not fail at all. It seems this issue does not exist anymore.

> Flaky test: 
> TestPrivilegeOperatePersistence.testGrantPrivilegeExternalComponentInvalidConf
> --
>
> Key: SENTRY-1296
> URL: https://issues.apache.org/jira/browse/SENTRY-1296
> Project: Sentry
>  Issue Type: Test
>Reporter: Sravya Tirukkovalur
>Assignee: Na Li
>Priority: Blocker
> Fix For: sentry-ha-redesign
>
>
> Seems like above test is failing very frequently in the past few days.
> output from log:
> org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence.testGrantPrivilegeExternalComponentInvalidConf
> Failing for the past 1 build (Since Failed#2613 )
> Took 1 sec.
> Error Message
> Expected exception: java.lang.Exception
> Stacktrace
> java.lang.AssertionError: Expected exception: java.lang.Exception
> Standard Output



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (SENTRY-1750) HMSFollower does not handle view update correctly

[Na Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Li resolved SENTRY-1750.
---
Resolution: Fixed

The test case is enabled in "SENTRY-1793: Reenable ignored unit tests from 
TestHDFSIntegrationEnd2End". So close this issue.

> HMSFollower does not handle view update correctly
> -
>
> Key: SENTRY-1750
> URL: https://issues.apache.org/jira/browse/SENTRY-1750
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Na Li
>Assignee: Na Li
>Priority: Minor
>  Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
>
> The location of a view is null, and it should be accepted by Sentry. However, 
> when there is view update such "create view testView as select * from test" 
> in TestHDFSIntegrationEnd2End.testViews, the location is null, and 
> HMSFollower throws exception when processing this notification.
> The notification uses HCatEventMessage.EventType. Should we add types for 
> view? So sentry can process view and table differently.
> The call stack of exception
> 2017-05-04 23:24:00,198 (pool-5-thread-1) [ERROR - 
> org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:300)] 
> Encounter SentryInvalidInputException|SentryInvalidHMSEventException while 
> processing notification log
> org.apache.sentry.core.common.exception.SentryInvalidHMSEventException: 
> Create table event has incomplete information. dbName = default, tableName = 
> testView, location = null
>   at 
> org.apache.sentry.service.thrift.HMSFollower.processNotificationEvents(HMSFollower.java:415)
>   at 
> org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:287)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1712) Add capability to force Sentry to get full snapshot from HMS

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065084#comment-16065084
 ] 

Na Li commented on SENTRY-1712:
---

SENTRY-1815 covers the second item "automatically detects when name node is out 
of sync with sentry, and returns full snapshot to Name Node's request."

> Add capability to force Sentry to get full snapshot from HMS
> 
>
> Key: SENTRY-1712
> URL: https://issues.apache.org/jira/browse/SENTRY-1712
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Na Li
>Priority: Minor
> Fix For: sentry-ha-redesign
>
>
> Right now, there is no way to manually force Sentry to get full snapshot. 
> We need to add this capability. That gives us more power to handle error 
> condition and enable testing. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1815) Send new HMS snapshots to HDFS requesting an old generation ID

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065080#comment-16065080
 ] 

Na Li commented on SENTRY-1815:
---

[~spena] will this cover the scenario in Sentry-1784? Will this cover the 
scenario that some changes are removed before NN gets them?

> Send new HMS snapshots to HDFS requesting an old generation ID
> --
>
> Key: SENTRY-1815
> URL: https://issues.apache.org/jira/browse/SENTRY-1815
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Fix For: sentry-ha-redesign
>
>
> This is part of the work for SENTRY-1760. 
> A new DB definition is already created that contains a generation ID for new 
> HMS snapshots (SENTRY-1805). This new generation ID is persisted and code is 
> implemented to read and update paths with a current generation ID 
> (SENTRY-1781).
> This patch will these new HMS snapshots to HDFS whenever HDFS requests an old 
> generation ID.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1712) Add capability to force Sentry to get full snapshot from HMS

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065073#comment-16065073
 ] 

Na Li commented on SENTRY-1712:
---

[~akolb] I talked with Sergio. He is working on SENTRY-1760, which does the 
following tasks:
1) automatically detects when sentry is out of sync with HMS, and should get 
full snapshot from HMS
2) automatically detects when name node is out of sync with sentry, and returns 
full snapshot to Name Node's request.

Once the above is done, do we need to provide API to allow user manually 
trigger the full snapshot? i.e., do we still need sentry-1712 and sentry-1520?

> Add capability to force Sentry to get full snapshot from HMS
> 
>
> Key: SENTRY-1712
> URL: https://issues.apache.org/jira/browse/SENTRY-1712
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Na Li
>Priority: Minor
> Fix For: sentry-ha-redesign
>
>
> Right now, there is no way to manually force Sentry to get full snapshot. 
> We need to add this capability. That gives us more power to handle error 
> condition and enable testing. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1760) HMSFollower should detect when a full snapshot from HMS is required

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065068#comment-16065068
 ] 

Na Li commented on SENTRY-1760:
---

[~spena] Will this update include automatically detecting when the Name Node is 
out of sync with sentry, so sentry will send full snapshot to NN?

> HMSFollower should detect when a full snapshot from HMS is required
> ---
>
> Key: SENTRY-1760
> URL: https://issues.apache.org/jira/browse/SENTRY-1760
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Alexander Kolbasov
>Assignee: Sergio Peña
>Priority: Blocker
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1760.1-sentry-ha-redesign.patch
>
>
> The current criteria for getting a full snapshot from HMS is insufficient. We 
> need to get full snapshot every time when there is a gap between last stored 
> notification and the smallest notification available from HMS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1712) Add capability to force Sentry to get full snapshot from HMS

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063793#comment-16063793
 ] 

Na Li commented on SENTRY-1712:
---

Both sentry-1712 and sentry-1520 add capability to force getting full HMS 
snapshot.

However sentry-1520 is about to force sentry to return full HMS snapshot to 
Name Node at HDFS SentryPlugin.

sentry-1712 is about to force sentry to get HMS full snapshot from HMS 
metastore.

They work at different stages



> Add capability to force Sentry to get full snapshot from HMS
> 
>
> Key: SENTRY-1712
> URL: https://issues.apache.org/jira/browse/SENTRY-1712
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Sentry
>Affects Versions: sentry-ha-redesign
>Reporter: Na Li
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: sentry-ha-redesign
>
>
> Right now, there is no way to manually force Sentry to get full snapshot. 
> We need to add this capability. That gives us more power to handle error 
> condition and enable testing. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SENTRY-1693) HMSFollower should handle adding a view with empty path.

[Na Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SENTRY-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063596#comment-16063596
 ] 

Na Li commented on SENTRY-1693:
---

the current behavior of HMSFollower is that when the location is null, the 
event is skipped and error message is logged. It does not throw exception any 
more.

if ((oldDbName == null) ||
  (oldTableName == null) ||
  (newDbName == null) ||
  (newTableName == null) ||
  (oldLocation == null) ||
  (newLocation == null)) {
  isNotificationProcessingSkipped = true;
  LOGGER.error(String.format("Alter table event " +
  "has incomplete information. oldDbName = %s, oldTableName = 
%s, oldLocation = %s, " +
  "newDbName = %s, newTableName = %s, newLocation = %s",
StringUtils.defaultIfBlank(oldDbName, "null"),
StringUtils.defaultIfBlank(oldTableName, "null"),
StringUtils.defaultIfBlank(oldLocation, "null"),
StringUtils.defaultIfBlank(newDbName, "null"),
StringUtils.defaultIfBlank(newTableName, "null"),
StringUtils.defaultIfBlank(newLocation, "null")));
  break;



> HMSFollower should handle adding a view with empty path. 
> -
>
> Key: SENTRY-1693
> URL: https://issues.apache.org/jira/browse/SENTRY-1693
> Project: Sentry
>  Issue Type: Sub-task
>Affects Versions: sentry-ha-redesign
>Reporter: kalyan kumar kalvagadda
>Assignee: Na Li
>Priority: Critical
> Fix For: sentry-ha-redesign
>
>
> When a view is created in Hive, notification log generated would have empty 
> path. HMS Follower throws an exception on such notifications. HMS Follower 
> doesn't update the notification-id and will continue getting same 
> notification again and again. As a result HMS will not be able to process 
> subsequent notifications. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)