[jira] [Commented] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[ https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401422#comment-16401422 ] Arjun Mishra commented on SENTRY-2183: -- That is true. It is purely empirical. Cases with million partitions took about 10 mins. Sometimes they take much longer. We can't factor in all cases. It is significantly larger than 3.3mins which is very conservative. I can run some tests to get numbers on the threshold time to number of hive objects > Increase default sentry-hdfs rpc timeout to 20 mins > --- > > Key: SENTRY-2183 > URL: https://issues.apache.org/jira/browse/SENTRY-2183 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2183.001.patch, SENTRY-2183.002.patch, > SENTRY-2183.01.patch > > > Full HMS Snapshots for large amounts of data (millions of databases, or > tables or partitions etc) take a lot more time than the current default of > 3.3mins. Also ideally since a full HMS snapshot should only occur once it is > best to increase the default to 20mins. This highly reduce the chance of > failure while sending a full snapshot which could be critical > Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-1855) Improve scalability of permission delta updates
[
https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-1855:
---
Affects Version/s: 2.1.0
> Improve scalability of permission delta updates
> ---
>
> Key: SENTRY-1855
> URL: https://issues.apache.org/jira/browse/SENTRY-1855
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-1855.001.patch,
> SENTRY-1855.002-sentry-ha-redesign.patch, SENTRY-1855.002.patch,
> SENTRY-1855.003-master.patch, SENTRY-1855.01-sentry-ha-redesign.patch
>
>
> Looking at the latest stress runs, we noticed that some of the transactions
> could fail to commit to the database (with Duplicate key exception) after
> exhausting all the retries.
> This problem has become more evident if we have more number of clients
> connecting to Sentry to issue the permission updates. Was able to reproduce
> consistently with 15 clients doing 100 operations each.
> In the past we introduced exponential backoff (SENTRY-1821) so as part of
> test run increased the defaults to 750ms sleep and 20 retries. But even after
> this, the cluster still shows up the transaction failures. This change also
> increases the latency of every transaction in sentry.
> We need to revisit this and come up with a better way to solve this problem.
> {code}
> 2017-07-13 13:18:14,449 ERROR
> org.apache.sentry.provider.db.service.persistent.TransactionManager: The
> transaction has reached max retry number, Exception thrown when executing
> query
> javax.jdo.JDOException: Exception thrown when executing query
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213)
> --
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
> at
> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
> ... 33 more
> 2017-07-13 13:18:14,450 ERROR
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor:
> Unknown error for request:
> TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805,
> privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1,
> dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*,
> createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The
> transaction has reached max retry number, Exception thrown when executing
> query
> java.lang.Exception: The transaction has reached max retry number, Exception
> thrown when executing query
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:255)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:209)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3330)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:593)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:633)
> at
>
[jira] [Commented] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[ https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401143#comment-16401143 ] Alexander Kolbasov commented on SENTRY-2183: Why do you think that 20 mins is a good number? What if it takes 25 mins? > Increase default sentry-hdfs rpc timeout to 20 mins > --- > > Key: SENTRY-2183 > URL: https://issues.apache.org/jira/browse/SENTRY-2183 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2183.001.patch, SENTRY-2183.002.patch, > SENTRY-2183.01.patch > > > Full HMS Snapshots for large amounts of data (millions of databases, or > tables or partitions etc) take a lot more time than the current default of > 3.3mins. Also ideally since a full HMS snapshot should only occur once it is > best to increase the default to 20mins. This highly reduce the chance of > failure while sending a full snapshot which could be critical > Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[
https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401118#comment-16401118
]
Hadoop QA commented on SENTRY-2183:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12914760/SENTRY-2183.002.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3701/console
This message is automatically generated.
> Increase default sentry-hdfs rpc timeout to 20 mins
> ---
>
> Key: SENTRY-2183
> URL: https://issues.apache.org/jira/browse/SENTRY-2183
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Arjun Mishra
>Assignee: Arjun Mishra
>Priority: Major
> Attachments: SENTRY-2183.001.patch, SENTRY-2183.002.patch,
> SENTRY-2183.01.patch
>
>
> Full HMS Snapshots for large amounts of data (millions of databases, or
> tables or partitions etc) take a lot more time than the current default of
> 3.3mins. Also ideally since a full HMS snapshot should only occur once it is
> best to increase the default to 20mins. This highly reduce the chance of
> failure while sending a full snapshot which could be critical
> Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[ https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Mishra updated SENTRY-2183: - Attachment: SENTRY-2183.002.patch > Increase default sentry-hdfs rpc timeout to 20 mins > --- > > Key: SENTRY-2183 > URL: https://issues.apache.org/jira/browse/SENTRY-2183 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2183.001.patch, SENTRY-2183.002.patch, > SENTRY-2183.01.patch > > > Full HMS Snapshots for large amounts of data (millions of databases, or > tables or partitions etc) take a lot more time than the current default of > 3.3mins. Also ideally since a full HMS snapshot should only occur once it is > best to increase the default to 20mins. This highly reduce the chance of > failure while sending a full snapshot which could be critical > Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[ https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Mishra updated SENTRY-2183: - Attachment: SENTRY-2183.001.patch > Increase default sentry-hdfs rpc timeout to 20 mins > --- > > Key: SENTRY-2183 > URL: https://issues.apache.org/jira/browse/SENTRY-2183 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2183.001.patch, SENTRY-2183.01.patch > > > Full HMS Snapshots for large amounts of data (millions of databases, or > tables or partitions etc) take a lot more time than the current default of > 3.3mins. Also ideally since a full HMS snapshot should only occur once it is > best to increase the default to 20mins. This highly reduce the chance of > failure while sending a full snapshot which could be critical > Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[
https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400886#comment-16400886
]
Arjun Mishra commented on SENTRY-2183:
--
Test passes locally. Will re-submit the same patch
{noformat}
$ mvn -f sentry-tests/sentry-tests-hive/pom.xml test
-Dtest=TestHmsNotificationProcessingWithOutSyncOnDrop
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running
org.apache.sentry.tests.e2e.dbprovider.TestHmsNotificationProcessingWithOutSyncOnDrop
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.931 s
- in
org.apache.sentry.tests.e2e.dbprovider.TestHmsNotificationProcessingWithOutSyncOnDrop
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 47.291 s
[INFO] Finished at: 2018-03-15T23:10:22+05:30
[INFO] Final Memory: 85M/836M
[INFO]
{noformat}
> Increase default sentry-hdfs rpc timeout to 20 mins
> ---
>
> Key: SENTRY-2183
> URL: https://issues.apache.org/jira/browse/SENTRY-2183
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Arjun Mishra
>Assignee: Arjun Mishra
>Priority: Major
> Attachments: SENTRY-2183.01.patch
>
>
> Full HMS Snapshots for large amounts of data (millions of databases, or
> tables or partitions etc) take a lot more time than the current default of
> 3.3mins. Also ideally since a full HMS snapshot should only occur once it is
> best to increase the default to 20mins. This highly reduce the chance of
> failure while sending a full snapshot which could be critical
> Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[
https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400788#comment-16400788
]
Hadoop QA commented on SENTRY-2183:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12914717/SENTRY-2183.01.patch
against master.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.tests.e2e.dbprovider.TestHmsNotificationProcessingWithOutSyncOnDrop
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3700/console
This message is automatically generated.
> Increase default sentry-hdfs rpc timeout to 20 mins
> ---
>
> Key: SENTRY-2183
> URL: https://issues.apache.org/jira/browse/SENTRY-2183
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Arjun Mishra
>Assignee: Arjun Mishra
>Priority: Major
> Attachments: SENTRY-2183.01.patch
>
>
> Full HMS Snapshots for large amounts of data (millions of databases, or
> tables or partitions etc) take a lot more time than the current default of
> 3.3mins. Also ideally since a full HMS snapshot should only occur once it is
> best to increase the default to 20mins. This highly reduce the chance of
> failure while sending a full snapshot which could be critical
> Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[ https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Mishra updated SENTRY-2183: - Status: Patch Available (was: Open) > Increase default sentry-hdfs rpc timeout to 20 mins > --- > > Key: SENTRY-2183 > URL: https://issues.apache.org/jira/browse/SENTRY-2183 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2183.01.patch > > > Full HMS Snapshots for large amounts of data (millions of databases, or > tables or partitions etc) take a lot more time than the current default of > 3.3mins. Also ideally since a full HMS snapshot should only occur once it is > best to increase the default to 20mins. This highly reduce the chance of > failure while sending a full snapshot which could be critical > Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[ https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Mishra updated SENTRY-2183: - Attachment: SENTRY-2183.01.patch > Increase default sentry-hdfs rpc timeout to 20 mins > --- > > Key: SENTRY-2183 > URL: https://issues.apache.org/jira/browse/SENTRY-2183 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2183.01.patch > > > Full HMS Snapshots for large amounts of data (millions of databases, or > tables or partitions etc) take a lot more time than the current default of > 3.3mins. Also ideally since a full HMS snapshot should only occur once it is > best to increase the default to 20mins. This highly reduce the chance of > failure while sending a full snapshot which could be critical > Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[ https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Mishra updated SENTRY-2183: - Description: Full HMS Snapshots for large amounts of data (millions of databases, or tables or partitions etc) take a lot more time than the current default of 3.3mins. Also ideally since a full HMS snapshot should only occur once it is best to increase the default to 20mins. This highly reduce the chance of failure while sending a full snapshot which could be critical Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120 was:Full HMS Snapshots for large amounts of data (millions of databases, or tables or partitions etc) take a lot more time than the current default of 3.3mins. Also ideally since a full HMS snapshot should only occur once it is best to increase the default to 20mins. This highly reduce the chance of failure while sending a full snapshot which could be critical > Increase default sentry-hdfs rpc timeout to 20 mins > --- > > Key: SENTRY-2183 > URL: https://issues.apache.org/jira/browse/SENTRY-2183 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > > Full HMS Snapshots for large amounts of data (millions of databases, or > tables or partitions etc) take a lot more time than the current default of > 3.3mins. Also ideally since a full HMS snapshot should only occur once it is > best to increase the default to 20mins. This highly reduce the chance of > failure while sending a full snapshot which could be critical > Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
Arjun Mishra created SENTRY-2183: Summary: Increase default sentry-hdfs rpc timeout to 20 mins Key: SENTRY-2183 URL: https://issues.apache.org/jira/browse/SENTRY-2183 Project: Sentry Issue Type: Improvement Components: Sentry Affects Versions: 2.1.0 Reporter: Arjun Mishra Assignee: Arjun Mishra Full HMS Snapshots for large amounts of data (millions of databases, or tables or partitions etc) take a lot more time than the current default of 3.3mins. Also ideally since a full HMS snapshot should only occur once it is best to increase the default to 20mins. This highly reduce the chance of failure while sending a full snapshot which could be critical -- This message was sent by Atlassian JIRA (v7.6.3#76005)
