[jira] [Commented] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[ https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417611#comment-16417611 ] Na Li commented on SENTRY-2183: --- [~arjunmishra13] I agree. Having a relatively large timeout as default would not hurt small users, and help avoid issues for large users. Otherwise, the large users will encounter issues, and then spend a lot of time to figure out that they need to increase this value. > Increase default sentry-hdfs rpc timeout to 20 mins > --- > > Key: SENTRY-2183 > URL: https://issues.apache.org/jira/browse/SENTRY-2183 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2183.001.patch, SENTRY-2183.002.patch, > SENTRY-2183.01.patch > > > Full HMS Snapshots for large amounts of data (millions of databases, or > tables or partitions etc) take a lot more time than the current default of > 3.3mins. Also ideally since a full HMS snapshot should only occur once it is > best to increase the default to 20mins. This highly reduce the chance of > failure while sending a full snapshot which could be critical > Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[ https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417081#comment-16417081 ] Arjun Mishra commented on SENTRY-2183: -- I think we should go with 20 mins. It doesn't affect performance. If small users take 3.3 mins this configuration won't hurt them. Full updates is usually large and it makes sense to have a larger default threshold than smaller. Let me know what you think. > Increase default sentry-hdfs rpc timeout to 20 mins > --- > > Key: SENTRY-2183 > URL: https://issues.apache.org/jira/browse/SENTRY-2183 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2183.001.patch, SENTRY-2183.002.patch, > SENTRY-2183.01.patch > > > Full HMS Snapshots for large amounts of data (millions of databases, or > tables or partitions etc) take a lot more time than the current default of > 3.3mins. Also ideally since a full HMS snapshot should only occur once it is > best to increase the default to 20mins. This highly reduce the chance of > failure while sending a full snapshot which could be critical > Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[ https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415808#comment-16415808 ] Sergio Peña commented on SENTRY-2183: - [~arjunmishra13] After thinking this for a while, I wonder if we really need this change? The timeout will vary based on the amount of data users use. For small users, 3.3min is enough, for large users, 20 or 30min is enough, for others 10min is enough, so being this a configuration value that can be customized on the sentry-site.xml file, then I don't think we should try to change the value. What do you think? > Increase default sentry-hdfs rpc timeout to 20 mins > --- > > Key: SENTRY-2183 > URL: https://issues.apache.org/jira/browse/SENTRY-2183 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2183.001.patch, SENTRY-2183.002.patch, > SENTRY-2183.01.patch > > > Full HMS Snapshots for large amounts of data (millions of databases, or > tables or partitions etc) take a lot more time than the current default of > 3.3mins. Also ideally since a full HMS snapshot should only occur once it is > best to increase the default to 20mins. This highly reduce the chance of > failure while sending a full snapshot which could be critical > Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[ https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401422#comment-16401422 ] Arjun Mishra commented on SENTRY-2183: -- That is true. It is purely empirical. Cases with million partitions took about 10 mins. Sometimes they take much longer. We can't factor in all cases. It is significantly larger than 3.3mins which is very conservative. I can run some tests to get numbers on the threshold time to number of hive objects > Increase default sentry-hdfs rpc timeout to 20 mins > --- > > Key: SENTRY-2183 > URL: https://issues.apache.org/jira/browse/SENTRY-2183 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2183.001.patch, SENTRY-2183.002.patch, > SENTRY-2183.01.patch > > > Full HMS Snapshots for large amounts of data (millions of databases, or > tables or partitions etc) take a lot more time than the current default of > 3.3mins. Also ideally since a full HMS snapshot should only occur once it is > best to increase the default to 20mins. This highly reduce the chance of > failure while sending a full snapshot which could be critical > Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[ https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401143#comment-16401143 ] Alexander Kolbasov commented on SENTRY-2183: Why do you think that 20 mins is a good number? What if it takes 25 mins? > Increase default sentry-hdfs rpc timeout to 20 mins > --- > > Key: SENTRY-2183 > URL: https://issues.apache.org/jira/browse/SENTRY-2183 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2183.001.patch, SENTRY-2183.002.patch, > SENTRY-2183.01.patch > > > Full HMS Snapshots for large amounts of data (millions of databases, or > tables or partitions etc) take a lot more time than the current default of > 3.3mins. Also ideally since a full HMS snapshot should only occur once it is > best to increase the default to 20mins. This highly reduce the chance of > failure while sending a full snapshot which could be critical > Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[
https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401118#comment-16401118
]
Hadoop QA commented on SENTRY-2183:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12914760/SENTRY-2183.002.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3701/console
This message is automatically generated.
> Increase default sentry-hdfs rpc timeout to 20 mins
> ---
>
> Key: SENTRY-2183
> URL: https://issues.apache.org/jira/browse/SENTRY-2183
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Arjun Mishra
>Assignee: Arjun Mishra
>Priority: Major
> Attachments: SENTRY-2183.001.patch, SENTRY-2183.002.patch,
> SENTRY-2183.01.patch
>
>
> Full HMS Snapshots for large amounts of data (millions of databases, or
> tables or partitions etc) take a lot more time than the current default of
> 3.3mins. Also ideally since a full HMS snapshot should only occur once it is
> best to increase the default to 20mins. This highly reduce the chance of
> failure while sending a full snapshot which could be critical
> Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[
https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400886#comment-16400886
]
Arjun Mishra commented on SENTRY-2183:
--
Test passes locally. Will re-submit the same patch
{noformat}
$ mvn -f sentry-tests/sentry-tests-hive/pom.xml test
-Dtest=TestHmsNotificationProcessingWithOutSyncOnDrop
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running
org.apache.sentry.tests.e2e.dbprovider.TestHmsNotificationProcessingWithOutSyncOnDrop
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.931 s
- in
org.apache.sentry.tests.e2e.dbprovider.TestHmsNotificationProcessingWithOutSyncOnDrop
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 47.291 s
[INFO] Finished at: 2018-03-15T23:10:22+05:30
[INFO] Final Memory: 85M/836M
[INFO]
{noformat}
> Increase default sentry-hdfs rpc timeout to 20 mins
> ---
>
> Key: SENTRY-2183
> URL: https://issues.apache.org/jira/browse/SENTRY-2183
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Arjun Mishra
>Assignee: Arjun Mishra
>Priority: Major
> Attachments: SENTRY-2183.01.patch
>
>
> Full HMS Snapshots for large amounts of data (millions of databases, or
> tables or partitions etc) take a lot more time than the current default of
> 3.3mins. Also ideally since a full HMS snapshot should only occur once it is
> best to increase the default to 20mins. This highly reduce the chance of
> failure while sending a full snapshot which could be critical
> Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[
https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400788#comment-16400788
]
Hadoop QA commented on SENTRY-2183:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12914717/SENTRY-2183.01.patch
against master.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.tests.e2e.dbprovider.TestHmsNotificationProcessingWithOutSyncOnDrop
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/3700/console
This message is automatically generated.
> Increase default sentry-hdfs rpc timeout to 20 mins
> ---
>
> Key: SENTRY-2183
> URL: https://issues.apache.org/jira/browse/SENTRY-2183
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Arjun Mishra
>Assignee: Arjun Mishra
>Priority: Major
> Attachments: SENTRY-2183.01.patch
>
>
> Full HMS Snapshots for large amounts of data (millions of databases, or
> tables or partitions etc) take a lot more time than the current default of
> 3.3mins. Also ideally since a full HMS snapshot should only occur once it is
> best to increase the default to 20mins. This highly reduce the chance of
> failure while sending a full snapshot which could be critical
> Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
