[jira] [Commented] (HADOOP-16709) Consider having the ability to turn off TTL in S3Guard + Authoritative mode
[ https://issues.apache.org/jira/browse/HADOOP-16709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975000#comment-16975000 ] Gabor Bota commented on HADOOP-16709: - Talking offline with [~ste...@apache.org] we figured out that it would be the most straightforward to handle authoritative directories without TTL (you can set directories to be authoritative, not the whole store). So there will be no metadata expiry check for an authoritative path or when the authoritative mode is on. [~rajesh.balamohan] if you want to switch TTL off for the full bucket, you can set a very large number for the expiry, so the metadata won't expire. This workaround can be used until this is fixed. > Consider having the ability to turn off TTL in S3Guard + Authoritative mode > --- > > Key: HADOOP-16709 > URL: https://issues.apache.org/jira/browse/HADOOP-16709 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Rajesh Balamohan >Priority: Minor > > Authoritative mode has TTL which is set to 15 minutes by default. However, > there are cases when we know for sure that the data wouldn't be > changed/updated. > In certain cases, AppMaster ends up spending good amount of time in getSplits > due to TTL expiry. It would be great to have an option to disable TTL (or > specify as -1 when TTL shouldn't be checked). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16709) Consider having the ability to turn off TTL in S3Guard + Authoritative mode
[ https://issues.apache.org/jira/browse/HADOOP-16709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974210#comment-16974210 ] Gabor Bota commented on HADOOP-16709: - Sure. I can do this. The general idea behind expiry was that you can set a large enough number to virtually _disable_ TTL, but sure we can add a -1 value to skip checking for expiry. > Consider having the ability to turn off TTL in S3Guard + Authoritative mode > --- > > Key: HADOOP-16709 > URL: https://issues.apache.org/jira/browse/HADOOP-16709 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Rajesh Balamohan >Priority: Minor > > Authoritative mode has TTL which is set to 15 minutes by default. However, > there are cases when we know for sure that the data wouldn't be > changed/updated. > In certain cases, AppMaster ends up spending good amount of time in getSplits > due to TTL expiry. It would be great to have an option to disable TTL (or > specify as -1 when TTL shouldn't be checked). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16709) Consider having the ability to turn off TTL in S3Guard + Authoritative mode
[ https://issues.apache.org/jira/browse/HADOOP-16709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974112#comment-16974112 ] Steve Loughran commented on HADOOP-16709: - Moved under HADOOP-15620. [~rajesh.balamohan] -can you add things under there so they don't get lost, tag version affected, etc. Thanks. +[~gabor.bota] > Consider having the ability to turn off TTL in S3Guard + Authoritative mode > --- > > Key: HADOOP-16709 > URL: https://issues.apache.org/jira/browse/HADOOP-16709 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Rajesh Balamohan >Priority: Minor > > Authoritative mode has TTL which is set to 15 minutes by default. However, > there are cases when we know for sure that the data wouldn't be > changed/updated. > In certain cases, AppMaster ends up spending good amount of time in getSplits > due to TTL expiry. It would be great to have an option to disable TTL (or > specify as -1 when TTL shouldn't be checked). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
