[ https://issues.apache.org/jira/browse/HADOOP-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16919525#comment-16919525 ]
Arun Ravi M V edited comment on HADOOP-16540 at 8/30/19 1:11 PM: ----------------------------------------------------------------- Yes, you are right, the main reason for this ticket is user credentials. I have a situation where a large number of datasets (a few thousand) is located at a single s3 bucket. I am trying to introduce Role-based access control here, AWS policies have size limitations and cannot be used as the only solution. In this case, I would like to define caching per dataset (bucket + root s3 prefix) instead of doing it at the s3 bucket level. was (Author: arunravimv): Yes, you are right, the main reason for this ticket is user credentials. I have a situation where a large number of datasets (a few thousand) is located at a single s3 bucket. I am trying to introduce Role-based access control here, AWS policies have size limitations and cannot be used as the only solution. In this case, I would like to define caching per dataset (bucket + root s3 prefix) instead of doing it at the s3 bucket level. > Pluggable Filesystem Caching Support in FileSystem Class > -------------------------------------------------------- > > Key: HADOOP-16540 > URL: https://issues.apache.org/jira/browse/HADOOP-16540 > Project: Hadoop Common > Issue Type: New Feature > Components: fs > Affects Versions: 3.3.0 > Reporter: Arun Ravi M V > Priority: Major > > Provide an option to use Custom Cache Class in FileSystem Class. Currently, > the caching is enabled by default and uses the URI schema and authority value > to determine whether to create a new FS instance for the given URI or to > fetch an already existing one from the cache. > In case of AWS S3 FS Impl, for an S3 path, the authority name will be bucket > name, ie Filesystem object will be cached at the bucket level, but providing > a custom caching logic can empower the user to cache it at some prefix level > and provide more flexibility. -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org