[
https://issues.apache.org/jira/browse/HADOOP-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921385#comment-16921385
]
Steve Loughran commented on HADOOP-16540:
-----------------------------------------
I think you are in trouble here -and that pluggable FS caching is not the
solution. There are some big assumptions in application code that an FS
instance can be used across any path in the FS, and that permissions checks are
done in the server. Once you have an instance you can pass in any path (or as
HADOOP-16482 implies, any S3 URI).
HADOOP-16445 is working on separate signers for S3 and DDB/STS; Sidd is looking
about being more adaptive here. And in the proposal linked off HADOOP-16456
I've discussed having a per request context which would go end-to-end across an
operation, so you could create a signer/set of creds per request (more
specifically, you'd need to cache them as
org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestSessionCredentials shows the
limits there). and those credentials would go round with the read/write/rename,
etc. We'll also need some credential factory API to take some (operation, UGI,
source, dest) params and return the creds for that operation. Unless you really
want to give the clients full credentials, you will be needing some credential
factory service over RPC there.
I think you should get involved with those bits of work, so you can make sure
it helps meet your needs.
I think we should close this JIRA as a WONTFIX; it doesn't do what you need.
> Pluggable Filesystem Caching Support in FileSystem Class
> --------------------------------------------------------
>
> Key: HADOOP-16540
> URL: https://issues.apache.org/jira/browse/HADOOP-16540
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Affects Versions: 3.3.0
> Reporter: Arun Ravi M V
> Priority: Major
>
> Provide an option to use Custom Cache Class in FileSystem Class. Currently,
> the caching is enabled by default and uses the URI schema and authority value
> to determine whether to create a new FS instance for the given URI or to
> fetch an already existing one from the cache.
> In case of AWS S3 FS Impl, for an S3 path, the authority name will be bucket
> name, ie Filesystem object will be cached at the bucket level, but providing
> a custom caching logic can empower the user to cache it at some prefix level
> and provide more flexibility.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
