[
https://issues.apache.org/jira/browse/HADOOP-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-13448:
----------------------------------
Attachment: HADOOP-13448-HADOOP-13345.002.patch
I'm posting a v2 patch which evolves the initial patch some. @cnauorth
is on vacation, but I wanted to keep the process moving with some suggested
modifications I made (for my work on subtask HADOOP-13452):
- Define concrete types PathMetadata and DirListingMetadata.
- Both are "containers" of FileStatuses. I use a type bound to allow them
to hold any subclass of FileStatus, e.g. S3AFileStatus.
- DirListingMetadata is a collection of FileStatus plus an
isAuthoritative flag as recently discussed in HADOOP-13345.
- I thought about having a common base class but it felt contrived, as things
like isAuthoratitive(), lookup/get/delete() only make sense for directory
listings, not single path metadata. That is, I didn't see a common "is a"
relationship.
- I removed MetadataStore's dependency on S3AFileSystem for the Logger. I'm
hoping this code can eventually be factored out of s3a and become useful for
other FS clients.
- Added another put() method to MetadataStore that takes a DirListingMetadata.
I explain the need for the isAuthorative bit on put() in my recent comment
on HADOOP-13345
Question / Next Steps:
- Should MetadataStore be an interface? I left it an abstract class but
it looks more like an Interface at the moment. Perhaps we'll want to add
back a common Logger object, that doesn't depend on S3AFileSystem?
- How do folks feel about this API consuming/producing FileStatus'? I
think it simplifies the client code quite a bit, which is one of my
top concerns. I'm fine with changing it if it becomes awkward.
- I'm wondering if DirListingMetadata will become a collection of PathMetadata
instead of FileStatus'. For now I did the latter, thinking of what FS clients
will need: FileStatus'. If we add things like content hashes or etags in
future I could see this becoming a collection of PathMetadata instead.
> S3Guard: Define MetadataStore interface.
> ----------------------------------------
>
> Key: HADOOP-13448
> URL: https://issues.apache.org/jira/browse/HADOOP-13448
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HADOOP-13448-HADOOP-13345.001.patch,
> HADOOP-13448-HADOOP-13345.002.patch
>
>
> Define the common interface for metadata store operations. This is the
> interface that any metadata back-end must implement in order to integrate
> with S3Guard.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
