steveloughran opened a new pull request #1839: HADOOP-16848. Refactoring: initial layering URL: https://github.com/apache/hadoop/pull/1839 First PoC of My planned layout model of the S3A FS. * There's a raw layer and a guarded layer * which are instantiated in sequence in a separate executor from S3AFileSystem.initalize And whose accessors block until completed or rethrow failures. The layers are being handed in all their dependencies from FS.initialize() and we currently block until started. What I plan to do, in a future iteration, is: * each layer extracts their own settings from the config and stores locally (list version, upload size etc) * have each layer instantiate their internal classes (AWS S3 client, transfer manager) internally * Also async create: metastore, DT binding * And all startup actions (check bucket, init multipart, ...) Then * move ops to the layers, raw* -> rawStore; inner -> S3AStore * move WriteOperationHelper, SelectBinding, etc, to all work against S3AStore rather than FS. S3AStore will become where most of the code moves to; S3AFilesystem more of the init and binding to hadoop FS API. RawS3A will be the accessor through which all AWS client access goes. Not going to change: all accessors on S3AFileSystem...not just tests use it but some external code (cloudstore) needs it to get at low level S3A, etc. Change-Id: I998c0d61cce2ee7fd0be804bf21da6b68fd69a6f ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
