Steve Loughran created HADOOP-19354:
---------------------------------------
Summary: S3AInputStream to be created by factory under S3AStore
Key: HADOOP-19354
URL: https://issues.apache.org/jira/browse/HADOOP-19354
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.2
Reporter: Steve Loughran
Assignee: Steve Loughran
Migrate S3AInputStream creation into a factory pattern, push down into S3AStore.
Proposed factories
* default: whatever this release has as default
* classic: current S3AInputStream
* prefetch: prefetching
* analytics: new analytics stream
* other: reads a classname from another prop, instantiates.
Also proposed
* stream to implement some stream capability to declare what they are (classic,
prefetch, analytics, other).
h2. Implementation
All callbacks used by the stream also to call directly onto S3AStore.
S3AFileSystem must not be invoked at all (if it is needed: PR is still not
ready).
Some interface from Instrumentation will be passed to factory; this shall
include a way to create new per-stream
The factory shall implement org.apache.hadoop.service.Service; S3AStore shall
do same and become a subclass of CompositeService. It shall attach the factory
as a child, so they can follow the same lifecycle. We shall do the same for
anything else that gets pushed down.
Everything related to stream creation must go from s3afs; and creation of the
factory itself. This must be done in S3AStore.initialize().
As usual, this will complicate mocking. But the streams themselves should not
require changes, at least significant ones.
Testing.
* The huge file tests should be tuned so each of the different ones uses a
different stream, always.
* use a -Dstream="factory name" to choose factory, rather than the -Dprefetch
* if not set, whatever is in auth-keys gets picked up.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
