[
https://issues.apache.org/jira/browse/HADOOP-19478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-19478:
------------------------------------
Description:
S3AFS i's too full of switches; it is why initialize() is so big, and there are
lots and lots of fields to record the values.
Many of these need to go down into S3AStoreImpl, but replicating the same
design just pushes the mess down.
Proposed: a child service StoreConfigurationService, which reads in the config
during serviceInit(), and set the state in there, from where it can be probed.
I am initially doing this purely for new configuration flags for the
conditional write feature
* moving other flags in there would be separate work
* new boolean config options should go in here
* we also need to think about integers, units of scale and durations.
Ideally, we should the annotations and reflection code from ABFS and
org.apache.hadoop.fs.azurebfs.contracts.annotations.ConfigurationValidationAnnotations,
* copy it into common
* adding duration and size attributes
* move reflection code from AbfsConfiguration#AbfsConfiguration into there too,
* use in s3a
* migrate abfs to the moved code
was:
I am doing this as part of HADOOP-19256, but wanted to call it out for specific
mention.
S3AFS i's too full of switches; it is why initialize() is so big, and there are
lots and lots of fields to record the values.
Many of these need to go down into S3AStoreImpl, but replicating the same
design just pushes the mess down.
Proposed: a child service StoreConfigurationService, which reads in the config
during serviceInit(), and set the state in there, from where it can be probed.
I am initially doing this purely for new configuration flags for the
conditional write feature
* moving other flags in there would be separate work
* new boolean config options should go in here
* we also need to think about integers, units of scale and durations.
Ideally, we should use something reflection based as abfs does.
> S3A: pull out new configuration load/probes under S3AStore
> ----------------------------------------------------------
>
> Key: HADOOP-19478
> URL: https://issues.apache.org/jira/browse/HADOOP-19478
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> S3AFS i's too full of switches; it is why initialize() is so big, and there
> are lots and lots of fields to record the values.
> Many of these need to go down into S3AStoreImpl, but replicating the same
> design just pushes the mess down.
> Proposed: a child service StoreConfigurationService, which reads in the
> config during serviceInit(), and set the state in there, from where it can be
> probed.
> I am initially doing this purely for new configuration flags for the
> conditional write feature
> * moving other flags in there would be separate work
> * new boolean config options should go in here
> * we also need to think about integers, units of scale and durations.
> Ideally, we should the annotations and reflection code from ABFS and
> org.apache.hadoop.fs.azurebfs.contracts.annotations.ConfigurationValidationAnnotations,
> * copy it into common
> * adding duration and size attributes
> * move reflection code from AbfsConfiguration#AbfsConfiguration into there
> too,
> * use in s3a
> * migrate abfs to the moved code
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
