[
https://issues.apache.org/jira/browse/HADOOP-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979369#comment-15979369
]
Siddharth Seth commented on HADOOP-14138:
-----------------------------------------
bq. those JIRAs are so old they are implicitly dead.
Don't think they're any less relevant today, than they were when they were
filed.
Realistically though, the jiras will likely not be fixed - 1) Incompatible, and
incompatible in a manner that is not easy to find since this is not a
compilation breakage. 2) Someone needs to actually put in some work to make
this happen.
bq. To me, having to change defaults is pretty common (we frequently have to
tweak core-default settings for a shipping product), and being able to do that
in a default config is very low-friction compared to code changes.
Isn't that what the site files are for?
A lot of people consider the core-default files as documentation. Available
Config, Default Value, Description.
In Tez we went the approach of explicitly not having a default file, and
generated an output file from the code defaults.
Hive uses a nice approach where HiveConf.get(ParamName) implicitly picks up
default values. No *-default.xml file here either.
That said, if we're moving to discussing core-default.xml vs Code defaults -
probably needs a wider audience.
The change helps with performance, so that's really good. Think this affects
simple invocations like hadoop fs -ls, and it's really good to see this run
faster. Hoping that a longer term change to fix service loaders goes in.
Unfortunately will not be able to contribute, the patch in any case.
> Remove S3A ref from META-INF service discovery, rely on existing core-default
> entry
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-14138
> URL: https://issues.apache.org/jira/browse/HADOOP-14138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Critical
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha3
>
> Attachments: HADOOP-14138.001.patch, HADOOP-14138-branch-2-001.patch
>
>
> As discussed in HADOOP-14132, the shaded AWS library is killing performance
> starting all hadoop operations, due to classloading on FS service discovery.
> This is despite the fact that there is an entry for fs.s3a.impl in
> core-default.xml, *we don't need service discovery here*
> Proposed:
> # cut the entry from
> {{/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem}}
> # when HADOOP-14132 is in, move to that, including declaring an XML file
> exclusively for s3a entries
> I want this one in first as its a major performance regression, and one we
> coula actually backport to 2.7.x, just to improve load time slightly there too
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
