[
https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HDFS-3263:
-------------------------------------
Attachment: HDFS-3263.patch
updated patch based on Eli's feedback (thx!!)
Follow ups on the non-integrated comments.
FileSystemAccessService.java:
* Can you re-use an existing mechanism, eg use HADOOP_CONF_DIR by default and
only require the configuration if that's not present?
TUCU: No we cannot, the current mechanism relies on having the HADOOP_CONF_DIR
in the classpath. HttpFS does not have HADOOP_CONF_DIR in the classpath
(furthermore, it can run in a setup where there is no other hadoop component
installed)
* Are core-site.xml and hdfs-site.xml the only files that should be pulled in,
eg how about hadoop-policy.xml?
TUCU: was not aware of the hadoop-policy.xml but how would the be used in
HttpFS context?
* Do we need the FILE_SYSTEM_SERVICE_CREATED check? Ie if the proxy is setup
correctly can users ever end up with a conf that doesn't have the hadoop conf
set. If so should be a one-time initialization level check then a check
performed on each call to execute.
This check is to prevent httpfs code (got bitten by this) to create and use a
Configuration object without the hadoop confs in it (per first bullet item)
* If a user gets an HO4 error it means "hdfs.configuration.dir" was not
defined, so worth including that in the message.
If hdfs.configuration.dir is not defined HttpFS conf dir is used. H04 is for
other thing (when a Configuration object not created by FSAS is used, see
previous bullet item).
----
On a related node is there a test that covers httpfs working w/ a non-HDFS
Hadoop filesystem? Would be good to have a test that uses httpfs with local
file system to catch the introduction of HDFS-isms. Can punt this to another
jira.
yeah, good idea, will do.
> HttpFS should read HDFS config from Hadoop site.xml files
> ---------------------------------------------------------
>
> Key: HDFS-3263
> URL: https://issues.apache.org/jira/browse/HDFS-3263
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 2.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.0
>
> Attachments: HDFS-3263.patch, HDFS-3263.patch, HDFS-3263.patch,
> HDFS-3263.patch, HDFS-3263.patch
>
>
> Currently HttpFS reads HDFS client configuration from the httfs-site.xml from
> any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY'
> This is a bit inconvenient.
> Instead we should support a single property 'httpfs.hadoop.configuration.dir'
> that can be pointed to HADOOP conf/ dir and the core-site.xml and
> hdfs-site.xml files would be read from there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
