[
https://issues.apache.org/jira/browse/METAMODEL-219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15051681#comment-15051681
]
ASF GitHub Bot commented on METAMODEL-219:
------------------------------------------
GitHub user kaspersorensen opened a pull request:
https://github.com/apache/metamodel/pull/78
Read hadoop configuration files (core-site.xml and hdfs-site.xml) in
HdfsResource
Suggested fix for METAMODEL-219.
I did a few minor/additional changes too:
* Moved the HDFS outputstream and inputstream classes to separate files
(HdfsResource was getting too big IMO). The classes are defined as default
(non-public) scoped, so not visible outside the package.
* Changed the FileHelper to support Java 7 AutoCloseable instead of
distinguishing between Closeable, Connection, Statement, ResultSet etc.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kaspersorensen/metamodel
METAMODEL-219-hadoop-configuration-files
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metamodel/pull/78.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #78
----
commit 14b63cdc262d40b0d6469de6cfc12103aa6239e3
Author: Kasper Sørensen <[email protected]>
Date: 2015-12-10T20:59:29Z
Moved HDFS resource stream classes to separate files.
commit 1e5dfe34d65cb9880a8ccd0e6adaa3a765640527
Author: Kasper Sørensen <[email protected]>
Date: 2015-12-10T21:06:09Z
Improved FileHelper.safeClose(...) method by using AutoCloseable
... which is now super-interface for Closeable, Connection, Statement,
ResultSet and more.
commit 7b5fb0c09c5dac7af3a2f2ff0e2aef8e0bbf2012
Author: Kasper Sørensen <[email protected]>
Date: 2015-12-10T21:24:39Z
METAMODEL-219: Added loading of core-site.xml and hdfs-site.xml
----
> HdfsResource's FileSystem should be createable based on custom core-site.xml
> and hdfs-site.xml files
> ----------------------------------------------------------------------------------------------------
>
> Key: METAMODEL-219
> URL: https://issues.apache.org/jira/browse/METAMODEL-219
> Project: Apache MetaModel
> Issue Type: Improvement
> Affects Versions: 4.4.1
> Reporter: Kasper Sørensen
>
> Today we always assume a very vanilla HDFS/Hadoop configuration which is
> based on defaults and overriding the one property {code}fs.defaultFS{code}.
> This behaviour is very limiting for the use of HdfsResource since many other
> custom properties may be needed.
> Instead we should offer a way where users of the class can specify a
> directory with the Hadoop configuration files. This could eventually also
> allow for a more flexibly style of representing the HDFS path, namely without
> the hostname and port (which would simply be dynamically loaded from the
> configuration file), like this:
> {code}
> hdfs:///path/to/file
> {code}
> How the configuration directory is provided should probably also be flexible.
> I would suggest that we offer three ways:
> (1) a new constructor option to provide the directory, and
> (2) a system property that will be checked, and
> (3) an environmental variable that will be checked (I suggest HADOOP_CONF_DIR
> and YARN_CONF_DIR to replicate common environmental variable names for this).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
