[
https://issues.apache.org/jira/browse/PARQUET-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255940#comment-14255940
]
Ryan Blue commented on PARQUET-154:
-----------------------------------
bq. Is there some reason we avoided using the fileSystem object in constructor ?
I generally avoid passing a FileSystem around because you can't always assume
that the Path and FileSystem that are passed in match. If you pass a handle to
the local FS and an explicit hdfs:// path, then any operation throws an
exception that the path and FS don't match. I think it is best practice to use
the Path object as a single source of truth for the file system so that you can
ignore the possibility. The caller has to use Path#makeQualified with the
expected FS, and an error can be handled at that time by the caller. File
systems are cached by FileSystem.get, so this is fairly low overhead.
> parquet Constructors not taking file System object
> --------------------------------------------------
>
> Key: PARQUET-154
> URL: https://issues.apache.org/jira/browse/PARQUET-154
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Affects Versions: parquet-mr_1.6.0
> Reporter: Manish Agarwal
>
> I am trying to create a file in parquet file format and in RC file format .
>
> No Parquet constructor accepts fileSystem object as an argument. This means
> that i will have to append the uri from the file system in front of the
> filepath object everytime i need to create a new file .
> In RC format the file system object is allowed to be passed in the
> constructor .
> The advantage of passing the file System object into the constructor is that
> i can specify my yarn instance file system pointer to be used while
> creating the file and its quite straight forward . For example RC file
> constructors uses the file System which we have passed .
> In parquet i see everywhere file System object being derived out of the
> Parameters or created a fresh .
> Is there some reason we avoided using the fileSystem object in constructor ?
> If we allow a file System object constructor as well , I would not have to
> worry about modifying my file name to contain the uri part .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
