[
https://issues.apache.org/jira/browse/BEAM-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davor Bonaci reassigned BEAM-2429:
----------------------------------
Assignee: Flavio Fiszman (was: Davor Bonaci)
> Conflicting filesystems with used of HadoopFileSystem
> -----------------------------------------------------
>
> Key: BEAM-2429
> URL: https://issues.apache.org/jira/browse/BEAM-2429
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
> Affects Versions: 2.0.0
> Reporter: François Wagner
> Assignee: Flavio Fiszman
>
> I'm facing issue when trying to use HadoopFileSystem in my pipeline. It looks
> like HadoopFileSystem is registring itself under the `file` schema
> (https://github.com/apache/beam/pull/2777/files#diff-330bd0854dcab6037ef0e52c05d68eb2L79),
> hence the following Exception is thrown when trying to register
> HadoopFileSystem.
> java.lang.IllegalStateException: Scheme: [file] has conflicting filesystems:
> [org.apache.beam.sdk.io.LocalFileSystem,
> org.apache.beam.sdk.io.hdfs.HadoopFileSystem]
> at
> org.apache.beam.sdk.io.FileSystems.verifySchemesAreUnique(FileSystems.java:498)
> What is the correct way to handle `hdfs` url out of the box with TextIO &
> AvroIO ?
> {code:java}
> String[] args = new String[]{
> "--hdfsConfiguration=[{\"dfs.client.use.datanode.hostname\":
> \"true\"}]"};
> HadoopFileSystemOptions options = PipelineOptionsFactory
> .fromArgs(args)
> .withValidation()
> .as(HadoopFileSystemOptions.class);
> Pipeline pipeline = Pipeline.create(options);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
