[ https://issues.apache.org/jira/browse/BEAM-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Davor Bonaci reassigned BEAM-2429: ---------------------------------- Assignee: Flavio Fiszman (was: Davor Bonaci) > Conflicting filesystems with used of HadoopFileSystem > ----------------------------------------------------- > > Key: BEAM-2429 > URL: https://issues.apache.org/jira/browse/BEAM-2429 > Project: Beam > Issue Type: Bug > Components: sdk-java-extensions > Affects Versions: 2.0.0 > Reporter: François Wagner > Assignee: Flavio Fiszman > > I'm facing issue when trying to use HadoopFileSystem in my pipeline. It looks > like HadoopFileSystem is registring itself under the `file` schema > (https://github.com/apache/beam/pull/2777/files#diff-330bd0854dcab6037ef0e52c05d68eb2L79), > hence the following Exception is thrown when trying to register > HadoopFileSystem. > java.lang.IllegalStateException: Scheme: [file] has conflicting filesystems: > [org.apache.beam.sdk.io.LocalFileSystem, > org.apache.beam.sdk.io.hdfs.HadoopFileSystem] > at > org.apache.beam.sdk.io.FileSystems.verifySchemesAreUnique(FileSystems.java:498) > What is the correct way to handle `hdfs` url out of the box with TextIO & > AvroIO ? > {code:java} > String[] args = new String[]{ > "--hdfsConfiguration=[{\"dfs.client.use.datanode.hostname\": > \"true\"}]"}; > HadoopFileSystemOptions options = PipelineOptionsFactory > .fromArgs(args) > .withValidation() > .as(HadoopFileSystemOptions.class); > Pipeline pipeline = Pipeline.create(options); > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)