[
https://issues.apache.org/jira/browse/BEAM-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16044757#comment-16044757
]
Flavio Fiszman commented on BEAM-2429:
--------------------------------------
Hi François! In your hdfsConfiguration, you're specifying
"use.datanode.hostname" for your use case. On top of that, you should also
specify the field "fs.defaultFS" with hdfs:///, specifying the whole URI. You
can just add that field to the map.
This is a related question for reference, and notice how fs.defaultFS is the
new name for fs.default.name, if you see both of them appearing online. Let me
know if you have any questions, thanks!
> Conflicting filesystems with used of HadoopFileSystem
> -----------------------------------------------------
>
> Key: BEAM-2429
> URL: https://issues.apache.org/jira/browse/BEAM-2429
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
> Affects Versions: 2.0.0
> Reporter: François Wagner
> Assignee: Flavio Fiszman
>
> I'm facing issue when trying to use HadoopFileSystem in my pipeline. It looks
> like HadoopFileSystem is registring itself under the `file` schema
> (https://github.com/apache/beam/pull/2777/files#diff-330bd0854dcab6037ef0e52c05d68eb2L79),
> hence the following Exception is thrown when trying to register
> HadoopFileSystem.
> java.lang.IllegalStateException: Scheme: [file] has conflicting filesystems:
> [org.apache.beam.sdk.io.LocalFileSystem,
> org.apache.beam.sdk.io.hdfs.HadoopFileSystem]
> at
> org.apache.beam.sdk.io.FileSystems.verifySchemesAreUnique(FileSystems.java:498)
> What is the correct way to handle `hdfs` url out of the box with TextIO &
> AvroIO ?
> {code:java}
> String[] args = new String[]{
> "--hdfsConfiguration=[{\"dfs.client.use.datanode.hostname\":
> \"true\"}]"};
> HadoopFileSystemOptions options = PipelineOptionsFactory
> .fromArgs(args)
> .withValidation()
> .as(HadoopFileSystemOptions.class);
> Pipeline pipeline = Pipeline.create(options);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
