[ https://issues.apache.org/jira/browse/BEAM-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587910#comment-16587910 ]
Tim Robertson commented on BEAM-2277: ------------------------------------- Thank you for investigating [~JozoVilcek]. bq. Is anyone have this problem if it is using HDFS paths in form with vs without authority? Yes, but I think you've deduced that already bq. The question is, what would be an elegant fix to that. I'll try and find time to explore, but I am afraid it won't be until later this week. It would be great if we can solve this for 2.7.0. > IllegalArgumentException when using Hadoop file system for WordCount example. > ----------------------------------------------------------------------------- > > Key: BEAM-2277 > URL: https://issues.apache.org/jira/browse/BEAM-2277 > Project: Beam > Issue Type: Bug > Components: z-do-not-use-sdk-java-extensions > Affects Versions: 2.6.0 > Reporter: Aviem Zur > Assignee: Aviem Zur > Priority: Blocker > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > IllegalArgumentException when using Hadoop file system for WordCount example. > Occurred when running WordCount example using Spark runner on a YARN cluster. > Command-line arguments: > {code:none} > --runner=SparkRunner --inputFile=hdfs:///user/myuser/kinglear.txt > --output=hdfs:///user/myuser/wc/wc > {code} > Stack trace: > {code:none} > java.lang.IllegalArgumentException: Expect srcResourceIds and destResourceIds > have the same scheme, but received file, hdfs. > at > org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) > at > org.apache.beam.sdk.io.FileSystems.validateSrcDestLists(FileSystems.java:394) > at org.apache.beam.sdk.io.FileSystems.copy(FileSystems.java:236) > at > org.apache.beam.sdk.io.FileBasedSink$WriteOperation.copyToOutputFiles(FileBasedSink.java:626) > at > org.apache.beam.sdk.io.FileBasedSink$WriteOperation.finalize(FileBasedSink.java:516) > at > org.apache.beam.sdk.io.WriteFiles$2.processElement(WriteFiles.java:592) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)