Hi All,
I have a pipeline; it writes to hdfs, using "org.apache.beam.sdk.io.hdfs"
package.
I was hoping that it will work with maprfs as well. However I have been
debugging for some days, with no success.
I do not provide hdfsConfiguration from command line, and instead use the
configuration file /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core-site.xml
(please find it attached with the mail)
However I get the stack trace ("No FileSystem for scheme: maprfs"). Please find
the detailed trace attached. I made sure that I have
/opt/mapr/lib/maprfs-5.2.2-mapr.jar on classpath. I use spark submit to run the
pipeline.
If you have already tried writing to maprfs from beam, ccould you please help?
Regards,
Akanksha
core-site
Description: core-site
Caused by: java.lang.IllegalArgumentException: Failed to construct Hadoop
filesystem with configuration Configuration:
/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core-site.xml,
/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/hdfs-site.xml
at
org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrar.fromOptions(HadoopFileSystemRegistrar.java:59)
at
org.apache.beam.sdk.io.FileSystems.verifySchemesAreUnique(FileSystems.java:505)
at
org.apache.beam.sdk.io.FileSystems.setDefaultPipelineOptions(FileSystems.java:495)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:47)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
at
com.ericsson.eea.cleanadapter.CleanAdapterPipeline.main(CleanAdapterPipeline.java:26)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)
Caused by: java.io.IOException: No FileSystem for scheme: maprfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2684)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2691)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:98)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2773)
at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2761)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:458)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:466)
at org.apache.beam.sdk.io.hdfs.HadoopFileSystem.(HadoopFileSystem.java:73)
at
org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrar.fromOptions(HadoopFileSystemRegistrar.java:57)
