Hi,
I'm running Spark on mesos and trying to read file from Maprcluster but not
have much success with that. I tried 2 versions of Apache Spark (with and
without Hadoop).
I can get to the spark-shell in the with-hadoop version, but still can't
access maprfs. Without-Hadoop version bails out with
org.apache.hadoop.fs.FSDataInputStream ClassNotFoundException.
Can anyone please help me figure out what am I doing wrong ?
#Spark without Hadoop:
~/spark/spark-1.4.0-bin-without-hadoop $ ./bin/spark-shell --master
mesos://mesos-master.local:5050 --driver-library-path=/opt/mapr/lib
--driver-class-path
/opt/mapr/hadoop/hadoop-0.20.2/conf:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar:/opt/mapr/lib/maprfs-4.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-4.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/zookeeper-3.4.5-mapr-1406.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/guava-13.0.1.jar:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/hdfs/hadoop-hdfs-2.5.1-mapr-1503.jar
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/fs/FSDataInputStream
at
org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:111)
at
org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:111)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:111)
at
org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:97)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:106)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.fs.FSDataInputStream
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 7 more
#Spark with Hadoop 2.6
~/spark/spark-1.4.0-bin-hadoop2.6$ ./bin/spark-shell --master
mesos://mesos-master.local:5050 --driver-library-path=/opt/mapr/lib
--driver-class-path
/opt/mapr/hadoop/hadoop-0.20.2/conf:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar:/opt/mapr/lib/maprfs-4.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-4.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/zookeeper-3.4.5-mapr-1406.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/guava-13.0.1.jar:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/hdfs/hadoop-hdfs-2.5.1-mapr-1503.jar
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please
use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties
files.
15/06/17 23:14:02 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicabl
.
.
.
scala> val textFile = sc.textFile("maprfs:///user/packages/CHANGES.txt")
scala> textFile.count()
java.io.IOException: No FileSystem for scheme: maprfs
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
Thank you,
Bikrant
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Reading-maprfs-from-Spark-tp23382.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
