what am I doing wrong here ? single instance ec2 with spark and zeppelin. it
works using spark/bin/pyspark, but in a zeppelin notebook it fails to find
com.databricks.spark.avro. thanks
from the zeppelin
notebook:%pysparksc._jsc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId","xxx")sc._jsc.hadoopConfiguration().set("fs.s3n.awsSecretAccessKey","xxx")df
=
sqlContext.read.format("com.databricks.spark.avro").load("s3n://mybucket/myfile.avro")
Py4JJavaError: An error occurred while calling o44.load.:
java.lang.ClassNotFoundException: Failed to find data source:
com.databricks.spark.avro. Please use Spark package
http://spark-packages.org/package/databricks/spark-avro
in conf/zeppelin-env.shexport SPARK_SUBMIT_OPTIONS="--packages
com.databricks:spark-avro_2.10:2.0.1,com.amazonaws:aws-java-sdk-pom:1.10.34,org.apache.hadoop:hadoop-aws:2.6.0
--driver-memory 16g"
in conf/spark-defaults.confspark.jars.packages
com.databricks:spark-avro_2.10:2.0.1
ps -ef | grep zeppelin (shows "--packages
com.databricks:spark-avro_2.10:2.0.1...")
ec2-user 32718 32706 99 18:58 pts/2 00:00:55 /usr/lib/jvm/jre/bin/java -cp
/usr/local/zeppelin/interpreter/spark/zeppelin-spark-0.5.6-incubating.jar:/usr/local/spark/conf/:/usr/local/spark/lib/spark-assembly-1.6.1-hadoop2.6.0.jar:/usr/local/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/local/spark/lib/datanucleus-api-jdo-3.2.6.jar:/usr/local/spark/lib/datanucleus-core-3.2.10.jar
-Xms16g -Xmx16g -Dfile.encoding=UTF-8 -Dfile.encoding=UTF-8
-Dzeppelin.log.file=/usr/local/zeppelin/logs/zeppelin-interpreter-spark-ec2-user-ip-10-248-5-9.log
-XX:MaxPermSize=256m org.apache.spark.deploy.SparkSubmit --conf
spark.driver.extraClassPath=::/usr/local/zeppelin/interpreter/spark/zeppelin-spark-0.5.6-incubating.jar
--conf spark.driver.memory=16g --conf spark.driver.extraJavaOptions=
-Dfile.encoding=UTF-8 -Dfile.encoding=UTF-8
-Dzeppelin.log.file=/usr/local/zeppelin/logs/zeppelin-interpreter-spark-ec2-user-ip-10-248-5-9.log
--class org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer
--packages
com.databricks:spark-avro_2.10:2.0.1,com.amazonaws:aws-java-sdk-pom:1.10.34,org.apache.hadoop:hadoop-aws:2.6.0
/usr/local/zeppelin/interpreter/spark/zeppelin-spark-0.5.6-incubating.jar 45438
