I added this and still get the same exception. The same property is also set in spark-defaults.conf.
After that didn't work, I also tried adding --conf spark.authenticate=true and --conf spark.authenticate.enableSaslEncryption=true, to match the other related settings in spark-defaults.conf. Still get the same classnotfoundexception. On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim <hss...@nflabs.com> wrote: > Please add the *export SPARK_SUBMIT_OPTIONS="--conf > spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and restart > zeppelin and retry your code. > > > 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy < > jonathan.esterh...@gmail.com>: > >> yes it does. i only see this problem in zeppelin. >> >> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <hss...@nflabs.com> >> wrote: >> >>> Hi Jonathan. >>> It's not easy to build the test environments but I am working on this. >>> I have question for you. >>> Does your code working well on spark-shell in the spark.authenticate >>> mode? >>> >>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy < >>> jonathan.esterh...@gmail.com>: >>> >>>> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm >>>> quite stumped. >>>> >>>> thx. >>>> >>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <hss...@nflabs.com> >>>> wrote: >>>> >>>>> Thank you. >>>>> Let me try. >>>>> >>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy < >>>>> jonathan.esterh...@gmail.com>: >>>>> >>>>>> Hyung, >>>>>> >>>>>> Yes, here they are. >>>>>> >>>>>> zeppelin-env.sh: >>>>>> >>>>>> export ZEPPELIN_PORT=8890 >>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf >>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin >>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin >>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid >>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook >>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps >>>>>> export MASTER=yarn-client >>>>>> export SPARK_HOME=/usr/lib/spark >>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf >>>>>> export >>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/* >>>>>> :/usr/share/aws/emr/emrfs/auxlib/*" >>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0 >>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket >>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin >>>>>> export >>>>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo >>>>>> >>>>>> spark-defaults.conf: >>>>>> >>>>>> spark.master yarn >>>>>> spark.driver.extraClassPath >>>>>> >>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf >>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* >>>>>> spark.driver.extraLibraryPath >>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native >>>>>> spark.executor.extraClassPath >>>>>> >>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf >>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* >>>>>> spark.executor.extraLibraryPath >>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native >>>>>> spark.eventLog.enabled true >>>>>> spark.eventLog.dir hdfs:///var/log/spark/apps >>>>>> spark.history.fs.logDirectory hdfs:///var/log/spark/apps >>>>>> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080 >>>>>> spark.history.ui.port 18080 >>>>>> spark.shuffle.service.enabled true >>>>>> spark.driver.extraJavaOptions >>>>>> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties >>>>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 >>>>>> -XX:MaxHeapFreeRatio=70 >>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M >>>>>> -XX:OnOutOfMemoryError='kill -9 %p' >>>>>> spark.dynamicAllocation.enabled true >>>>>> spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails >>>>>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC >>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM >>>>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p' >>>>>> spark.executor.memory 8640m >>>>>> spark.executor.cores 7 >>>>>> spark.authenticate.enableSaslEncryption true >>>>>> spark.driver.memory 1g >>>>>> spark.network.sasl.serverAlwaysEncrypt true >>>>>> spark.driver.cores 1 >>>>>> spark.ssl.protocol TLSv1.2 >>>>>> spark.ssl.keyStorePassword password >>>>>> spark.yarn.maxAppAttempts 1 >>>>>> spark.ssl.keyStore /etc/emr/security/keystore.jks >>>>>> spark.authenticate true >>>>>> spark.ssl.keyPassword password >>>>>> spark.ssl.enabled true >>>>>> spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_CBC_SHA >>>>>> spark.ssl.trustStore /etc/emr/security/truststore.jks >>>>>> spark.authenticate.secret secret >>>>>> spark.ssl.trustStorePassword password >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim <hss...@nflabs.com> >>>>>> wrote: >>>>>> >>>>>>> Hi. >>>>>>> Could you share your conf/zeppelin-env.sh and spark-defaults.conf ? >>>>>>> >>>>>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy < >>>>>>> jonathan.esterh...@gmail.com>: >>>>>>> >>>>>>>> I am having trouble using zeppelin in a spark cluster that has >>>>>>>> spark node authentication turned on (e.g. with spark.authenticate=true, >>>>>>>> spark.authenticate.secret=...) >>>>>>>> >>>>>>>> Notebook code that calls built-in spark functions (or other things >>>>>>>> on executor classpath) work fine, but functions defined in the notebook >>>>>>>> (anonymous or named) throw ClassNotFoundExceptions when called from an >>>>>>>> executor. >>>>>>>> >>>>>>>> For example, this code works: >>>>>>>> >>>>>>>> val rdd = sc.textFile("hdfs://my-text-file") >>>>>>>> rdd.take(1).foreach(println) >>>>>>>> >>>>>>>> rdd.saveAsTextFile("hdfs:///my-other-text-file") >>>>>>>> >>>>>>>> but code like this... >>>>>>>> >>>>>>>> rdd.filter(_.contains("my data")) >>>>>>>> >>>>>>>> fails with >>>>>>>> >>>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 >>>>>>>> at >>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84) >>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>>> at java.lang.Class.forName(Class.java:348) >>>>>>>> ... >>>>>>>> >>>>>>>> >>>>>>>> I get the same kind of error if the filter function is defined as a >>>>>>>> named function in the notebook, or as a member of singleton object >>>>>>>> defined >>>>>>>> in the notebook. >>>>>>>> >>>>>>>> When I look at the executor's log output, I see this error: >>>>>>>> >>>>>>>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to check >>>>>>>> existence of class >>>>>>>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on >>>>>>>> REPL >>>>>>>> class server at https://172.30.54.30:34980 >>>>>>>> java.lang.NullPointerException >>>>>>>> at >>>>>>>> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113) >>>>>>>> at >>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146) >>>>>>>> at >>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76) >>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>>> at java.lang.Class.forName(Class.java:348) >>>>>>>> >>>>>>>> ... >>>>>>>> >>>>>>>> >>>>>>>> If I disable spark authentication, everything works as expected. I >>>>>>>> am running zeppelin 0.5.6 on spark 1.6.1 with yarn. >>>>>>>> >>>>>>>> Has anyone been able to get zeppelin working with spark >>>>>>>> authentication? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >