Hi. Jonathan. If it is possible, could you use the zeppelin master on the github? Zeppelin is working well with spark authentication when i tested.
2016-07-02 3:05 GMT+09:00 Jonathan Esterhazy <jonathan.esterh...@gmail.com>: > Hyung, thx for your help. I opened these: > > https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala problem) > https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking > python problem) > > LMK if I can provide more info or help in some way. > > On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim <hss...@nflabs.com> wrote: > >> Hi Jonathan. >> Unfortunately I got same error in my test bed. >> Do you mind create an jira issue for this? >> >> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy < >> jonathan.esterh...@gmail.com>: >> >>> I added this and still get the same exception. The same property is also >>> set in spark-defaults.conf. >>> >>> After that didn't work, I also tried adding --conf >>> spark.authenticate=true and --conf >>> spark.authenticate.enableSaslEncryption=true, to match the other related >>> settings in spark-defaults.conf. Still get the same classnotfoundexception. >>> >>> >>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim <hss...@nflabs.com> >>> wrote: >>> >>>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf >>>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and restart >>>> zeppelin and retry your code. >>>> >>>> >>>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy < >>>> jonathan.esterh...@gmail.com>: >>>> >>>>> yes it does. i only see this problem in zeppelin. >>>>> >>>>> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <hss...@nflabs.com> >>>>> wrote: >>>>> >>>>>> Hi Jonathan. >>>>>> It's not easy to build the test environments but I am working on this. >>>>>> I have question for you. >>>>>> Does your code working well on spark-shell in the spark.authenticate >>>>>> mode? >>>>>> >>>>>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy < >>>>>> jonathan.esterh...@gmail.com>: >>>>>> >>>>>>> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm >>>>>>> quite stumped. >>>>>>> >>>>>>> thx. >>>>>>> >>>>>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <hss...@nflabs.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thank you. >>>>>>>> Let me try. >>>>>>>> >>>>>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy < >>>>>>>> jonathan.esterh...@gmail.com>: >>>>>>>> >>>>>>>>> Hyung, >>>>>>>>> >>>>>>>>> Yes, here they are. >>>>>>>>> >>>>>>>>> zeppelin-env.sh: >>>>>>>>> >>>>>>>>> export ZEPPELIN_PORT=8890 >>>>>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf >>>>>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin >>>>>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin >>>>>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid >>>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook >>>>>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps >>>>>>>>> export MASTER=yarn-client >>>>>>>>> export SPARK_HOME=/usr/lib/spark >>>>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf >>>>>>>>> export >>>>>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/* >>>>>>>>> :/usr/share/aws/emr/emrfs/auxlib/*" >>>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0 >>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket >>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin >>>>>>>>> export >>>>>>>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo >>>>>>>>> >>>>>>>>> spark-defaults.conf: >>>>>>>>> >>>>>>>>> spark.master yarn >>>>>>>>> spark.driver.extraClassPath >>>>>>>>> >>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf >>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* >>>>>>>>> spark.driver.extraLibraryPath >>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native >>>>>>>>> spark.executor.extraClassPath >>>>>>>>> >>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf >>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* >>>>>>>>> spark.executor.extraLibraryPath >>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native >>>>>>>>> spark.eventLog.enabled true >>>>>>>>> spark.eventLog.dir hdfs:///var/log/spark/apps >>>>>>>>> spark.history.fs.logDirectory hdfs:///var/log/spark/apps >>>>>>>>> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080 >>>>>>>>> spark.history.ui.port 18080 >>>>>>>>> spark.shuffle.service.enabled true >>>>>>>>> spark.driver.extraJavaOptions >>>>>>>>> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties >>>>>>>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 >>>>>>>>> -XX:MaxHeapFreeRatio=70 >>>>>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M >>>>>>>>> -XX:OnOutOfMemoryError='kill -9 %p' >>>>>>>>> spark.dynamicAllocation.enabled true >>>>>>>>> spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails >>>>>>>>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC >>>>>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM >>>>>>>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p' >>>>>>>>> spark.executor.memory 8640m >>>>>>>>> spark.executor.cores 7 >>>>>>>>> spark.authenticate.enableSaslEncryption true >>>>>>>>> spark.driver.memory 1g >>>>>>>>> spark.network.sasl.serverAlwaysEncrypt true >>>>>>>>> spark.driver.cores 1 >>>>>>>>> spark.ssl.protocol TLSv1.2 >>>>>>>>> spark.ssl.keyStorePassword password >>>>>>>>> spark.yarn.maxAppAttempts 1 >>>>>>>>> spark.ssl.keyStore /etc/emr/security/keystore.jks >>>>>>>>> spark.authenticate true >>>>>>>>> spark.ssl.keyPassword password >>>>>>>>> spark.ssl.enabled true >>>>>>>>> spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_CBC_SHA >>>>>>>>> spark.ssl.trustStore /etc/emr/security/truststore.jks >>>>>>>>> spark.authenticate.secret secret >>>>>>>>> spark.ssl.trustStorePassword password >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim < >>>>>>>>> hss...@nflabs.com> wrote: >>>>>>>>> >>>>>>>>>> Hi. >>>>>>>>>> Could you share your conf/zeppelin-env.sh and spark-defaults.conf >>>>>>>>>> ? >>>>>>>>>> >>>>>>>>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy < >>>>>>>>>> jonathan.esterh...@gmail.com>: >>>>>>>>>> >>>>>>>>>>> I am having trouble using zeppelin in a spark cluster that has >>>>>>>>>>> spark node authentication turned on (e.g. with >>>>>>>>>>> spark.authenticate=true, >>>>>>>>>>> spark.authenticate.secret=...) >>>>>>>>>>> >>>>>>>>>>> Notebook code that calls built-in spark functions (or other >>>>>>>>>>> things on executor classpath) work fine, but functions defined in >>>>>>>>>>> the >>>>>>>>>>> notebook (anonymous or named) throw ClassNotFoundExceptions when >>>>>>>>>>> called >>>>>>>>>>> from an executor. >>>>>>>>>>> >>>>>>>>>>> For example, this code works: >>>>>>>>>>> >>>>>>>>>>> val rdd = sc.textFile("hdfs://my-text-file") >>>>>>>>>>> rdd.take(1).foreach(println) >>>>>>>>>>> >>>>>>>>>>> rdd.saveAsTextFile("hdfs:///my-other-text-file") >>>>>>>>>>> >>>>>>>>>>> but code like this... >>>>>>>>>>> >>>>>>>>>>> rdd.filter(_.contains("my data")) >>>>>>>>>>> >>>>>>>>>>> fails with >>>>>>>>>>> >>>>>>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 >>>>>>>>>>> at >>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84) >>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>>>>>> at java.lang.Class.forName(Class.java:348) >>>>>>>>>>> ... >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I get the same kind of error if the filter function is defined >>>>>>>>>>> as a named function in the notebook, or as a member of singleton >>>>>>>>>>> object >>>>>>>>>>> defined in the notebook. >>>>>>>>>>> >>>>>>>>>>> When I look at the executor's log output, I see this error: >>>>>>>>>>> >>>>>>>>>>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to >>>>>>>>>>> check existence of class >>>>>>>>>>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on >>>>>>>>>>> REPL >>>>>>>>>>> class server at https://172.30.54.30:34980 >>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>> at >>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113) >>>>>>>>>>> at >>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146) >>>>>>>>>>> at >>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76) >>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>>>>>> at java.lang.Class.forName(Class.java:348) >>>>>>>>>>> >>>>>>>>>>> ... >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> If I disable spark authentication, everything works as expected. >>>>>>>>>>> I am running zeppelin 0.5.6 on spark 1.6.1 with yarn. >>>>>>>>>>> >>>>>>>>>>> Has anyone been able to get zeppelin working with spark >>>>>>>>>>> authentication? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
