I think it is safe to merge into 0.6.1
On Fri, Jul 29, 2016 at 1:08 AM, Jonathan Esterhazy < jonathan.esterh...@gmail.com> wrote: > yes, this PR fixes my classnotfoundexceptions. awesome! > > any idea when it might make it into a release? > > On Wed, Jul 27, 2016 at 11:25 PM, Jeff Zhang <zjf...@gmail.com> wrote: > >> I think it is fixed in https://github.com/apache/zeppelin/pull/1237, >> please try that. >> >> On Thu, Jul 28, 2016 at 1:58 AM, Jonathan Esterhazy < >> jonathan.esterh...@gmail.com> wrote: >> >>> I just tried this with using zeppelin master, and the classnotfound >>> error is still there. >>> >>> >>> On Thu, Jul 7, 2016 at 7:07 AM, Hyung Sung Shim <hss...@nflabs.com> >>> wrote: >>> >>>> Hi. Jonathan. >>>> If it is possible, could you use the zeppelin master on the github? >>>> Zeppelin is working well with spark authentication when i tested. >>>> >>>> >>>> >>>> 2016-07-02 3:05 GMT+09:00 Jonathan Esterhazy < >>>> jonathan.esterh...@gmail.com>: >>>> >>>>> Hyung, thx for your help. I opened these: >>>>> >>>>> https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala >>>>> problem) >>>>> https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking >>>>> python problem) >>>>> >>>>> LMK if I can provide more info or help in some way. >>>>> >>>>> On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim <hss...@nflabs.com> >>>>> wrote: >>>>> >>>>>> Hi Jonathan. >>>>>> Unfortunately I got same error in my test bed. >>>>>> Do you mind create an jira issue for this? >>>>>> >>>>>> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy < >>>>>> jonathan.esterh...@gmail.com>: >>>>>> >>>>>>> I added this and still get the same exception. The same property is >>>>>>> also set in spark-defaults.conf. >>>>>>> >>>>>>> After that didn't work, I also tried adding --conf >>>>>>> spark.authenticate=true and --conf >>>>>>> spark.authenticate.enableSaslEncryption=true, to match the other related >>>>>>> settings in spark-defaults.conf. Still get the same >>>>>>> classnotfoundexception. >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim <hss...@nflabs.com >>>>>>> > wrote: >>>>>>> >>>>>>>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf >>>>>>>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and >>>>>>>> restart zeppelin and retry your code. >>>>>>>> >>>>>>>> >>>>>>>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy < >>>>>>>> jonathan.esterh...@gmail.com>: >>>>>>>> >>>>>>>>> yes it does. i only see this problem in zeppelin. >>>>>>>>> >>>>>>>>> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim < >>>>>>>>> hss...@nflabs.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Jonathan. >>>>>>>>>> It's not easy to build the test environments but I am working on >>>>>>>>>> this. >>>>>>>>>> I have question for you. >>>>>>>>>> Does your code working well on spark-shell in the >>>>>>>>>> spark.authenticate mode? >>>>>>>>>> >>>>>>>>>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy < >>>>>>>>>> jonathan.esterh...@gmail.com>: >>>>>>>>>> >>>>>>>>>>> Hyung, did you have any luck w/ zeppelin + spark authentication? >>>>>>>>>>> I'm quite stumped. >>>>>>>>>>> >>>>>>>>>>> thx. >>>>>>>>>>> >>>>>>>>>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim < >>>>>>>>>>> hss...@nflabs.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thank you. >>>>>>>>>>>> Let me try. >>>>>>>>>>>> >>>>>>>>>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy < >>>>>>>>>>>> jonathan.esterh...@gmail.com>: >>>>>>>>>>>> >>>>>>>>>>>>> Hyung, >>>>>>>>>>>>> >>>>>>>>>>>>> Yes, here they are. >>>>>>>>>>>>> >>>>>>>>>>>>> zeppelin-env.sh: >>>>>>>>>>>>> >>>>>>>>>>>>> export ZEPPELIN_PORT=8890 >>>>>>>>>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf >>>>>>>>>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin >>>>>>>>>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin >>>>>>>>>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid >>>>>>>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook >>>>>>>>>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps >>>>>>>>>>>>> export MASTER=yarn-client >>>>>>>>>>>>> export SPARK_HOME=/usr/lib/spark >>>>>>>>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf >>>>>>>>>>>>> export >>>>>>>>>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/* >>>>>>>>>>>>> :/usr/share/aws/emr/emrfs/auxlib/*" >>>>>>>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0 >>>>>>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket >>>>>>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin >>>>>>>>>>>>> export >>>>>>>>>>>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo >>>>>>>>>>>>> >>>>>>>>>>>>> spark-defaults.conf: >>>>>>>>>>>>> >>>>>>>>>>>>> spark.master yarn >>>>>>>>>>>>> spark.driver.extraClassPath >>>>>>>>>>>>> >>>>>>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf >>>>>>>>>>>>> >>>>>>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* >>>>>>>>>>>>> spark.driver.extraLibraryPath >>>>>>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native >>>>>>>>>>>>> spark.executor.extraClassPath >>>>>>>>>>>>> >>>>>>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf >>>>>>>>>>>>> >>>>>>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* >>>>>>>>>>>>> spark.executor.extraLibraryPath >>>>>>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native >>>>>>>>>>>>> spark.eventLog.enabled true >>>>>>>>>>>>> spark.eventLog.dir hdfs:///var/log/spark/apps >>>>>>>>>>>>> spark.history.fs.logDirectory hdfs:///var/log/spark/apps >>>>>>>>>>>>> spark.yarn.historyServer.address >>>>>>>>>>>>> ip-172-30-54-30.ec2.internal:18080 >>>>>>>>>>>>> spark.history.ui.port 18080 >>>>>>>>>>>>> spark.shuffle.service.enabled true >>>>>>>>>>>>> spark.driver.extraJavaOptions >>>>>>>>>>>>> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties >>>>>>>>>>>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 >>>>>>>>>>>>> -XX:MaxHeapFreeRatio=70 >>>>>>>>>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M >>>>>>>>>>>>> -XX:OnOutOfMemoryError='kill -9 %p' >>>>>>>>>>>>> spark.dynamicAllocation.enabled true >>>>>>>>>>>>> spark.executor.extraJavaOptions -verbose:gc >>>>>>>>>>>>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC >>>>>>>>>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 >>>>>>>>>>>>> -XX:+CM >>>>>>>>>>>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p' >>>>>>>>>>>>> spark.executor.memory 8640m >>>>>>>>>>>>> spark.executor.cores 7 >>>>>>>>>>>>> spark.authenticate.enableSaslEncryption true >>>>>>>>>>>>> spark.driver.memory 1g >>>>>>>>>>>>> spark.network.sasl.serverAlwaysEncrypt true >>>>>>>>>>>>> spark.driver.cores 1 >>>>>>>>>>>>> spark.ssl.protocol TLSv1.2 >>>>>>>>>>>>> spark.ssl.keyStorePassword password >>>>>>>>>>>>> spark.yarn.maxAppAttempts 1 >>>>>>>>>>>>> spark.ssl.keyStore /etc/emr/security/keystore.jks >>>>>>>>>>>>> spark.authenticate true >>>>>>>>>>>>> spark.ssl.keyPassword password >>>>>>>>>>>>> spark.ssl.enabled true >>>>>>>>>>>>> spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_CBC_SHA >>>>>>>>>>>>> spark.ssl.trustStore >>>>>>>>>>>>> /etc/emr/security/truststore.jks >>>>>>>>>>>>> spark.authenticate.secret secret >>>>>>>>>>>>> spark.ssl.trustStorePassword password >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim < >>>>>>>>>>>>> hss...@nflabs.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi. >>>>>>>>>>>>>> Could you share your conf/zeppelin-env.sh >>>>>>>>>>>>>> and spark-defaults.conf ? >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy < >>>>>>>>>>>>>> jonathan.esterh...@gmail.com>: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am having trouble using zeppelin in a spark cluster that >>>>>>>>>>>>>>> has spark node authentication turned on (e.g. with >>>>>>>>>>>>>>> spark.authenticate=true, >>>>>>>>>>>>>>> spark.authenticate.secret=...) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Notebook code that calls built-in spark functions (or other >>>>>>>>>>>>>>> things on executor classpath) work fine, but functions defined >>>>>>>>>>>>>>> in the >>>>>>>>>>>>>>> notebook (anonymous or named) throw ClassNotFoundExceptions >>>>>>>>>>>>>>> when called >>>>>>>>>>>>>>> from an executor. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For example, this code works: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> val rdd = sc.textFile("hdfs://my-text-file") >>>>>>>>>>>>>>> rdd.take(1).foreach(println) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> rdd.saveAsTextFile("hdfs:///my-other-text-file") >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> but code like this... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> rdd.filter(_.contains("my data")) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> fails with >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84) >>>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>>>>>>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>>>>>>>>>> at java.lang.Class.forName(Class.java:348) >>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I get the same kind of error if the filter function is >>>>>>>>>>>>>>> defined as a named function in the notebook, or as a member of >>>>>>>>>>>>>>> singleton >>>>>>>>>>>>>>> object defined in the notebook. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> When I look at the executor's log output, I see this error: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to >>>>>>>>>>>>>>> check existence of class >>>>>>>>>>>>>>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 >>>>>>>>>>>>>>> on REPL >>>>>>>>>>>>>>> class server at https://172.30.54.30:34980 >>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76) >>>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>>>>>>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>>>>>>>>>> at java.lang.Class.forName(Class.java:348) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If I disable spark authentication, everything works as >>>>>>>>>>>>>>> expected. I am running zeppelin 0.5.6 on spark 1.6.1 with yarn. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Has anyone been able to get zeppelin working with spark >>>>>>>>>>>>>>> authentication? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > -- Best Regards Jeff Zhang
