Hi Jonathan.
Unfortunately I got same error in my test bed.
Do you mind create an jira issue for this?
2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <jonathan.esterh...@gmail.com>:
> I added this and still get the same exception. The same property is also
> set in spark-defaults.conf.
>
> After that didn't work, I also tried adding --conf spark.authenticate=true
> and --conf spark.authenticate.enableSaslEncryption=true, to match the other
> related settings in spark-defaults.conf. Still get the same
> classnotfoundexception.
>
>
> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim <hss...@nflabs.com>
> wrote:
>
>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and restart
>> zeppelin and retry your code.
>>
>>
>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com>:
>>
>>> yes it does. i only see this problem in zeppelin.
>>>
>>> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <hss...@nflabs.com>
>>> wrote:
>>>
>>>> Hi Jonathan.
>>>> It's not easy to build the test environments but I am working on this.
>>>> I have question for you.
>>>> Does your code working well on spark-shell in the spark.authenticate
>>>> mode?
>>>>
>>>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>>>> jonathan.esterh...@gmail.com>:
>>>>
>>>>> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm
>>>>> quite stumped.
>>>>>
>>>>> thx.
>>>>>
>>>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <hss...@nflabs.com>
>>>>> wrote:
>>>>>
>>>>>> Thank you.
>>>>>> Let me try.
>>>>>>
>>>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>>>>> jonathan.esterh...@gmail.com>:
>>>>>>
>>>>>>> Hyung,
>>>>>>>
>>>>>>> Yes, here they are.
>>>>>>>
>>>>>>> zeppelin-env.sh:
>>>>>>>
>>>>>>> export ZEPPELIN_PORT=8890
>>>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>>>>>> export MASTER=yarn-client
>>>>>>> export SPARK_HOME=/usr/lib/spark
>>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>>> export
>>>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>>>>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>>>>>> export
>>>>>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>>>>>>
>>>>>>> spark-defaults.conf:
>>>>>>>
>>>>>>> spark.master yarn
>>>>>>> spark.driver.extraClassPath
>>>>>>>
>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>> spark.driver.extraLibraryPath
>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>> spark.executor.extraClassPath
>>>>>>>
>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>> spark.executor.extraLibraryPath
>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>> spark.eventLog.enabled true
>>>>>>> spark.eventLog.dir hdfs:///var/log/spark/apps
>>>>>>> spark.history.fs.logDirectory hdfs:///var/log/spark/apps
>>>>>>> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
>>>>>>> spark.history.ui.port 18080
>>>>>>> spark.shuffle.service.enabled true
>>>>>>> spark.driver.extraJavaOptions
>>>>>>> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
>>>>>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
>>>>>>> -XX:MaxHeapFreeRatio=70
>>>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
>>>>>>> -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>> spark.dynamicAllocation.enabled true
>>>>>>> spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails
>>>>>>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
>>>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
>>>>>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>> spark.executor.memory 8640m
>>>>>>> spark.executor.cores 7
>>>>>>> spark.authenticate.enableSaslEncryption true
>>>>>>> spark.driver.memory 1g
>>>>>>> spark.network.sasl.serverAlwaysEncrypt true
>>>>>>> spark.driver.cores 1
>>>>>>> spark.ssl.protocol TLSv1.2
>>>>>>> spark.ssl.keyStorePassword password
>>>>>>> spark.yarn.maxAppAttempts 1
>>>>>>> spark.ssl.keyStore /etc/emr/security/keystore.jks
>>>>>>> spark.authenticate true
>>>>>>> spark.ssl.keyPassword password
>>>>>>> spark.ssl.enabled true
>>>>>>> spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_CBC_SHA
>>>>>>> spark.ssl.trustStore /etc/emr/security/truststore.jks
>>>>>>> spark.authenticate.secret secret
>>>>>>> spark.ssl.trustStorePassword password
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim <hss...@nflabs.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi.
>>>>>>>> Could you share your conf/zeppelin-env.sh and spark-defaults.conf ?
>>>>>>>>
>>>>>>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy <
>>>>>>>> jonathan.esterh...@gmail.com>:
>>>>>>>>
>>>>>>>>> I am having trouble using zeppelin in a spark cluster that has
>>>>>>>>> spark node authentication turned on (e.g. with
>>>>>>>>> spark.authenticate=true,
>>>>>>>>> spark.authenticate.secret=...)
>>>>>>>>>
>>>>>>>>> Notebook code that calls built-in spark functions (or other things
>>>>>>>>> on executor classpath) work fine, but functions defined in the
>>>>>>>>> notebook
>>>>>>>>> (anonymous or named) throw ClassNotFoundExceptions when called from an
>>>>>>>>> executor.
>>>>>>>>>
>>>>>>>>> For example, this code works:
>>>>>>>>>
>>>>>>>>> val rdd = sc.textFile("hdfs://my-text-file")
>>>>>>>>> rdd.take(1).foreach(println)
>>>>>>>>>
>>>>>>>>> rdd.saveAsTextFile("hdfs:///my-other-text-file")
>>>>>>>>>
>>>>>>>>> but code like this...
>>>>>>>>>
>>>>>>>>> rdd.filter(_.contains("my data"))
>>>>>>>>>
>>>>>>>>> fails with
>>>>>>>>>
>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
>>>>>>>>> at
>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I get the same kind of error if the filter function is defined as
>>>>>>>>> a named function in the notebook, or as a member of singleton object
>>>>>>>>> defined in the notebook.
>>>>>>>>>
>>>>>>>>> When I look at the executor's log output, I see this error:
>>>>>>>>>
>>>>>>>>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to check
>>>>>>>>> existence of class
>>>>>>>>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on
>>>>>>>>> REPL
>>>>>>>>> class server at https://172.30.54.30:34980
>>>>>>>>> java.lang.NullPointerException
>>>>>>>>> at
>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76)
>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If I disable spark authentication, everything works as expected. I
>>>>>>>>> am running zeppelin 0.5.6 on spark 1.6.1 with yarn.
>>>>>>>>>
>>>>>>>>> Has anyone been able to get zeppelin working with spark
>>>>>>>>> authentication?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
