Hi Jonathan. It's not easy to build the test environments but I am working on this. I have question for you. Does your code working well on spark-shell in the spark.authenticate mode?
2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <jonathan.esterh...@gmail.com> : > Hyung, did you have any luck w/ zeppelin + spark authentication? I'm quite > stumped. > > thx. > > On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <hss...@nflabs.com> > wrote: > >> Thank you. >> Let me try. >> >> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy < >> jonathan.esterh...@gmail.com>: >> >>> Hyung, >>> >>> Yes, here they are. >>> >>> zeppelin-env.sh: >>> >>> export ZEPPELIN_PORT=8890 >>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf >>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin >>> export ZEPPELIN_PID_DIR=/var/run/zeppelin >>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid >>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook >>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps >>> export MASTER=yarn-client >>> export SPARK_HOME=/usr/lib/spark >>> export HADOOP_CONF_DIR=/etc/hadoop/conf >>> export >>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/* >>> :/usr/share/aws/emr/emrfs/auxlib/*" >>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0 >>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket >>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin >>> export >>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo >>> >>> spark-defaults.conf: >>> >>> spark.master yarn >>> spark.driver.extraClassPath >>> >>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf >>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* >>> spark.driver.extraLibraryPath >>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native >>> spark.executor.extraClassPath >>> >>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf >>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* >>> spark.executor.extraLibraryPath >>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native >>> spark.eventLog.enabled true >>> spark.eventLog.dir hdfs:///var/log/spark/apps >>> spark.history.fs.logDirectory hdfs:///var/log/spark/apps >>> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080 >>> spark.history.ui.port 18080 >>> spark.shuffle.service.enabled true >>> spark.driver.extraJavaOptions >>> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties >>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 >>> -XX:MaxHeapFreeRatio=70 >>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M >>> -XX:OnOutOfMemoryError='kill -9 %p' >>> spark.dynamicAllocation.enabled true >>> spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails >>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC >>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM >>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p' >>> spark.executor.memory 8640m >>> spark.executor.cores 7 >>> spark.authenticate.enableSaslEncryption true >>> spark.driver.memory 1g >>> spark.network.sasl.serverAlwaysEncrypt true >>> spark.driver.cores 1 >>> spark.ssl.protocol TLSv1.2 >>> spark.ssl.keyStorePassword password >>> spark.yarn.maxAppAttempts 1 >>> spark.ssl.keyStore /etc/emr/security/keystore.jks >>> spark.authenticate true >>> spark.ssl.keyPassword password >>> spark.ssl.enabled true >>> spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_CBC_SHA >>> spark.ssl.trustStore /etc/emr/security/truststore.jks >>> spark.authenticate.secret secret >>> spark.ssl.trustStorePassword password >>> >>> >>> >>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim <hss...@nflabs.com> >>> wrote: >>> >>>> Hi. >>>> Could you share your conf/zeppelin-env.sh and spark-defaults.conf ? >>>> >>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy < >>>> jonathan.esterh...@gmail.com>: >>>> >>>>> I am having trouble using zeppelin in a spark cluster that has spark >>>>> node authentication turned on (e.g. with spark.authenticate=true, >>>>> spark.authenticate.secret=...) >>>>> >>>>> Notebook code that calls built-in spark functions (or other things on >>>>> executor classpath) work fine, but functions defined in the notebook >>>>> (anonymous or named) throw ClassNotFoundExceptions when called from an >>>>> executor. >>>>> >>>>> For example, this code works: >>>>> >>>>> val rdd = sc.textFile("hdfs://my-text-file") >>>>> rdd.take(1).foreach(println) >>>>> >>>>> rdd.saveAsTextFile("hdfs:///my-other-text-file") >>>>> >>>>> but code like this... >>>>> >>>>> rdd.filter(_.contains("my data")) >>>>> >>>>> fails with >>>>> >>>>> Caused by: java.lang.ClassNotFoundException: >>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 >>>>> at >>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>> at java.lang.Class.forName0(Native Method) >>>>> at java.lang.Class.forName(Class.java:348) >>>>> ... >>>>> >>>>> >>>>> I get the same kind of error if the filter function is defined as a >>>>> named function in the notebook, or as a member of singleton object defined >>>>> in the notebook. >>>>> >>>>> When I look at the executor's log output, I see this error: >>>>> >>>>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to check >>>>> existence of class >>>>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on REPL >>>>> class server at https://172.30.54.30:34980 >>>>> java.lang.NullPointerException >>>>> at >>>>> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113) >>>>> at >>>>> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146) >>>>> at >>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>> at java.lang.Class.forName0(Native Method) >>>>> at java.lang.Class.forName(Class.java:348) >>>>> >>>>> ... >>>>> >>>>> >>>>> If I disable spark authentication, everything works as expected. I am >>>>> running zeppelin 0.5.6 on spark 1.6.1 with yarn. >>>>> >>>>> Has anyone been able to get zeppelin working with spark authentication? >>>>> >>>>> >>>>> >>>> >>> >> >