I am having trouble using zeppelin in a spark cluster that has spark node
authentication turned on (e.g. with spark.authenticate=true,
spark.authenticate.secret=...)
Notebook code that calls built-in spark functions (or other things on
executor classpath) work fine, but functions defined in the notebook
(anonymous or named) throw ClassNotFoundExceptions when called from an
executor.
For example, this code works:
val rdd = sc.textFile("hdfs://my-text-file")
rdd.take(1).foreach(println)
rdd.saveAsTextFile("hdfs:///my-other-text-file")
but code like this...
rdd.filter(_.contains("my data"))
fails with
Caused by: java.lang.ClassNotFoundException:
$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
at
org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
...
I get the same kind of error if the filter function is defined as a named
function in the notebook, or as a member of singleton object defined in the
notebook.
When I look at the executor's log output, I see this error:
16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to check existence
of class $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
on REPL class server at https://172.30.54.30:34980
java.lang.NullPointerException
at
org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113)
at
org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146)
at
org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
...
If I disable spark authentication, everything works as expected. I am
running zeppelin 0.5.6 on spark 1.6.1 with yarn.
Has anyone been able to get zeppelin working with spark authentication?
