Tae Jun Kim created ZEPPELIN-1511:
-------------------------------------
Summary: Problem with Spark interpreter dependency loading using
interpreter setting GUI
Key: ZEPPELIN-1511
URL: https://issues.apache.org/jira/browse/ZEPPELIN-1511
Project: Zeppelin
Issue Type: Bug
Components: Interpreters
Affects Versions: 0.7.0
Environment: h2. Environments
- Spark version: 2.0
- Scala version: 2.11
- Zeppelin version: 0.7.0-SNAPSHOT
- OS: MacOS (Sierra)
- Core: 8
- Driver Mem: 4G
- Not using embedded Spark. I have $SPARK_HOME
- My build options:
{code}
./dev/change_scala_version.sh 2.11
mvn clean package -Pspark-2.0 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr
-Pscala-2.11 -DskipTests
{code}
Reporter: Tae Jun Kim
h2. Depenedency with problem
*{{org.bitbucket.eunjeon:seunjeon_2.11:1.1.0}}* (Korean Morpheme Ananlyzer)
It's success if you can import {{org.bitbucket.eunjeon.seunjeon.Analyzer}}
h2. Description
I'm trying to add the dependency with GUI(interpreter setting). It works just
fine with {{com.databricks:spark-xml_2.11:0.4.0}}, but not with
{{org.bitbucket.eunjeon:seunjeon_2.11:1.1.0}} :(
{{sc.version}} prints nothing. I will attach my screenshot.
However, it works with SPARK_SUBMIT_OPTIONS! Which is:
{code:title=conf/zeppelin-env.sh}
export SPARK_SUBMIT_OPTIONS="--packages
org.bitbucket.eunjeon:seunjeon_2.11:1.1.0 --driver-memory 4G"
{code}
So, for now, I'm using this method.
I also added my Spark interpreter log below.
I saw codes of problem, but I don't know why T_T
Please help!
h2. Spark Interpreter Log
{code:title=zeppelin-interpreter-spark-xxx.log}
INFO [2016-09-30 17:18:45,817] ({Thread-0}
RemoteInterpreterServer.java[run]:90) - Starting remote interpreter server on
port 54273
INFO [2016-09-30 17:18:49,468] ({pool-1-thread-2}
RemoteInterpreterServer.java[createInterpreter]:186) - Instantiate interpreter
org.apache.zeppelin.spark.SparkInterpreter
INFO [2016-09-30 17:18:49,488] ({pool-1-thread-2}
RemoteInterpreterServer.java[createInterpreter]:186) - Instantiate interpreter
org.apache.zeppelin.spark.SparkSqlInterpreter
INFO [2016-09-30 17:18:49,491] ({pool-1-thread-2}
RemoteInterpreterServer.java[createInterpreter]:186) - Instantiate interpreter
org.apache.zeppelin.spark.DepInterpreter
INFO [2016-09-30 17:18:49,495] ({pool-1-thread-2}
RemoteInterpreterServer.java[createInterpreter]:186) - Instantiate interpreter
org.apache.zeppelin.spark.PySparkInterpreter
INFO [2016-09-30 17:18:49,496] ({pool-1-thread-2}
RemoteInterpreterServer.java[createInterpreter]:186) - Instantiate interpreter
org.apache.zeppelin.spark.SparkRInterpreter
INFO [2016-09-30 17:18:49,524] ({pool-2-thread-2}
SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1475223529523
started by scheduler org.apache.zeppelin.spark.SparkInterpreter1779785062
ERROR [2016-09-30 17:18:49,988] ({pool-2-thread-2} Job.java[run]:189) - Job
failed
java.lang.NoSuchMethodError:
scala.reflect.internal.settings.MutableSettings$SettingValue.valueSetByUser()Lscala/Option;
at scala.tools.nsc.Global.<init>(Global.scala:334)
at scala.tools.nsc.interpreter.IMain$$anon$1.<init>(IMain.scala:247)
at scala.tools.nsc.interpreter.IMain.newCompiler(IMain.scala:247)
at scala.tools.nsc.interpreter.IMain.<init>(IMain.scala:93)
at scala.tools.nsc.interpreter.IMain.<init>(IMain.scala:113)
at
scala.tools.nsc.interpreter.ILoop$ILoopInterpreter.<init>(ILoop.scala:108)
at scala.tools.nsc.interpreter.ILoop.createInterpreter(ILoop.scala:118)
at
org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:722)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:390)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
INFO [2016-09-30 17:18:49,993] ({pool-2-thread-2}
SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1475223529523
finished by scheduler org.apache.zeppelin.spark.SparkInterpreter1779785062
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
