Zepellin doesn't work with external Apache Spark, but I can launch Spark
with h2o package from shell.
I use a string export "SPARK_HOME=/path_with_spark_1.5" in zeppelin-env.sh.
But I'm not sure that zeppelin sees the extermal interpreter.
On Mon, Feb 29, 2016 at 7:35 PM, Silvio Fiorito <
silvio.fior...@granturing.com> wrote:
>
>
> Can you try running it from just a Spark shell to confirm it works that
> way (no other conflict)?
>
>
>
> bin/spark-shell --master local[*] --packages
> ai.h2o:sparkling-water-core_2.10:1.5.10
>
>
>
> Also, are you able to run the Spark interpreter without the h2o package?
>
>
>
> Thanks,
>
> Silvio
>
>
>
> *From: *Aleksandr Modestov <aleksandrmodes...@gmail.com>
> *Sent: *Monday, February 29, 2016 11:30 AM
> *To: *users@zeppelin.incubator.apache.org
> *Subject: *Re: problem with start H2OContent
>
>
> I use Spark 1.5
> The problem with the external Spark with internal Spark I can not launch
> h2oContent :)
> The error is:
>
> "ERROR [2016-02-29 19:28:16,609] ({pool-1-thread-3}
> NotebookServer.java[afterStatusChange]:766) - Error
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:268)
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:104)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:198)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:169)
> at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:322)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused
> at
> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
> at
> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
> at
> org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
> at
> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
> at
> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
> at
> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:139)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:266)
> ... 11 more
> Caused by: org.apache.thrift.transport.TTransportException:
> java.net.ConnectException: Connection refused
> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
> at
> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
> ... 18 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
> ... 19 more"
>
> On Mon, Feb 29, 2016 at 7:07 PM, Silvio Fiorito <
> silvio.fior...@granturing.com> wrote:
>
>> In your zeppelin-env you set SPARK_HOME and SPARK_SUBMIT_OPTIONS ?
>> Anything in the logs? Looks like the interpreter failed to start.
>>
>>
>>
>> Also, Sparkling Water currently supports up to 1.5 only, last I checked.
>>
>>
>>
>> Thanks,
>>
>> Silvio
>>
>>
>>
>>
>>
>>
>>
>> *From: *Aleksandr Modestov <aleksandrmodes...@gmail.com>
>> *Sent: *Monday, February 29, 2016 10:43 AM
>> *To: *users@zeppelin.incubator.apache.org
>> *Subject: *Re: problem with start H2OContent
>>
>>
>> When I use external Spark I get exeption:
>>
>> java.net.ConnectException: Connection refused at
>> java.net.PlainSocketImpl.socketConnect(Native Method) at
>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>> at
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
>> java.net.Socket.connect(Socket.java:579) at
>> org.apache.thrift.transport.TSocket.open(TSocket.java:182) at
>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
>> at
>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
>> at
>> org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
>> at
>> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
>> at
>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
>> at
>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:139)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(RemoteInterpreter.java:129)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:257)
>> at
>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:104)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:198) at
>> org.apache.zeppelin.scheduler.Job.run(Job.java:169) at
>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:322)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> On Mon, Feb 29, 2016 at 5:43 PM, Silvio Fiorito <
>> silvio.fior...@granturing.com> wrote:
>>
>>> It doesn’t seem to be loading transitive dependencies properly. When I
>>> was helping someone else set this up recently, I had to use
>>> SPARK_SUBMIT_OPTIONS=“--packages ai.h2o:sparkling-water-core_2.10:1.5.10”
>>> with an external Spark installation (vs using bundled Spark in Zeppelin).
>>>
>>> From: Aleksandr Modestov <aleksandrmodes...@gmail.com>
>>> Reply-To: "users@zeppelin.incubator.apache.org" <
>>> users@zeppelin.incubator.apache.org>
>>> Date: Monday, February 29, 2016 at 9:30 AM
>>> To: "users@zeppelin.incubator.apache.org" <
>>> users@zeppelin.incubator.apache.org>
>>> Subject: Re: problem with start H2OContent
>>>
>>> In a conf-file I wrote package but it doesn't work and I use '
>>> z.load("...")'.
>>>
>>> On Mon, Feb 29, 2016 at 5:25 PM, vincent gromakowski <
>>> vincent.gromakow...@gmail.com> wrote:
>>>
>>>> your H2O jar is not loaded in spark classpath. Maybe retry to load with
>>>> z.load("...") or add spark.jars parameter in spark interpreter
>>>> configuration
>>>>
>>>> 2016-02-29 15:23 GMT+01:00 Aleksandr Modestov <
>>>> aleksandrmodes...@gmail.com>:
>>>>
>>>>> I did import "import org.apache.spark.h2o._"
>>>>> What do you mean " it's probably a problem with your classpath."?
>>>>>
>>>>>
>>>>> On Mon, Feb 29, 2016 at 5:19 PM, vincent gromakowski <
>>>>> vincent.gromakow...@gmail.com> wrote:
>>>>>
>>>>>> Don't forget to do the import. If done it's probably a problem with
>>>>>> your classpath...
>>>>>>
>>>>>> 2016-02-29 15:03 GMT+01:00 Aleksandr Modestov <
>>>>>> aleksandrmodes...@gmail.com>:
>>>>>>
>>>>>>> Hello all,
>>>>>>> There is a problem when I start to initialize H2OContent.
>>>>>>> Does anybody know the answer?
>>>>>>>
>>>>>>> java.lang.NoClassDefFoundError: water/api/HandlerFactory at
>>>>>>> org.apache.spark.h2o.H2OContext.start(H2OContext.scala:107) at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:65)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:70)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:72)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:74)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:76)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:78)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:80)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:82)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:84)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:86)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:88)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:90)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:92)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:94)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:96)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:98)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:100)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:102)
>>>>>>> at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:104)
>>>>>>> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:106) at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:108) at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:110) at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:112) at
>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:114) at
>>>>>>> $iwC$$iwC$$iwC$$iwC.<init>(<console>:116) at
>>>>>>> $iwC$$iwC$$iwC.<init>(<console>:118) at $iwC$$iwC.<init>(<console>:120)
>>>>>>> at
>>>>>>> $iwC.<init>(<console>:122) at <init>(<console>:124) at
>>>>>>> .<init>(<console>:128) at .<clinit>(<console>) at .<init>(<console>:7)
>>>>>>> at
>>>>>>> .<clinit>(<console>) at $print(<console>) at
>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>> at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) at
>>>>>>> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>>>>>>> at
>>>>>>> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
>>>>>>> at
>>>>>>> org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>>>>>>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) at
>>>>>>> org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) at
>>>>>>> org.apache.zeppelin.spark.SparkInterpreter.interpretInput(SparkInterpreter.java:709)
>>>>>>> at
>>>>>>> org.apache.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:674)
>>>>>>> at
>>>>>>> org.apache.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:667)
>>>>>>> at
>>>>>>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57)
>>>>>>> at
>>>>>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
>>>>>>> at
>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:300)
>>>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:169) at
>>>>>>> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:134)
>>>>>>> at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>>> at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745) Caused by:
>>>>>>> java.lang.ClassNotFoundException: water.api.HandlerFactory at
>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
>>>>>>> java.security.AccessController.doPrivileged(Native Method) at
>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 59 more
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
