Hello DuyHai, Original problem reported by Arpan Saha is related to Datastax. I am using Spark + Zeppelin. Below is the configuration.
- Spark 2.0.2 - Zeppelin: 0.6.2 - Java 1.8.0_111 - R: 3.3.1 Thanks, Abul On Wed, Nov 16, 2016 at 3:44 PM, DuyHai Doan <doanduy...@gmail.com> wrote: > I recommend downloading my pre-built Zeppelin for Datastax. Shared folder > link: https://drive.google.com/folderview?id=0B6wR2aj4Cb6wQ01aR3ItR0xUNms > > On Wed, Nov 16, 2016 at 11:13 AM, DuyHai Doan <doanduy...@gmail.com> > wrote: > >> Ok I understand why you have issue. >> >> You are using Spark 2.0.2 and latest Datastax 5.0.3 is still using Spark >> version 1.6.X >> >> >> >> On Wed, Nov 16, 2016 at 10:23 AM, Abul Basar <aba...@einext.com> wrote: >> >>> I am facing a similar issue while using Spark R. >>> >>> My environment: >>> >>> - Spark 2.0.2 >>> - Zeppelin: 0.6.2 >>> - Java 1.8.0_111 >>> - R: 3.3.1 >>> >>> >>> SPARK_HOME is set. I am trying to run a simple paragraph. >>> >>> %r >>> print("hello ...") >>> >>> I get the following exception >>> >>> *Interpreter Log* >>> > # getZeppelinR >>> >>> > .zeppelinR = SparkR:::callJStatic("org.apache.zeppelin.spark.ZeppelinR", >>> "getZeppelinR", hashCode) >>> >>> at org.apache.zeppelin.spark.ZeppelinR.waitForRScriptInitialize >>> d(ZeppelinR.java:295) >>> at org.apache.zeppelin.spark.ZeppelinR.request(ZeppelinR.java:235) >>> at org.apache.zeppelin.spark.ZeppelinR.eval(ZeppelinR.java:183) >>> at org.apache.zeppelin.spark.ZeppelinR.open(ZeppelinR.java:172) >>> at org.apache.zeppelin.spark.SparkRInterpreter.open(SparkRInter >>> preter.java:85) >>> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(Laz >>> yOpenInterpreter.java:69) >>> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgr >>> ess(LazyOpenInterpreter.java:110) >>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServ >>> er.getProgress(RemoteInterpreterServer.java:404) >>> at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterServ >>> ice$Processor$getProgress.getResult(RemoteInterpreterService.java:1509) >>> at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterServ >>> ice$Processor$getProgress.getResult(RemoteInterpreterService.java:1494) >>> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) >>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) >>> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run >>> (TThreadPoolServer.java:285) >>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >>> *Zeppelin Log* >>> ERROR [2016-11-16 14:42:05,664] ({Thread-377} >>> JobProgressPoller.java[run]:54) - Can not get or update progress >>> org.apache.zeppelin.interpreter.InterpreterException: >>> org.apache.thrift.transport.TTransportException >>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.get >>> Progress(RemoteInterpreter.java:373) >>> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgr >>> ess(LazyOpenInterpreter.java:111) >>> at org.apache.zeppelin.notebook.Paragraph.progress(Paragraph.java:237) >>> at org.apache.zeppelin.scheduler.JobProgressPoller.run(JobProgr >>> essPoller.java:51) >>> Caused by: org.apache.thrift.transport.TTransportException >>> at org.apache.thrift.transport.TIOStreamTransport.read(TIOStrea >>> mTransport.java:132) >>> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) >>> at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryPr >>> otocol.java:429) >>> at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryPr >>> otocol.java:318) >>> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin( >>> TBinaryProtocol.java:219) >>> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) >>> at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterServ >>> ice$Client.recv_getProgress(RemoteInterpreterService.java:296) >>> at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterServ >>> ice$Client.getProgress(RemoteInterpreterService.java:281) >>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.get >>> Progress(RemoteInterpreter.java:370) >>> ... 3 more >>> >>> >>> >>> When I tried run R commands using sparkR by launching >>> $SPARK_HOME/bin/sparkR, it works. >>> >>> >>> Would much appreciate any help. >>> >>> Thanks! >>> Abul >>> >>> >>> On Fri, Sep 16, 2016 at 5:36 PM, Hyung Sung Shim <hss...@nflabs.com> >>> wrote: >>> >>>> Hello. >>>> I think that `zeppelin.interpreter.host` and `zeppelin.interpreter.port` >>>> values on Interpreter setting UI should be removed because Interpreter >>>> will automatically be launched when you run a paragraph. >>>> >>>> 2016-09-16 3:56 GMT+09:00 Arpan Saha <arpan.s...@mindtree.com>: >>>> >>>>> Yes. It is a zeppelin log. Not spark interpreter log. Can you please >>>>> tell me where can I get spark interpreter log? I don't find anything under >>>>> zeppelin/logs/ path. >>>>> >>>>> Thanks, >>>>> Arpan >>>>> ________________________________________ >>>>> From: DuyHai Doan [doanduy...@gmail.com] >>>>> Sent: Thursday, September 15, 2016 11:20 AM >>>>> To: users@zeppelin.apache.org >>>>> Subject: Re: FW: Issue with Zeppelin setup on Datastax-Spark >>>>> >>>>> Is it Zeppelin log or Spark interpreter log ? I guess we need also the >>>>> spark interpreter log to understand what is breaking Spark under the hood. >>>>> org.apache.thrift.transport.TTransportException is just technical >>>>> exception on inter-jvm communication >>>>> >>>>> On Thu, Sep 15, 2016 at 7:13 PM, Arpan Saha <arpan.s...@mindtree.com >>>>> <mailto:arpan.s...@mindtree.com>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> Please find below full stack trace. >>>>> >>>>> ERROR [2016-09-15 16:49:15,117] ({pool-1-thread-2} >>>>> RemoteInterpreter.java[init]:182) - Failed to create interpreter: >>>>> org.apache.zeppelin.spark.SparkInterpreter >>>>> ERROR [2016-09-15 16:49:15,118] ({pool-1-thread-2} Job.java[run]:189) >>>>> - Job failed >>>>> org.apache.zeppelin.interpreter.InterpreterException: >>>>> org.apache.thrift.transport.TTransportException >>>>> at org.apache.zeppelin.interprete >>>>> r.remote.RemoteInterpreter.init(RemoteInterpreter.java:183) >>>>> at org.apache.zeppelin.interprete >>>>> r.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:328) >>>>> at org.apache.zeppelin.interprete >>>>> r.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:105) >>>>> at org.apache.zeppelin.notebook.P >>>>> aragraph.jobRun(Paragraph.java:260) >>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:176) >>>>> at org.apache.zeppelin.scheduler. >>>>> RemoteScheduler$JobRunner.run(RemoteScheduler.java:328) >>>>> at java.util.concurrent.Executors >>>>> $RunnableAdapter.call(Executors.java:511) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>> at java.util.concurrent.Scheduled >>>>> ThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledT >>>>> hreadPoolExecutor.java:180) >>>>> at java.util.concurrent.Scheduled >>>>> ThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPo >>>>> olExecutor.java:293) >>>>> at java.util.concurrent.ThreadPoo >>>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>> at java.util.concurrent.ThreadPoo >>>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> Caused by: org.apache.thrift.transport.TTransportException >>>>> at org.apache.thrift.transport.TI >>>>> OStreamTransport.read(TIOStreamTransport.java:132) >>>>> at org.apache.thrift.transport.TT >>>>> ransport.readAll(TTransport.java:86) >>>>> at org.apache.thrift.protocol.TBi >>>>> naryProtocol.readAll(TBinaryProtocol.java:429) >>>>> at org.apache.thrift.protocol.TBi >>>>> naryProtocol.readI32(TBinaryProtocol.java:318) >>>>> at org.apache.thrift.protocol.TBi >>>>> naryProtocol.readMessageBegin(TBinaryProtocol.java:219) >>>>> at org.apache.thrift.TServiceClie >>>>> nt.receiveBase(TServiceClient.java:69) >>>>> at org.apache.zeppelin.interprete >>>>> r.thrift.RemoteInterpreterService$Client.recv_createInterpre >>>>> ter(RemoteInterpreterService.java:184) >>>>> at org.apache.zeppelin.interprete >>>>> r.thrift.RemoteInterpreterService$Client.createInterpreter(R >>>>> emoteInterpreterService.java:168) >>>>> at org.apache.zeppelin.interprete >>>>> r.remote.RemoteInterpreter.init(RemoteInterpreter.java:172) >>>>> ... 12 more >>>>> ERROR [2016-09-15 16:49:15,118] ({Thread-18} >>>>> RemoteInterpreterEventPoller.java[run]:88) - Can't get >>>>> RemoteInterpreterEvent >>>>> >>>>> Thanks, >>>>> Arpan. >>>>> >>>>> ________________________________________ >>>>> From: DuyHai Doan [doanduy...@gmail.com<mailto:doanduy...@gmail.com>] >>>>> Sent: Thursday, September 15, 2016 4:29 AM >>>>> To: users@zeppelin.apache.org<mailto:users@zeppelin.apache.org> >>>>> Subject: Re: Issue with Zeppelin setup on Datastax-Spark >>>>> >>>>> Can you provide the complete stack trace ? >>>>> >>>>> Also, worth reading my blog post about integrating Spark/Zeppelin/C* >>>>> or DSE : http://www.doanduyhai.com/blog/?p=2325 >>>>> >>>>> On Thu, Sep 15, 2016 at 5:17 AM, Arpan Saha <arpan.s...@mindtree.com >>>>> <mailto:arpan.s...@mindtree.com><mailto:arpan.s...@mindtree.com<ma >>>>> ilto:arpan.s...@mindtree.com>>> wrote: >>>>> Hi Team, >>>>> >>>>> I am trying to integrate Zeppelin 0.6.0 with DataStax 4.8.8 (which has >>>>> Spark 1.4.2). After I configured following properties in zeppelin-env.sh >>>>> when I start zeppelin daemon it started and in the browser I can see >>>>> zeppelin is running but when I am trying to execute spark query in the >>>>> notebook it is throwing below Error. Could you please help me to solve >>>>> this >>>>> issue. >>>>> >>>>> export JAVA_HOME=<JAVA Home Path> >>>>> export SPARK_HOME=/etc/dse/spark >>>>> export HADOOP_CONF_DIR=/etc/dse/hadoop >>>>> export MASTER=spark://<masterhost>:7077 >>>>> >>>>> Also, added/updated below properties in the Spark Interpreter screen >>>>> of Zeppelin UI. >>>>> >>>>> Master >>>>> spark.app.name<http://spark.app.name><http://spark.app.name> >>>>> spark.cassandra.auth.password >>>>> spark.cassandra.auth.username >>>>> spark.cassandra.connection.host >>>>> spark.cores.max >>>>> spark.executor.memory >>>>> zeppelin.interpreter.host >>>>> zeppelin.interpreter.port >>>>> >>>>> Trying to execute below statement in the notebook: >>>>> %spark >>>>> sc.version >>>>> >>>>> Below Error is getting: >>>>> java.net.SocketException: Connection reset at >>>>> java.net.SocketInputStream.read(SocketInputStream.java:209) at >>>>> java.net.SocketInputStream.read(SocketInputStream.java:141) at >>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at >>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at >>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:345) at >>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) >>>>> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) >>>>> at >>>>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) >>>>> at >>>>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) >>>>> at >>>>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) >>>>> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) >>>>> at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterServ >>>>> ice$Client.recv_createInterpreter(RemoteInterpreterService.java:184) >>>>> at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterServ >>>>> ice$Client.createInterpreter(RemoteInterpreterService.java:168) at >>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(RemoteInterpreter.java:172) >>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.get >>>>> FormType(RemoteInterpreter.java:328) at org.apache.zeppelin.interprete >>>>> r.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:105) >>>>> >>>>> users-subscr...@zeppelin.apache.org<mailto:users-subscribe@z >>>>> eppelin.apache.org><mailto:users-subscr...@zeppelin.apache.org<mailto: >>>>> users-subscr...@zeppelin.apache.org>> >>>>> >>>>> Thanks, >>>>> Arpan. >>>>> >>>>> ________________________________ >>>>> Notice: This e-mail message has been sent by an employee or contractor >>>>> of American Express Global Business Travel (“GBT”), a joint venture that >>>>> is >>>>> not wholly-owned by American Express Company or any of its subsidiaries >>>>> (“American Express”). E-mail from GBT employees and contractors may >>>>> continue to utilize an American Express domain for some period; however >>>>> American Express is not responsible for the content of this email, which >>>>> is >>>>> controlled by GBT. “American Express Global Business Travel”, “American >>>>> Express” and the American Express Logo are trademarks of American Express >>>>> and are used by GBT under limited license. >>>>> ________________________________ >>>>> Important : Cet email a été envoyé par un employé ou un sous-traitant >>>>> d’American Express Global Business Travel (« GBT »), une joint-venture qui >>>>> n’est pas intégralement détenue par American Express Company ou l’une de >>>>> ses filiales (« American Express »). Les emails des employés et >>>>> sous-traitants de GBT peuvent continuer de comporter un nom de domaine >>>>> d’American Express pour une certaine période ; toutefois, American Express >>>>> n’est pas responsable du contenu de cet email, qui est contrôlé par GBT. « >>>>> American Express Global Business Travel », « American Express » et le logo >>>>> American Express sont des marques d’American Express et sont utilisées par >>>>> GBT aux termes d’une licence limitée. >>>>> >>>>> ________________________________ >>>>> >>>>> http://www.mindtree.com/email/disclaimer.html >>>>> >>>>> >>>>> >>>> >>> >> >
