Re: Thrift Exception when executing a custom written Interpreter

[Fazlan Nazeem]

Hi Moon.

Thanks for the reply. I was able to pin down the problem by analyzing the
log file for the interpreter. It was returning a null FromType. Thanks :)

On Thu, Jan 14, 2016 at 9:26 PM, moon soo Lee  wrote:

> Hi,
>
> The error can happen when ZeppelinServer can not connect to the
> interpreter process. You can check log file for your custom interpreter
>
> logs/zeppelin-interpreter-[your interpreter name]-*
>
> And the log file will have some clue.
>
> Thanks,
> moon
>
> On Wed, Jan 13, 2016 at 10:10 PM Fazlan Nazeem  wrote:
>
>> Hi,
>>
>> I wrote a sample interpreter and configured it. When I type in the name
>> of the interpreter and click run I get the following error in the notebook.
>>
>> org.apache.thrift.transport.TTransportException at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at
>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
>> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_getFormType(RemoteInterpreterService.java:288)
>> at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.getFormType(RemoteInterpreterService.java:275)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:273)
>> at
>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:104)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:198) at
>> org.apache.zeppelin.scheduler.Job.run(Job.java:171) at
>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:326)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> How do I solve this issue?
>>
>> --
>> Thanks & Regards,
>>
>> Fazlan Nazeem
>>
>>


-- 
Thanks & Regards,

Fazlan Nazeem

*Software Engineer*

*WSO2 Inc*
Mobile : +94772338839
<%2B94%20%280%29%20773%20451194>
fazl...@wso2.com

Re: Notebook Rest API does not work with Zeppelin EMR (while other rest APIs works)

[Jonathan Kelly]

Ophir,

It's not possible to have stop/start be the last part of the line, as this
is simply how upstart works. stop/start in this case is actually the name
of the script. They are actually just symlinks to initctl. (That is,
/sbin/{stop,start,etc.} are just symlinks to /sbin/initctl, and
/sbin/initctl can either be called directly with an action as one of the
arguments, or by using one of the symlinks to indicate the action. See "man
initctl" for more information.)

What is your use case for needing to restart the Zeppelin server on EMR so
often?

And yes, it seems that there is a bug in the stop script for Zeppelin. I
think I have fixed it and will include the fix in a future release of EMR.
Sorry for any inconvenience this has caused you! In the meantime you can
kill all processes being run by the zeppelin user, and the Zeppelin server
will automatically start back up in a few seconds (so long as you have not
run "sudo stop zeppelin").

~ Jonathan

On Sun, Jan 17, 2016 at 2:16 AM Ophir Cohen  wrote:

> Got you.
> OK, good to know that you guys moved to upstart.
> Personally I think that the stop/start command should be the last in the
> line as this is what you mostly change and it is more easy to change.
> In test mode you need to go back to the start/stop all the time and its
> pretty annoying.
>
> Anyway, can it be that the stop command does not work?
> For some reason the start command starts my zeppelin but stop does not,
> well, stop...
>
> On Fri, Jan 15, 2016 at 7:22 PM, Jonathan Kelly 
> wrote:
>
>> Thanks!
>>
>> I think rather than "sudo /etc/init.d/zeppelin start" the usual way would
>> be "sudo service zeppelin start". However, on EMR 4.x, we actually use
>> upstart (http://upstart.ubuntu.com) to manage the service processes and
>> ensure that they are restarted automatically if they die, so the
>> corresponding command on EMR is "sudo start zeppelin".
>>
>> On Jan 14, 2016, at 11:13 PM, Ophir Cohen  wrote:
>>
>> Yes,  I just checked that out and find out it does not work on 0.6.
>> Ok, good luck, you are doing great job in emr!
>>
>> BTW
>> If I may say, you need to add service support to Zeppelin.
>> I.e. to allow start of Zeppelin this way:
>> sudo /etc/init.d/zeppelin start
>>
>> *that will start zeppelin under Zeppelin user.
>> On Jan 15, 2016 9:03 AM, "Jonathan Kelly"  wrote:
>>
>>> Yes, there are a lot of changes since 0.5.5, of course, but we're not
>>> going to support an unreleased version on EMR. We did that for emr-4.1.0
>>> (the first version to support Zeppelin), but it's much less confusing for
>>> us to support only released versions on EMR. As I mentioned earlier, we
>>> will upgrade to 0.5.6 once it's released, but as I also mentioned earlier,
>>> while the Notebook API appears fixed on 0.5.6, the Job API is not. The
>>> Job API, btw, is what allows you to run notebooks via the REST API. The
>>> Notebook API only lets you do CRUD on notebooks/paragraphs/crons.
>>>
>>> FWIW, I also tried building Zeppelin from the tip of the master branch
>>> but still hit the same error when using the Job API, but that's not too
>>> surprising because v0.5.6 isn't too different from the tip of the master
>>> branch, particularly in the REST API code (other than the new Shiro
>>> Security API).
>>>
>>> On Thu, Jan 14, 2016 at 10:45 PM Ophir Cohen  wrote:
>>>
 I havn't try the job api, in fact what is the differences between
 notebook and job api?
 Anyway, why not jumping to 0.6.0?
 Looks like there is MANY improvments and bug fixes there
 Sounds to me that you are debugging last year snow 😉
 On Jan 15, 2016 3:07 AM, "Jonathan Kelly" 
 wrote:

> But now some bad news. I was playing around with the REST API a little
> more, and I noticed that I get the same AbstractMethodError on Zeppelin
> 0.5.6 for a POST to /api/job/2A94M5J1Z. :( Does anybody have any idea how
> this error could possibly have been fixed for /api/notebook in one 
> release,
> for /api/notebook/* in the next release, and yet still fail for /api/job/*
> in that same release?
>
> On Thu, Jan 14, 2016 at 4:13 PM Jonathan Kelly 
> wrote:
>
>> Oh, hey, good news! I just tried Zeppelin 0.5.6 on EMR, and the
>> Notebook API works fine. So now we just need Zeppelin 0.5.6 to be
>> officially released, and then we can support it on a future version of 
>> EMR.
>>
>> On Thu, Jan 14, 2016 at 2:01 AM Ophir Cohen  wrote:
>>
>>> OK, I've made it work.
>>> As Jonathan stated above, on emr 4.1 the notebook APIs does not work
>>> at all. On both, emr 4.2 and emr 4.3 the /api/notebook works but the 
>>> query
>>> for a specific notebook not.
>>>
>>> Deploying vanilla Zeppelin (master branch) on the same cluster
>>> worked as a charm.
>>>
>>> It probably origin by Zeppelin versions (0.5.5 on emr vs. 0.6.0 the
>>> latest).
>>>
>>> Now I'm encountering different error that, as far as I can 

Re: Can I use PostgreSQL for other JDBC compliant database engine?

[Woonsan Ko]

Hi Jongyoul,

Thank you so much for the PR link, which looks really constructive and
cooperative. You guys are the best!

Cheers,

Woonsan


On Sun, Jan 17, 2016 at 11:50 AM, Jongyoul Lee  wrote:
> Hi Woonsan,
>
> We already have the PR or this issue. Could you please check
> https://github.com/apache/incubator-zeppelin/pull/361 ?
>
> Regards,
> Jongyoul
>
> On Sun, Jan 17, 2016 at 4:07 AM, Woonsan Ko  wrote:
>>
>> Hi,
>>
>> The documentation page [1] sounds like it's a postgresql specific
>> interpreter, but I found an article [2] saying it's possible to use
>> with MySQL. The source [3] seems to use only JDBC standard API,
>> nothing specific for postgresql (yet), by the way.
>> So, should the interpreter be considered only for postgresql engine in
>> the longer term? Or is it just a matter of naming and so can we use it
>> for any other database? If it's the latter, why not renaming it to
>> something like sql interpreter instead?
>>
>> Thanks in advance,
>>
>> Woonsan
>>
>> [1]
>> https://zeppelin.incubator.apache.org/docs/0.5.5-incubating/interpreter/postgresql.html
>> [2] http://thedataist.com/tutorial-using-apache-zeppelin-with-mysql/
>> [3]
>> https://github.com/apache/incubator-zeppelin/blob/master/postgresql/src/main/java/org/apache/zeppelin/postgresql/PostgreSqlInterpreter.java
>
>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net

Re: R interpreters

[enzo]

+1 For R interpreter.  Soon please!

Incidentally, I wonder if it will display htmlwidgets family of packages as 
well?


Enzo
e...@smartinsightsfromdata.com



> On 18 Jan 2016, at 06:53, Sourav Mazumder  wrote:
> 
> Hi,
> 
> Is there a date/plan to integrate this PR to main branch of Zeppelin ?
> 
> I have found this to be very useful in sharing data/model's prediction result 
> between SparkR and R and then use visualization packages of R to visualize 
> the prediction results. Also demonstrated the same to many others who also 
> liked and found it useful.
> 
> Especially given that in Jupyter the data cannot be shared between R and 
> Spark without serializing/persisting the data, this feature is pretty much 
> differentiatior for Zeppelin. So wondering is there a plan/date for this PR 
> to be available in the main branch ?
> 
> Regards,
> Sourav
> 
> 
> 
> On Thu, Dec 24, 2015 at 11:01 AM, Amos B. Elberg  > wrote:
> Yeah, I’ve looked into it a little bit :p
> 
> rZeppelin (the one in repository Zeppelin-With-R) is an accepted pull request 
> that is pending integration with Zeppelin. 
> 
> In an rZeppelin session, R shares the same Spark backend with the rest of the 
> Zeppelin Spark-based interpreters.  That’s why multiple languages can share 
> the same Spark data pipeline. 
> 
> Datalayer’s R interpreter is not integrated with the rest of Zeppelin.  It 
> launches its own instance of Spark which is separate and distinct from the 
> one being used by the rest of Zeppelin.  
> 
> rZeppelin also supports moving data between languages using the Zeppelin 
> Context, which allows you to share variables with scala and python.  
> Datalayer has their own implementation for moving variables between R and 
> scala. 
> 
> rZeppelin supports interactive visualizations.  I don’t believe that 
> Datalayer’s does. 
> 
> There are other differences as well.  This is explained in the rZeppelin 
> readme. 
> 
> 
> From: Girish Reddy  
> Reply: users@zeppelin.incubator.apache.org 
>  
>  
> 
> Date: December 24, 2015 at 1:52:11 PM
> To: users@zeppelin.incubator.apache.org 
>  
>  
> 
> Subject:  R interpreters 
> 
>> Hello:
>> 
>> Looks like there are two R interpreters - 
>> https://github.com/datalayer/zeppelin-R 
>>  and 
>> https://github.com/elbamos/Zeppelin-With-R 
>> 
>> 
>> From the docs, Zeppelin-With-R supports passing objects between languages 
>> using registerTempTable.  Not sure if the other interpreter supports it or 
>> what other differences are.  Has anyone looked into this?
>> 
>