Re: Are there any way a notebook to be black box?

[Hyung Sung Shim]

Thank you very much to describe in detail.
I totally understand your question now.
There is no way to solve your question for now but it's really interesting
issue.
so Could you make a Jira issue for this with your experience?
Thanks!

2017-07-05 10:16 GMT+09:00 小野圭二 <onoke...@gmail.com>:

> Thx for your reply, Hyung.
>
> Thank you for introducing some docs, but i knew them already.And my point
> was each notebook execute but hide/invisible the code.
> The zeppelin auth system manage every notebooks at once. That feature
> effect in the case of all user share same zeppelin system, like cloud
> computing.
>
> Against it, how about in the case of just sharing notebook?
> For example i set strict roles  into my notebook, and share it with
> someone who has their zeppelin. After they download and import my strict
> auth notebook into their zeppelin, they can arrange my roles setting by
> themselves. In this case, they can see my hided code on their zeppelin.
>
> I do not misunderstand about the zeppelin auth system, do i ?
>
> To prevent this issue, may need another authenticate/management feature on
> Zeppelin, this is my point. :^)
>
> What for it, imagine such a case.
> In the case of selling my big data analyze application through zeppelin
> eco-system, i write and test my code on my zeppelin. Then export it and
> upload to sharing site.
> May someone see my sharing notebook on the site, then they download and
> import it into their zeppelin, and execute it,
>
> Ok that is good, fantastic, viva Zeppelin, if i am a volunteer programmer.
>
> But i would like to get some revenue from my code if someone go down my
> analyzing, hopefully.
> In this case, i just want to provide the result of analyzing, not my
> algorithm.
> Even my client share my notebook on his zeppelin, my code is protected by
> independent authentication, encryption and so on.
>
> This is my point. :-)
> Sorry for getting longer.
>
> -Ono
>
>
> 2017-07-04 23:35 GMT+09:00 Hyung Sung Shim <hss...@nflabs.com>:
>
>> Hello onokeiji.
>>
>> You can refer http://zeppelin.apache.org/docs/0.7.2/quickstart/explo
>> rezeppelinui.html#note-toolbar if you want to hide code in the notebook.
>> and http://zeppelin.apache.org/docs/0.7.2/security/notebook_
>> authorization.html#authorization-setting is relate to lock code your
>> note.
>>
>> Hope this helps.
>>
>>
>> 2017-07-04 22:05 GMT+09:00 小野圭二 <onoke...@gmail.com>:
>>
>>> Hey, how do you think about my idea of BLACKBOX-NOTEBOOK?
>>>
>>> -Ono.
>>>
>>>
>>> <https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>>>  ウイルス
>>> フリー｡ www.avast.com
>>> <https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>>> <#m_-8962903669315512691_m_3961825290808015369_m_-7798815867934407536_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>
>>> 2017-06-30 10:38 GMT+09:00 小野圭二 <onoke...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> I wonder if this mail title would be realize.
>>>> I mean, when I do not want to be changed my notebook programing code,
>>>> but just give a way to change the query of programing by dynamic forms.
>>>>
>>>> 1. are there way to hide all code in the notebook?
>>>> 2. are there rock system to prevent changing the code in notebook?
>>>>
>>>> If No.1 would be available, it makes easy to provide commercial codes
>>>> and/or secret codes to widely.
>>>> In fact, from the view of code writers, especially in the case of
>>>> commercial products, they do not want to provide any source code to their
>>>> customers definitely. Another hand, it is costly to force R or Python
>>>> programer to write Java or Scala program to change their naked codes to
>>>> binary code.
>>>>
>>>> I have a image Script programers write and test their
>>>> R/Python/Spark-Shell/SQL codes on notebooks and be it to
>>>> BLACKBOX-NOTEBOOK, then upload it to github/svn where endusers who want
>>>> to use those notebooks could access by themselves. The endusers down load
>>>> those BLACKBOX-NOTEBOOK and import into their Zeppelin and it works fine.
>>>> Sound like happy the both.
>>>>
>>>> No.2 seems like easier than No.1, because the issue is depend on
>>>> authorization. But it is incomplete, because the codes reveal to the
>>>> endusers.
>>>>
>>>> I saw zepl.com as well.
>>>> Seems like Zeppelin and ZELP are aimed to help collaboration between
>>>> analysts. That is good. And if adding the above NO.1 feature into it, they
>>>> will be the best, i think.
>>>>
>>>> Do not You think so?
>>>>
>>>> -Ono
>>>>
>>>
>>>
>>
>

Re: Are there any way a notebook to be black box?

[Hyung Sung Shim]

Hello onokeiji.

You can refer
http://zeppelin.apache.org/docs/0.7.2/quickstart/explorezeppelinui.html#note-toolbar
if you want to hide code in the notebook.
and
http://zeppelin.apache.org/docs/0.7.2/security/notebook_authorization.html#authorization-setting
is relate to lock code your note.

Hope this helps.


2017-07-04 22:05 GMT+09:00 小野圭二 :

> Hey, how do you think about my idea of BLACKBOX-NOTEBOOK?
>
> -Ono.
>
>
> 
>  ウイルス
> フリー｡ www.avast.com
> 
> <#m_-7798815867934407536_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> 2017-06-30 10:38 GMT+09:00 小野圭二 :
>
>> Hi,
>>
>> I wonder if this mail title would be realize.
>> I mean, when I do not want to be changed my notebook programing code, but
>> just give a way to change the query of programing by dynamic forms.
>>
>> 1. are there way to hide all code in the notebook?
>> 2. are there rock system to prevent changing the code in notebook?
>>
>> If No.1 would be available, it makes easy to provide commercial codes
>> and/or secret codes to widely.
>> In fact, from the view of code writers, especially in the case of
>> commercial products, they do not want to provide any source code to their
>> customers definitely. Another hand, it is costly to force R or Python
>> programer to write Java or Scala program to change their naked codes to
>> binary code.
>>
>> I have a image Script programers write and test their
>> R/Python/Spark-Shell/SQL codes on notebooks and be it to
>> BLACKBOX-NOTEBOOK, then upload it to github/svn where endusers who want
>> to use those notebooks could access by themselves. The endusers down load
>> those BLACKBOX-NOTEBOOK and import into their Zeppelin and it works fine.
>> Sound like happy the both.
>>
>> No.2 seems like easier than No.1, because the issue is depend on
>> authorization. But it is incomplete, because the codes reveal to the
>> endusers.
>>
>> I saw zepl.com as well.
>> Seems like Zeppelin and ZELP are aimed to help collaboration between
>> analysts. That is good. And if adding the above NO.1 feature into it, they
>> will be the best, i think.
>>
>> Do not You think so?
>>
>> -Ono
>>
>
>

Re: python prints "..." in the place of comments in output

[Hyung Sung Shim]

Hi.
The problem is fixing on https://github.com/apache/zeppelin/pull/2106.
Thanks.

2017-03-16 17:58 GMT+09:00 :

> I also met this problem
>
> Send from my iPhone
> Lei Wang
>
> 2017年3月16日 +0800 PM4:39 Meethu Mathew ，写道：
>
> Hi,
>
> The output of following code prints unexpected dots in the result if there
> is a comment in the code. Is it a bug with zeppelin?
>
> *Code :*
>
> %python
> v = [1,2,3]
> #comment 1
> #comment
> print v
>
> *output*
> ... ... [1, 2, 3]
>
> Regards,
> Meethu Mathew
>
>

Re: "spark ui" button in spark interpreter does not show Spark web-ui

[Hyung Sung Shim]

Hello.
Thank you for sharing the problem.
Could you file a jira issue for this?

2017년 3월 13일 (월) 오후 3:18, Meethu Mathew 님이 작성:

> Hi,
>
> I have noticed the same problem
>
> Regards,
>
>
> Meethu Mathew
>
>
> On Mon, Mar 13, 2017 at 9:56 AM, Xiaohui Liu  wrote:
>
> Hi,
>
> We used 0.7.1-snapshot with our Mesos cluster, almost all our needed
> features (ldap login, notebook acl control, livy/pyspark/rspark/scala,
> etc.) work pretty well.
>
> But one thing does not work for us is the 'spark ui' button does not
> response to user clicks. No errors in browser side.
>
> Anyone has met similar issues? Any suggestions about where I should check?
>
> Regards
> Xiaohui
>
>
>

Re: pyspark run a specific paragraph

[Hyung Sung Shim]

hi Jonathan.

Currently pyspark interpreter does't have z.run() functionality but spark
interpreter does.


2017년 3월 1일 (수) 오전 1:35, Jonathan Greenleaf 님이
작성:

Haven't had luck with documentation.  What is the syntax to run a specific
paragraph using the pyspark interpreter?  I'm using Version 0.8.0-SNAPSHOT.

%pyspark
z.run('20170227-213250_876580864')

..
AttributeError: 'PyZeppelinContext' object has no attribute 'run'

Thank you
Jonathan Greenleaf

Re: Zeppelin - Executors Issue

[Hyung Sung Shim]

hi.
Can you try to set  *spark.executor.instances* property on your spark
interpreter ?



2017-02-28 21:26 GMT+09:00 Gilad Landau :

> Hello
>
>
>
> Anybody else encountered executors issues for the Zeppelin job on spark?
>
> Although the interpreter is set with spark executors num set to 6, I only
> actually get 2 executors from Yarn. I see that the spark-submit is sent
> with the right amount of executors. This behavior doesn’t occur with other
> spark jobs, or with pyspark/sparkshell.
>
>
>
> Any idea?
>
>
>
> Gilad
>
>
>
>

Re: How can I use pyspark in zeppelin?

[Hyung Sung Shim]

hello.
Could you set spark-2.1.0-bin-hadoop2.7 path as SPARK_HOME ?
You can refer to
http://zeppelin.apache.org/docs/0.7.0/interpreter/spark.html#1-export-spark_home
.

2017-02-24 15:44 GMT+09:00 Beth Lee :

> I installed spark-2.1.0-bin-hadoop2.7.tgz and zeppelin-0.7.0-bin-all.tgz in
> ubuntu.
>
> I set the zeppelin-env.sh like below.
>
> export PYTHONPATH=/usr/bin/python
> export PYSPARK_PYTHON=/home/jin/spark/python
>
> So I try to use pyspark in the zeppelin notebook.
>
> %spark.pyspark
>
> print(2+2)
>
> There are errors occurred in zeppelin notebook.
>
> java.lang.NullPointerException
> at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38)
> at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33)
> at 
> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_2(SparkInterpreter.java:380)
> at 
> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:369)
> at 
> org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:144)
> at 
> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:817)
> at 
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
> at 
> org.apache.zeppelin.spark.PySparkInterpreter.getSparkInterpreter(PySparkInterpreter.java:546)
> at 
> org.apache.zeppelin.spark.PySparkInterpreter.createGatewayServerAndStartScript(PySparkInterpreter.java:206)
> at 
> org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:160)
> at 
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
> at 
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:482)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
> at 
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> I don't know why these errors are occurred.
>
> Would you give me some advice?
>

Re: [ANNOUNCE] Apache Zeppelin 0.7.0 released

[Hyung Sung Shim]

Congratulations!
and Thank you all Zeppelin users and devs!!
2017년 2월 6일 (월) 오후 8:48, Md. Rezaul Karim 님이 작성:

> Congratulations! Thanks for the great news. I appreciate your hard work
> for every release.
>
> On Feb 6, 2017 7:07 AM, Mina Lee  wrote:
>
> The Apache Zeppelin community is pleased to announce the availability of
> the 0.7.0 release.
>
> Zeppelin is a collaborative data analytics and visualization tool for
> distributed, general-purpose data processing system such as Apache Spark,
> Apache Flink, etc.
>
> The community put significant effort into improving Apache Zeppelin since
> the last release, focusing on multi-user support, pluggable visualization,
> better interpreter support. More than 100+ contributors provided 700+
> patches
> for new features, improvements and bug fixes. More than 480+ issues have
> been resolved.
>
> We encourage you to download the latest release from
> http://zeppelin.apache.org/download.html
>
> Release note is available at
> http://zeppelin.apache.org/releases/zeppelin-release-0.7.0.html
>
> We welcome your help and feedback. For more information on the project and
> how to get involved, visit our website at http://zeppelin.apache.org/
>
> Thank you all users and contributors who have helped to improve Apache
> Zeppelin.
>
> Regards,
> The Apache Zeppelin community
>
>
>

Re: [VOTE] Release Apache Zeppelin 0.7.0 (RC4)

[Hyung Sung Shim]

+1
Thanks mina for your effort.

2017-02-03 11:58 GMT+09:00 Renjith Kamath :

> +1
>
> On Fri, Feb 3, 2017 at 8:01 AM, Prabhjyot Singh  >
> wrote:
>
> > +1
> >
> > On Feb 2, 2017 8:25 PM, "Alexander Bezzubov"  wrote:
> >
> >> +1,
> >>
> >> and thank you for an awesome work Mina!
> >> Your persistence in making RCs and incorporating feedback is admirable.
> >>
> >> Verified:
> >>  - checksums, signatures + keys for sources and bin-all
> >>  - bin-all can run all Spark Zeppelin Tutorial in local mode
> >>  - sources do compile, but only without tests.
> >>Build \w tests fails on zeppelin-zengine for me
> >>
> >> 1) Failed tests:
> >>   NotebookTest.testSchedulePoolUsage:397 expected: but
> >> was:
> >>
> >> 2) frontend build on Linux also failed mysteriously executing yarn
> >> command, most probably due to local env configuration.
> >>
> >>
> >> --
> >> Alex
> >>
> >> On Thu, Feb 2, 2017 at 5:28 PM, Jeff Zhang  wrote:
> >>
> >>> +1
> >>>
> >>> Ahyoung Ryu 于2017年2月2日周四 下午9:36写道：
> >>>
> >>> +1
> >>>
> >>> On Thu, Feb 2, 2017 at 10:07 PM, Jun Kim  wrote:
> >>>
> >>> +1
> >>> 2017년 2월 2일 (목) 오후 9:49, Sora Lee 님이 작성:
> >>>
> >>> +1
> >>>
> >>> On Thu, Feb 2, 2017 at 9:40 PM Khalid Huseynov 
> >>> wrote:
> >>>
> >>> > +1
> >>> >
> >>> > On Thu, Feb 2, 2017 at 9:21 PM, DuyHai Doan 
> >>> wrote:
> >>> >
> >>> > +1
> >>> >
> >>> > On Thu, Feb 2, 2017 at 9:56 AM, Mina Lee  wrote:
> >>> >
> >>> > > I propose the following RC to be released for the Apache Zeppelin
> >>> 0.7.0
> >>> > > release.
> >>> > >
> >>> > > The commit id is df007f2284a09caa7c8b35f8b59d5f1993fe8b64 which is
> >>> > > corresponds to the tag v0.7.0-rc4:
> >>> > > *
> >>> > https://git-wip-us.apache.org/repos/asf?p=zeppelin.git;a=sho
> >>> rtlog;h=refs/tags/v0.7.0-rc4
> >>> > > <
> >>> > https://git-wip-us.apache.org/repos/asf?p=zeppelin.git;a=sho
> >>> rtlog;h=refs/tags/v0.7.0-rc4
> >>>
> >>> > >*
> >>> > >
> >>> > > The release archives (tgz), signature, and checksums are here
> >>> > > https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.
> 7.0-rc4/
> >>> > >
> >>> > > The release candidate consists of the following source distribution
> >>> > > archive
> >>> > > zeppelin-0.7.0.tgz
> >>> > >
> >>> > > In addition, the following supplementary binary distributions are
> >>> > provided
> >>> > > for user convenience at the same location
> >>> > > zeppelin-0.7.0-bin-all.tgz
> >>> > > zeppelin-0.7.0-bin-netinst.tgz
> >>> > >
> >>> > > The maven artifacts are here
> >>> > >
> >>> > https://repository.apache.org/content/repositories/orgapache
> >>> zeppelin-1027
> >>> > >
> >>> > > You can find the KEYS file here:
> >>> > > https://dist.apache.org/repos/dist/release/zeppelin/KEYS
> >>> > >
> >>> > > Release notes available at
> >>> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> >>> > > version=12336544=12316221
> >>> > >
> >>> > > Vote will be open for next 72 hours (close at 01:00 5/Feb PST).
> >>> > >
> >>> > > [ ] +1 approve
> >>> > > [ ] 0 no opinion
> >>> > > [ ] -1 disapprove (and reason why)
> >>> > >
> >>> >
> >>> >
> >>> >
> >>>
> >>> --
> >>> Taejun Kim
> >>>
> >>> Data Mining Lab.
> >>> School of Electrical and Computer Engineering
> >>> University of Seoul
> >>>
> >>>
> >>>
> >>
>

Re: [VOTE] Release Apache Zeppelin 0.7.0 (RC3)

[Hyung Sung Shim]

+1
2017년 1월 24일 (화) 오전 1:36, Windy Qin 님이 작성:

>
> +1
> On 2017-01-24 17:13 (+0800), Sora Lee  wrote:
> > +1
> >
> > On Tue, Jan 24, 2017 at 3:09 PM Jongyoul Lee  wrote:
> >
> > > +1 (binding)
> > >
> > > On Tue, Jan 24, 2017 at 2:43 PM, Mina Lee  wrote:
> > >
> > > I propose the following RC to be released for the Apache Zeppelin 0.7.0
> > > release.
> > >
> > > The commit id is 48ad70e8c62975bdb00779bed5919eaca98c5b5d which is
> > > corresponds to the tag v0.7.0-rc3:
> > > *
> https://git-wip-us.apache.org/repos/asf?p=zeppelin.git;a=commit;h=48ad70e8c62975bdb00779bed5919eaca98c5b5d
> > > <
> https://git-wip-us.apache.org/repos/asf?p=zeppelin.git;a=commit;h=48ad70e8c62975bdb00779bed5919eaca98c5b5d
> >*
> > >
> > > The release archives (tgz), signature, and checksums are here
> > > https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.7.0-rc3/
> > >
> > > The release candidate consists of the following source distribution
> > > archive
> > > zeppelin-0.7.0.tgz
> > >
> > > In addition, the following supplementary binary distributions are
> provided
> > > for user convenience at the same location
> > > zeppelin-0.7.0-bin-all.tgz
> > > zeppelin-0.7.0-bin-netinst.tgz
> > >
> > > The maven artifacts are here
> > >
> https://repository.apache.org/content/repositories/orgapachezeppelin-1024
> > >
> > > You can find the KEYS file here:
> > > https://dist.apache.org/repos/dist/release/zeppelin/KEYS
> > >
> > > Release notes available at
> > >
> > >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12336544=12316221
> > >
> > > Vote will be open for next 72 hours (close at 22:00 26/Jan PST).
> > >
> > > [ ] +1 approve
> > > [ ] 0 no opinion
> > > [ ] -1 disapprove (and reason why)
> > >
> > >
> > >
> > >
> > > --
> > > 이종열, Jongyoul Lee, 李宗烈
> > > http://madeng.net
> > >
> >
>

Re: [VOTE] Release Apache Zeppelin 0.7.0 (RC2)

[Hyung Sung Shim]

+1
2017년 1월 22일 (일) 오전 12:54, Ahyoung Ryu 님이 작성:

> +1
>
> On Sun, Jan 22, 2017 at 5:52 PM, Jeff Zhang  wrote:
>
> > +1
> >
> >
> >
> > DuyHai Doan 于2017年1月22日周日 下午4:01写道：
> >
> >> +1
> >>
> >> On Sun, Jan 22, 2017 at 8:59 AM, Mina Lee  wrote:
> >>
> >> I propose the following RC to be released for the Apache Zeppelin 0.7.0
> >> release.
> >>
> >> The commit id is d0ddf8fdff75b4229543f7ebfd5eb001f685382d which is
> >> corresponds to the tag v0.7.0-rc2:
> >> *
> https://git-wip-us.apache.org/repos/asf?p=zeppelin.git;a=commit;h=d0ddf8fdff75b4229543f7ebfd5eb001f685382d
> >> <
> https://git-wip-us.apache.org/repos/asf?p=zeppelin.git;a=commit;h=d0ddf8fdff75b4229543f7ebfd5eb001f685382d
> >*
> >>
> >> The release archives (tgz), signature, and checksums are here
> >> https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.7.0-rc2/
> >>
> >> The release candidate consists of the following source distribution
> >> archive
> >> zeppelin-0.7.0.tgz
> >>
> >> In addition, the following supplementary binary distributions are
> provided
> >> for user convenience at the same location
> >> zeppelin-0.7.0-bin-all.tgz
> >> zeppelin-0.7.0-bin-netinst.tgz
> >>
> >> The maven artifacts are here
> >>
> https://repository.apache.org/content/repositories/orgapachezeppelin-1023
> >>
> >> You can find the KEYS file here:
> >> https://dist.apache.org/repos/dist/release/zeppelin/KEYS
> >>
> >> Release notes available at
> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> >> version=12336544=12316221
> >>
> >> Vote will be open for next 72 hours (close at 23:59 24/Jan PST).
> >>
> >> [ ] +1 approve
> >> [ ] 0 no opinion
> >> [ ] -1 disapprove (and reason why)
> >>
> >>
> >>
>

Re: [VOTE] Release Apache Zeppelin 0.7.0 (RC1)

[Hyung Sung Shim]

+1
2017년 1월 21일 (토) 오전 4:55, Jeff Zhang 님이 작성:

> -1,  Can we include ZEPPELIN-1985
>  ? This issue cause
> only user1 can see the pig tutorial note which is a very bad user
> experience.
>
>
>
> Prabhjyot Singh 于2017年1月21日周六 下午7:25写道：
>
> +1
>
> On Jan 21, 2017 4:52 PM, "Jun Kim"  wrote:
>
> +1
> 2017년 1월 21일 (토) 오후 8:21, CloverHearts 님이 작성:
>
> +1
>
>
>
> The Zeppelin 0.7 release includes some amazing features.
>
> In particular, a major bug fix for the remote interpreter.
>
> I am actively in favor of this release.
>
>
>
>
>
> *출발**: *Mina Lee 
> *회신* *대상**:* 
> *날짜**: *2017년 1월 21일 토요일 오전 3:17
> *받는* *사람**: *, 
> *주제**: *[VOTE] Release Apache Zeppelin 0.7.0 (RC1)
>
>
>
> Hi folks,
>
> I propose the following RC to be released for the Apache Zeppelin 0.7.0
> release.
>
> The commit id is bcbe0855175f55eddd10b918303806eb5e5db091 which is
> corresponds to the tag v0.7.0-rc1:
>
>
> https://git-wip-us.apache.org/repos/asf?p=zeppelin.git;a=commit;h=bcbe0855175f55eddd10b918303806eb5e5db091
>
> The release archives (tgz), signature, and checksums are here
> https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.7.0-rc1/
>
> The release candidate consists of the following source distribution
> archive
>
> zeppelin-0.7.0.tgz
>
> In addition, the following supplementary binary distributions are provided
> for user convenience at the same location
>
> zeppelin-0.7.0-bin-all.tgz
> zeppelin-0.7.0-bin-netinst.tgz
>
>
> The maven artifacts are here
> https://repository.apache.org/content/repositories/orgapachezeppelin-1022
>
> You can find the KEYS file here:
> https://dist.apache.org/repos/dist/release/zeppelin/KEYS
>
> Release notes available at
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12336544=12316221
>
> Vote will be open for next 72 hours (close at 3:30 am 24/Jan PST).
>
> [ ] +1 approve
> [ ] 0 no opinion
> [ ] -1 disapprove (and reason why)
>
> --
> Taejun Kim
>
> Data Mining Lab.
> School of Electrical and Computer Engineering
> University of Seoul
>
>

Re: Re: Re: contextual relevance in the previous paragraph and the latter paragraph in Zeppelin

[Hyung Sung Shim]

Hi.

Actually I tested with jdbc(mysql) but it would be the same with hive right?
and the latest version what i mentioned is the master in the github repo.

BTW could you share your interpreter setting for hive?


2016-12-28 22:37 GMT-08:00 Pengcheng Fang <nuspengch...@163.com>:

> Hi Shim,
>
> thanks for your reply. I am using the HDP2.5 built-in Zeppelin , but I
> also tested with the latest version Zeppelin (zeppelin-0.6.2-bin-all.tgz).
>
> However, both of them does not work at my side .
>
> Could I ask if you are indeed using the "%jdbc(hive)" interpreter when you
> performed the test , since %sql interpreter is working during my testing .
> Thanks again.
>
> At 2016-12-29 13:41:46, "Hyung Sung Shim" <hss...@nflabs.com> wrote:
>
> Hi Fang.
> What zeppelin version are you using?
> I just tested the latest version of master and It works expected.
>
> 2016-12-28 0:15 GMT-08:00 Pengcheng Fang <nuspengch...@163.com>:
>
>> Hi Mohit ,
>>
>> thank you for your reply, I just ran those paragraphs one by one from top
>> to bottom.
>>
>> thanks
>>
>> At 2016-12-28 13:03:24, "Mohit Jaggi" <mohitja...@gmail.com> wrote:
>>
>> it should depend on the order in which you run the paragraphs...did you
>> run in the same order that you listed them above?
>>
>> On Tue, Dec 27, 2016 at 5:20 PM, Pengcheng Fang <nuspengch...@163.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I have one more question regarding the contextual relevance in
>>> the previous paragraph and the latter paragraph in Zeppelin, for example :
>>>
>>> --paragraph1--
>>> %jdbc(hive)
>>> use database_one
>>> ---
>>>
>>> --paragraph2--
>>> %jdbc(hive)
>>> show tables
>>> ---
>>>
>>> --paragraph3--
>>> %jdbc(hive)
>>> use database_two
>>> ---
>>>
>>> --paragraph4--
>>> %jdbc(hive)
>>> show tables
>>> ---
>>>
>>> If the tables in databases_one and databases_two are totally different,
>>> will the output in "paragraph2" and "paragraph4" be different ?
>>>
>>> From my understanding , the answer should be "YES" , but from test
>>> result at my side , the output in "paragraph2" and "paragraph4" are same
>>> ...
>>>
>>> Does anyone know the reason , or did I go worng ?
>>>
>>> Thanks and Best regards
>>>
>>> Pengcheng
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>
>
>
>
>

Re: Re: contextual relevance in the previous paragraph and the latter paragraph in Zeppelin

[Hyung Sung Shim]

Hi Fang.
What zeppelin version are you using?
I just tested the latest version of master and It works expected.

2016-12-28 0:15 GMT-08:00 Pengcheng Fang :

> Hi Mohit ,
>
> thank you for your reply, I just ran those paragraphs one by one from top
> to bottom.
>
> thanks
>
> At 2016-12-28 13:03:24, "Mohit Jaggi"  wrote:
>
> it should depend on the order in which you run the paragraphs...did you
> run in the same order that you listed them above?
>
> On Tue, Dec 27, 2016 at 5:20 PM, Pengcheng Fang 
> wrote:
>
>> Hi All,
>>
>> I have one more question regarding the contextual relevance in
>> the previous paragraph and the latter paragraph in Zeppelin, for example :
>>
>> --paragraph1--
>> %jdbc(hive)
>> use database_one
>> ---
>>
>> --paragraph2--
>> %jdbc(hive)
>> show tables
>> ---
>>
>> --paragraph3--
>> %jdbc(hive)
>> use database_two
>> ---
>>
>> --paragraph4--
>> %jdbc(hive)
>> show tables
>> ---
>>
>> If the tables in databases_one and databases_two are totally different,
>> will the output in "paragraph2" and "paragraph4" be different ?
>>
>> From my understanding , the answer should be "YES" , but from test result
>> at my side , the output in "paragraph2" and "paragraph4" are same ...
>>
>> Does anyone know the reason , or did I go worng ?
>>
>> Thanks and Best regards
>>
>> Pengcheng
>>
>>
>>
>>
>
>
>
>
>

Re: Cannot use Hive Interpreter in Zeppelin 0.6.0 - Hive class java.lang.ClassNotFoundException

[Hyung Sung Shim]

Please make sure that your hive server is running on localhost:1 .

2016년 12월 19일 (월) 오후 7:56, Nabajyoti Dash 님이 작성:

> Thanks.It was a typo..(my nightmare).
> But after fixing this I am getting the ConnectionRefused exception.I am
> using Hive on mysql and has given the mysql username /password in zeppelin
> jdbc interpreter. Am I wrong?
>
> java.sql.SQLException: Could not open client transport with JDBC Uri:
> jdbc:hive2://localhost:1: java.net.ConnectException: Connection refused
> (Connection refused)
> at
> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:231)
> at
> org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:176)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> at
>
> org.apache.commons.dbcp2.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:79)
> at
>
> org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:205)
> at
>
> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
> at
>
> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
> at
>
> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
> at
> org.apache.commons.dbcp2.PoolingDriver.connect(PoolingDriver.java:129)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:270)
> at
>
> org.apache.zeppelin.jdbc.JDBCInterpreter.getConnectionFromPool(JDBCInterpreter.java:322)
> at
>
> org.apache.zeppelin.jdbc.JDBCInterpreter.getConnection(JDBCInterpreter.java:340)
> at
>
> org.apache.zeppelin.jdbc.JDBCInterpreter.executeSql(JDBCInterpreter.java:401)
> at
>
> org.apache.zeppelin.jdbc.JDBCInterpreter.interpret(JDBCInterpreter.java:526)
> at
>
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
> at
>
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:489)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
> at
>
> org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.run(ParallelScheduler.java:162)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.thrift.transport.TTransportException:
> java.net.ConnectException: Connection refused (Connection refused)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
> at
>
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> at
> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
> ... 27 more
> Caused by: java.net.ConnectException: Connection refused (Connection
> refused)
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net
> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at
> java.net
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
> ... 30 more
>
>
>
>
>
> --
> View this message in context:
> http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Cannot-use-Hive-Interpreter-in-Zeppelin-0-6-0-Hive-class-java-lang-ClassNotFoundException-tp3635p4765.html
> Sent from the Apache Zeppelin Users (incubating) mailing list mailing list
> archive at Nabble.com.
>

Re: Cannot use Hive Interpreter in Zeppelin 0.6.0 - Hive class java.lang.ClassNotFoundException

[Hyung Sung Shim]

Hi.
It seems there is typo on default.driver.
default.driver value should be org.apache.hive.jdbc.HiveDriver not *c*
org.apache.hive.jdbc.HiveDriver.

2016-12-19 18:24 GMT+09:00 Nabajyoti Dash :

> I am facing the same issue even after adding both the jars to the
> interpreter.
> please suggest anything else i could be me missing.
>  x6.nabble.com/file/n4758/Screen_Shot_2016-12-19_at_2.png>
>
>
>
> --
> View this message in context: http://apache-zeppelin-users-
> incubating-mailing-list.75479.x6.nabble.com/Cannot-use-Hive-
> Interpreter-in-Zeppelin-0-6-0-Hive-class-java-lang-ClassNotFoundException-
> tp3635p4758.html
> Sent from the Apache Zeppelin Users (incubating) mailing list mailing list
> archive at Nabble.com.
>

Re: R interpreter build fails

[Hyung Sung Shim]

Hello.
In order to install R Interpreter, you need to install some packages.

[1]
http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/r.html

2016-11-28 23:51 GMT+09:00 Ruslan Dautkhanov :

> The same problem in 0.6.2 and 0.7.0-snapshot.
> R interpreter build fails with below error stack.
>
> R is installed locally through yum.
> Is there any special requirements for R interpreter build?
>
>
> [INFO] Zeppelin: Packaging distribution ... SUCCESS [
>  6.250 s]
> [INFO] Zeppelin: R Interpreter  FAILURE [
> 24.092 s]
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 10:54 min
> [INFO] Finished at: 2016-11-28T07:44:51-07:00
> [INFO] Final Memory: 221M/3106M
> [INFO] 
> 
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec
> (default) on project zeppelin-zrinterpreter_2.10: Command execution failed.
> Process exited with an error: 1 (Exit value: 1) -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (default) on project
> zeppelin-zrinterpreter_2.10: Command execution failed.
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:212)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:153)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:145)
> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.
> buildProject(LifecycleModuleBuilder.java:116)
> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.
> buildProject(LifecycleModuleBuilder.java:80)
> at org.apache.maven.lifecycle.internal.builder.singlethreaded.
> SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at org.apache.maven.lifecycle.internal.LifecycleStarter.
> execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.codehaus.plexus.classworlds.launcher.Launcher.
> launchEnhanced(Launcher.java:289)
> at org.codehaus.plexus.classworlds.launcher.Launcher.
> launch(Launcher.java:229)
> at org.codehaus.plexus.classworlds.launcher.Launcher.
> mainWithExitCode(Launcher.java:415)
> at org.codehaus.plexus.classworlds.launcher.Launcher.
> main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Command
> execution failed.
> at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:362)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(
> DefaultBuildPluginManager.java:134)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:207)
> ... 20 more
> Caused by: org.apache.commons.exec.ExecuteException: Process exited with
> an error: 1 (Exit value: 1)
> at org.apache.commons.exec.DefaultExecutor.executeInternal(
> DefaultExecutor.java:377)
> at org.apache.commons.exec.DefaultExecutor.execute(
> DefaultExecutor.java:160)
> at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(
> ExecMojo.java:610)
> at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:352)
> ... 22 more
> [ERROR]
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/
> MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR]   mvn  -rf :zeppelin-zrinterpreter_2.10
>
>
> --
> Ruslan Dautkhanov
>

Re: how to build Zeppelin with a defined list of interpreters

[Hyung Sung Shim]

Sorry for confusing you Ruslan.


2016-11-28 23:20 GMT+09:00 Ruslan Dautkhanov <dautkha...@gmail.com>:

> Thank you moon!
>
>
> On Mon, Nov 28, 2016 at 7:04 AM moon soo Lee <m...@apache.org> wrote:
>
>> Hi,
>>
>> You can use '-pl' for your maven build command to exclude submodules. For
>> example,
>>
>>  mvn -DskipTests -pl '!scio,!alluxio'
>>
>> Thanks,
>> moon
>>
>>
>> On Mon, Nov 28, 2016 at 5:58 AM Ruslan Dautkhanov <dautkha...@gmail.com>
>> wrote:
>>
>> Thank you Hyung.
>>
>> I was asking primarily how to switch off compiling certain interpreters.
>> Is there is a way?
>>
>> Our organization will not use more than half of those interpreters.
>>
>> On Mon, Nov 28, 2016 at 4:42 AM Hyung Sung Shim <hss...@nflabs.com>
>> wrote:
>>
>> Hello.
>> https://github.com/apache/zeppelin/pull/1685 might related in your case.
>> 2016년 11월 28일 (월) 오후 4:12, Ruslan Dautkhanov <dautkha...@gmail.com>님이 작성:
>>
>> Getting [1] error stack when trying to build Zeppelin from 0.7-snapshot.
>> We will not use most of the built-in Zeppelin interpreters, including
>> Scio which is failing.
>> How we can switch off (black-list) certain interpreters from Zeppelin
>> build at all?
>>
>> http://zeppelin.apache.org/docs/snapshot/install/build.html
>> does not answer this question.
>>
>>
>>
>> [1]
>>
>> [INFO] Zeppelin: Flink  SUCCESS [
>> 16.732 s]
>> [INFO] Zeppelin: Apache Ignite interpreter  SUCCESS [
>>  1.645 s]
>> [INFO] Zeppelin: Kylin interpreter  SUCCESS [
>>  0.854 s]
>> [INFO] Zeppelin: Python interpreter ... SUCCESS [
>>  0.872 s]
>> [INFO] Zeppelin: Lens interpreter . SUCCESS [
>> 10.669 s]
>> [INFO] Zeppelin: Apache Cassandra interpreter . SUCCESS [
>> 58.011 s]
>> [INFO] Zeppelin: Elasticsearch interpreter  SUCCESS [
>>  6.940 s]
>> [INFO] Zeppelin: BigQuery interpreter . SUCCESS [
>>  2.345 s]
>> [INFO] Zeppelin: Alluxio interpreter .. SUCCESS [
>>  8.735 s]
>> [INFO] Zeppelin: Scio . FAILURE [
>> 31.941 s]
>> [INFO] Zeppelin: web Application .. SKIPPED
>> [INFO] Zeppelin: Server ... SKIPPED
>> [INFO] Zeppelin: Packaging distribution ... SKIPPED
>> [INFO] Zeppelin: R Interpreter  SKIPPED
>> [INFO] 
>> 
>> [INFO] BUILD FAILURE
>> [INFO] 
>> 
>> [INFO] Total time: 06:26 min
>> [INFO] Finished at: 2016-11-28T00:08:10-07:00
>> [INFO] Final Memory: 203M/6540M
>> [INFO] 
>> 
>> ---
>>
>>

Re: 0.6.2 build fails

[Hyung Sung Shim]

Hello.
Thank you for sharing your problem.

Could you add *-Pvendor-repo *option to build Zeppelin with CDH?
You can refer to
http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/install/build.html#build-command-examples
.
Thanks.

2016-11-23 16:05 GMT+09:00 Ruslan Dautkhanov :

> Following build fails with
>
> mvn clean package -DskipTests -Pspark-1.6 -Ppyspark
> -Dhadoop.version=2.6.0-cdh5.8.3 -Phadoop-2.6 -Pyarn -Pr
>
> [*ERROR*] Failed to execute goal on project zeppelin-spark-dependencies_2.10:
> Could not resolve dependencies for project org.apache.zeppelin:zeppelin-
> spark-dependencies_2.10:jar:0.6.2: The following artifacts could not be
> resolved: org.apache.hadoop:hadoop-client:jar:2.6.0-cdh5.8.3,
> org.apache.hadoop:hadoop-yarn-api:jar:2.6.0-cdh5.8.3: Could not find
> artifact org.apache.hadoop:hadoop-client:jar:2.6.0-cdh5.8.3 in central (
> https://repo.maven.apache.org/maven2) -> [Help 1]
>
> Maven 3.3.9
> Zeppelin 0.6.2
> Java 7
>
> I was trying to follow
> http://blog.cloudera.com/blog/2015/07/how-to-install-apache-
> zeppelin-on-cdh/
> which is a bit outdated, but still very helpful.
>
> Any ideas?
>
>
>
> [1]
> Explicit exports before build:
>
> export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
> export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive
> export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
> export HADOOP_CONF_DIR=/etc/hadoop/conf
> export HIVE_CONF_DIR=/etc/hive/conf
>
> [2]
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Zeppelin ... SUCCESS [
>  5.504 s]
> [INFO] Zeppelin: Interpreter .. SUCCESS [
> 13.012 s]
> [INFO] Zeppelin: Zengine .. SUCCESS [
>  7.975 s]
> [INFO] Zeppelin: Display system apis .. SUCCESS [
> 16.979 s]
> [INFO] Zeppelin: Spark dependencies ... FAILURE [
> 14.471 s]
> [INFO] Zeppelin: Spark  SKIPPED
> [INFO] Zeppelin: Markdown interpreter . SKIPPED
> [INFO] Zeppelin: Angular interpreter .. SKIPPED
>
>
>
>
>

Re: ArrayIndexOutOfBoundsException on Zeppelin notebook example

[Hyung Sung Shim]

Good to hear it helps.
2016년 11월 18일 (금) 오전 1:52, Muhammad Rezaul Karim <reza_cse...@yahoo.com>님이
작성:

> Hi Shim,
>
> Now it works perfectly. Thank you so much. Actually, I am from Java
> background and learning the Scala.
>
>
> Thanks and Regards,
> -
> *Md. Rezaul Karim*
> PhD Researcher, Insight Centre for Data Analytics
> National University of Ireland Galway
> *E-mail:* rezaul.ka...@insight-centre.org
> <donal.carr...@insight-centre.org>
> *Web*: www.insight-centre.org
> *Phone:* +353892311519
>
>
> On Thursday, November 17, 2016 2:00 PM, Hyung Sung Shim <hss...@nflabs.com>
> wrote:
>
>
> Hello Muhammad.
>
> Please check your bank-full.csv file first and you can filter item length
> in your scala code
> for example *val bank = bankText.map(s => s.split(";")).filter(s =>
> (s.size)>5).filter(s => s(0) != "\"age\"")*
>
> Hope this helps.
>
>
>
> 2016-11-17 21:26 GMT+09:00 Dayong <will...@gmail.com>:
>
> Try to debug your code in IDE. You should look at your array s since it
> complains about array index.
>
> Thanks,
> Wd
>
>
> On Nov 16, 2016, at 10:44 PM, Muhammad Rezaul Karim <reza_cse...@yahoo.com>
> wrote:
>
> Hi All,
>
> I have the following Scala code (taken from https://zeppelin.apache.org/
> docs/0.6.2/quickstart/ tutorial.html#data-retrieval
> <https://zeppelin.apache.org/docs/0.6.2/quickstart/tutorial.html#data-retrieval>)
> that deals with the sample Bank-details data:
>
>
> -- -*
> -- -- 
>
> val bankText = sc.textFile("/home/asif/ zeppelin-0.6.2-bin-all/bin/
> bank-full.csv")
>
>
> case class Bank(age:Integer, job:String, marital:String, education:String,
> balance:Integer)
>
> // split each line, filter out header (starts with "age"), and map it into
> Bank case class
> val bank = bankText.map(s=>s.split(";")). filter(s=>s(0)!="\"age\""). map(
> s=>Bank(s(0),
> s(1).replaceAll("\"", ""),
> s(2).replaceAll("\"", ""),
> s(3).replaceAll("\"", ""),
> s(5).replaceAll("\"", "")
> )
> )
> // convert to DataFrame and create temporal table
> bank.toDF().registerTempTable( "bank")
>
> -- -*
> -- -- 
> The above code segment runs successfully. However, when I am trying to
> execute the following line of code: *bank.collect(), *I am getting the
> following error:
>
> org.apache.spark. SparkException: Job aborted due to stage failure: Task 1
> in stage 6.0 failed 1 times, most recent failure: Lost task 1.0 in stage
> 6.0 (TID 7, localhost): java.lang. ArrayIndexOutOfBoundsException : 2
>
>
> at $anonfun$3.apply(:91)
> at $anonfun$3.apply(:89)
>
>
> Moreover, I cannot execute the below SQL queries, but getting the same
> error message (i.e., ArrayIndexOutOfBoundsException : 2 ):
>
> 1.%sql select age, count(1) from bank where age < 30 group by age
> order by age
> 2. %sql select age, count(1) from bank where age < ${maxAge=30} group by
> age order by age
>
> 3. %sql select age, count(1) from bank where marital="${marital=single,
> single|divorced|married}" group by age order by age
>
>
> *Note: However, when I am executing the following SQL statements I am not
> getting any error:*
>
> 1. %sql select age from bank
> 2. %sql select * from bank
>
> I don't understand what's wrong I am doing here! Please help me, someone,
> to get rid of it.
>
>
> Thanks and Regards,
> -- ---
> *Md. Rezaul Karim*
> PhD Researcher, Insight Centre for Data Analytics
> National University of Ireland Galway
> *E-mail:* rezaul.karim@insight-centre. org
> <donal.carr...@insight-centre.org>
> *Web*: www.insight-centre.org
> *Phone:* +353892311519
>
>

Re: ArrayIndexOutOfBoundsException on Zeppelin notebook example

[Hyung Sung Shim]

Hello Muhammad.

Please check your bank-full.csv file first and you can filter item length
in your scala code
for example *val bank = bankText.map(s => s.split(";")).filter(s =>
(s.size)>5).filter(s => s(0) != "\"age\"")*

Hope this helps.




2016-11-17 21:26 GMT+09:00 Dayong :

> Try to debug your code in IDE. You should look at your array s since it
> complains about array index.
>
> Thanks,
> Wd
>
> On Nov 16, 2016, at 10:44 PM, Muhammad Rezaul Karim 
> wrote:
>
> Hi All,
>
> I have the following Scala code (taken from https://zeppelin.apache.org/
> docs/0.6.2/quickstart/tutorial.html#data-retrieval) that deals with the
> sample Bank-details data:
>
> ---*
> 
> val bankText = sc.textFile("/home/asif/zeppelin-0.6.2-bin-all/bin/
> bank-full.csv")
> case class Bank(age:Integer, job:String, marital:String, education:String,
> balance:Integer)
>
> // split each line, filter out header (starts with "age"), and map it into
> Bank case class
> val bank = bankText.map(s=>s.split(";")).filter(s=>s(0)!="\"age\"").map(
> s=>Bank(s(0),
> s(1).replaceAll("\"", ""),
> s(2).replaceAll("\"", ""),
> s(3).replaceAll("\"", ""),
> s(5).replaceAll("\"", "")
> )
> )
> // convert to DataFrame and create temporal table
> bank.toDF().registerTempTable("bank")
> ---*
> 
> The above code segment runs successfully. However, when I am trying to
> execute the following line of code: *bank.collect(), *I am getting the
> following error:
>
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 1
> in stage 6.0 failed 1 times, most recent failure: Lost task 1.0 in stage
> 6.0 (TID 7, localhost): java.lang.ArrayIndexOutOfBoundsException: 2
> at $anonfun$3.apply(:91)
> at $anonfun$3.apply(:89)
>
> Moreover, I cannot execute the below SQL queries, but getting the same
> error message (i.e., ArrayIndexOutOfBoundsException: 2 ):
>
> 1.%sql select age, count(1) from bank where age < 30 group by age
> order by age
> 2. %sql select age, count(1) from bank where age < ${maxAge=30} group by
> age order by age
> 3. %sql select age, count(1) from bank where marital="${marital=single,
> single|divorced|married}" group by age order by age
>
> *Note: However, when I am executing the following SQL statements I am not
> getting any error:*
>
> 1. %sql select age from bank
> 2. %sql select * from bank
>
> I don't understand what's wrong I am doing here! Please help me, someone,
> to get rid of it.
>
>
> Thanks and Regards,
> -
> *Md. Rezaul Karim*
> PhD Researcher, Insight Centre for Data Analytics
> National University of Ireland Galway
> *E-mail:* rezaul.ka...@insight-centre.org
> 
> *Web*: www.insight-centre.org
> *Phone:* +353892311519
>
>

Re: Zeppelin in local computer using yarn on distant cluster

[Hyung Sung Shim]

Could you share your zeppelin-env.sh ?
2016년 11월 2일 (수) 오후 4:57, Benoit Hanotte <benoit.h...@gmail.com>님이 작성:

> Thanks for your reply,
> I have tried setting it within zeppelin-env.sh but it doesn't work any
> better.
>
> Thanks
>
> On Wed, Nov 2, 2016 at 2:13 AM, Hyung Sung Shim <hss...@nflabs.com> wrote:
>
> Hello.
> You should set the HADOOP_CONF_DIR to /usr/local/lib/hadoop/etc/hadoop/ in
> the conf/zeppelin-env.sh.
> Thanks.
> 2016년 11월 2일 (수) 오전 5:07, Benoit Hanotte <benoit.h...@gmail.com>님이 작성:
>
> Hello,
>
> I'd like to use zeppelin on my local computer and use it to run spark
> executors on a distant yarn cluster since I can't easily install zeppelin
> on the cluster gateway.
>
> I installed the correct hadoop version (2.6), and compiled zeppelin (from
> the master branch) as following:
>
> *mvn clean package -DskipTests -Phadoop-2.6
> -Dhadoop.version=2.6.0-cdh5.5.0 -Pyarn -Pspark-2.0 -Pscala-2.11*
>
> I also set HADOOP_HOME_DIR to /usr/local/lib/hadoop where my hadoop is
> installed (I also tried with /usr/local/lib/hadoop/etc/hadoop/ where the
> conf files such as yarn-site.xml are). I set yarn.resourcemanager.hostname
> to the resource manager of the cluster (I copied the value from the config
> file on the cluster) but when I start a spark command it still tries to
> connect to 0.0.0.0:8032 as one can see in the logs:
>
> *INFO [2016-11-01 20:48:26,581] ({pool-2-thread-2}
> Client.java[handleConnectionFailure]:862) - Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032 <http://0.0.0.0/0.0.0.0:8032>. Already tried 9
> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)*
>
> Am I missing something something? Is there any additional parameters to
> set?
>
> Thanks!
>
> Benoit
>
>
>
>

Re: JDBC Connections

[Hyung Sung Shim]

Hello.
AFAIK The connections did not closed until restart JDBC Interpreter.
so https://github.com/apache/zeppelin/pull/1396 use ConnectionPool for
control sessions.


2016-10-19 2:43 GMT+09:00 Benjamin Kim :

> We are using Zeppelin 0.6.0 as a self-service for our clients to query our
> PostgreSQL databases. We are noticing that the connections are not closing
> after each one of them are done. What is the normal operating procedure to
> have these connections close when idle? Our scope for the JDBC interpreter
> is “shared”, which I thought would make 1 connection for all notebooks. It
> would seem that I am wrong. Anyone have any ideas on what would help?
>
> Thanks,
> Ben
>
>

Re: Embedding %html within %table

[Hyung Sung Shim]

Hi Kevin.
Could you try https://github.com/apache/zeppelin/pull/1433 ?
It would be great if you could let me know the result.
Thanks.

2016-09-17 10:15 GMT+09:00 moon soo Lee :

> I'm not sure since when, but %html inside of the cell doesn't work if it
> is first column. If you add any other column which is not using %html on
> left, table will be rendered correctly.
>
> Hope this helps
>
> Thanks,
> moon
>
>
> On Fri, Sep 16, 2016 at 2:21 PM Kevin Niemann 
> wrote:
>
>> According to the example here https://zeppelin.apache.
>> org/docs/0.6.1/displaysystem/basicdisplaysystem.html#table it is
>> possible to use %html within a %table paragraph and get benefits of both.
>> However when I run that it appears to render as HTML only and no native
>> table view.
>>
>>
>> Is my syntax incorrect or does anyone else have this same issue? e.g.
>>
>>
>> print(s"""%table
>>
>> name\tsize\tquantity
>>
>> %html sun\t100\t50
>>
>> %html > />moon\t10\t20""")
>>
>>
>> Thanks,
>>
>> kevin
>>
>

Re: FW: Issue with Zeppelin setup on Datastax-Spark

[Hyung Sung Shim]

Hello.
I think that `zeppelin.interpreter.host` and `zeppelin.interpreter.port`
values on Interpreter setting UI should be removed because Interpreter will
automatically be launched when you run a paragraph.

2016-09-16 3:56 GMT+09:00 Arpan Saha :

> Yes. It is a zeppelin log. Not spark interpreter log. Can you please tell
> me where can I get spark interpreter log? I don't find anything under
> zeppelin/logs/ path.
>
> Thanks,
> Arpan
> 
> From: DuyHai Doan [doanduy...@gmail.com]
> Sent: Thursday, September 15, 2016 11:20 AM
> To: users@zeppelin.apache.org
> Subject: Re: FW: Issue with Zeppelin setup on Datastax-Spark
>
> Is it Zeppelin log or Spark interpreter log ? I guess we need also the
> spark interpreter log to understand what is breaking Spark under the hood.
> org.apache.thrift.transport.TTransportException is just technical
> exception on inter-jvm communication
>
> On Thu, Sep 15, 2016 at 7:13 PM, Arpan Saha  mailto:arpan.s...@mindtree.com>> wrote:
>
> Hi,
>
> Please find below full stack trace.
>
> ERROR [2016-09-15 16:49:15,117] ({pool-1-thread-2}
> RemoteInterpreter.java[init]:182) - Failed to create interpreter:
> org.apache.zeppelin.spark.SparkInterpreter
> ERROR [2016-09-15 16:49:15,118] ({pool-1-thread-2} Job.java[run]:189) -
> Job failed
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.thrift.transport.TTransportException
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(
> RemoteInterpreter.java:183)
> at org.apache.zeppelin.interpreter.remote.
> RemoteInterpreter.getFormType(RemoteInterpreter.java:328)
> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.
> getFormType(LazyOpenInterpreter.java:105)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.
> java:260)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
> RemoteScheduler.java:328)
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.thrift.transport.TTransportException
> at org.apache.thrift.transport.TIOStreamTransport.read(
> TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.
> java:86)
> at org.apache.thrift.protocol.TBinaryProtocol.readAll(
> TBinaryProtocol.java:429)
> at org.apache.thrift.protocol.TBinaryProtocol.readI32(
> TBinaryProtocol.java:318)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(
> TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(
> TServiceClient.java:69)
> at org.apache.zeppelin.interpreter.thrift.
> RemoteInterpreterService$Client.recv_createInterpreter(
> RemoteInterpreterService.java:184)
> at org.apache.zeppelin.interpreter.thrift.
> RemoteInterpreterService$Client.createInterpreter(
> RemoteInterpreterService.java:168)
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(
> RemoteInterpreter.java:172)
> ... 12 more
> ERROR [2016-09-15 16:49:15,118] ({Thread-18} 
> RemoteInterpreterEventPoller.java[run]:88)
> - Can't get RemoteInterpreterEvent
>
> Thanks,
> Arpan.
>
> 
> From: DuyHai Doan [doanduy...@gmail.com]
> Sent: Thursday, September 15, 2016 4:29 AM
> To: users@zeppelin.apache.org
> Subject: Re: Issue with Zeppelin setup on Datastax-Spark
>
> Can you provide the complete stack trace ?
>
> Also, worth reading my blog post about integrating Spark/Zeppelin/C* or
> DSE : http://www.doanduyhai.com/blog/?p=2325
>
> On Thu, Sep 15, 2016 at 5:17 AM, Arpan Saha  mailto:arpan.s...@mindtree.com>>> wrote:
> Hi Team,
>
> I am trying to integrate Zeppelin 0.6.0 with DataStax 4.8.8 (which has
> Spark 1.4.2). After I configured following properties in zeppelin-env.sh
> when I start zeppelin daemon it started and in the browser I can see
> zeppelin is running but when I am trying to execute spark query in the
> notebook it is throwing below Error. Could you please help me to solve this
> issue.
>
> export JAVA_HOME=
> export SPARK_HOME=/etc/dse/spark
> export 

Re: Completion

[Hyung Sung Shim]

Maybe it was my misunderstood, I thought that doesn't work pop-up for
completion.
I just tested spark auto completion but it doesn't work as you said.
I think this issue is from spark-2.0 supporting.
Do you mind creating jira issue for this?

2016-09-13 0:11 GMT+09:00 Cyril Scetbon <cyril.scet...@free.fr>:

> Hey,
>
> not working too :(
> I tried "ctrl.", "cmd.", "alt.", nothing works
>
> On Sep 12, 2016, at 10:00, Hyung Sung Shim <hss...@nflabs.com> wrote:
>
> Hello.
> Could you put the key "Ctrl ." (Ctrl + dot) for the completion?
>
> 2016-09-12 22:43 GMT+09:00 Cyril Scetbon <cyril.scet...@free.fr>:
>
>> Hi,
>>
>> I'm testing Zeppelin 0.6.0 and I've not been able to use completion on
>> OSX. I tried the combination ctrl+ with both Safari and Chrome without
>> success. Am I missing something ? Does it work only with some interpreters
>> and not others ? I tried to use only with the default interpreter (%spark)
>>
>> Thanks
>
>
>
>

Re: Completion

[Hyung Sung Shim]

Hello.
Could you put the key "Ctrl ." (Ctrl + dot) for the completion?

2016-09-12 22:43 GMT+09:00 Cyril Scetbon :

> Hi,
>
> I'm testing Zeppelin 0.6.0 and I've not been able to use completion on
> OSX. I tried the combination ctrl+ with both Safari and Chrome without
> success. Am I missing something ? Does it work only with some interpreters
> and not others ? I tried to use only with the default interpreter (%spark)
>
> Thanks

Re: Running R on Zeppelin EMR Cluster

[Hyung Sung Shim]

and EMR-5.0.0 supports Zeppelin 0.6.1.


2016-09-07 0:24 GMT+09:00 Hyung Sung Shim <hss...@nflabs.com>:

> Hi.
> Unfortunately Zeppelin 0.5.6 does not support R interpreter.
> Could you upgrade your Zeppelin to higher version?
>
> 2016-09-06 23:53 GMT+09:00 Mark Mikolajczak - 07855 306 064 <
> m...@flayranalytics.co.uk>:
>
>> Hi All,
>>
>> I am trying to setup the R interpreter to run in Zeppelin which is
>> currently running on EMR. Zeppelin is working perfectly and I am able to
>> write script in Scala and Python. When I use %r, %sparkR or %knitr I
>> receive an error : "r interpreter not found"
>>
>> The applications which I have running in my emr-4.7.2 cluster are: Hive
>> 1.0.0, Zeppelin-Sandbox 0.5.6, Spark 1.6.2, Pig 0.14.0
>>
>> Within the interpreter there is no mention of R so figure I am missing
>> something but do not know what.
>>
>> Any pointers greatly appreciated.
>>
>
>

Re: Running R on Zeppelin EMR Cluster

[Hyung Sung Shim]

Hi.
Unfortunately Zeppelin 0.5.6 does not support R interpreter.
Could you upgrade your Zeppelin to higher version?

2016-09-06 23:53 GMT+09:00 Mark Mikolajczak - 07855 306 064 <
m...@flayranalytics.co.uk>:

> Hi All,
>
> I am trying to setup the R interpreter to run in Zeppelin which is
> currently running on EMR. Zeppelin is working perfectly and I am able to
> write script in Scala and Python. When I use %r, %sparkR or %knitr I
> receive an error : "r interpreter not found"
>
> The applications which I have running in my emr-4.7.2 cluster are: Hive
> 1.0.0, Zeppelin-Sandbox 0.5.6, Spark 1.6.2, Pig 0.14.0
>
> Within the interpreter there is no mention of R so figure I am missing
> something but do not know what.
>
> Any pointers greatly appreciated.
>

Re: Spark error when loading phoenix-spark dependency

[Hyung Sung Shim]

I think [1] is might related issue.
Could you use 4.7.0 instead 4.4.0?

[1]
https://community.hortonworks.com/questions/17861/error-starting-spark-shell-with-phoenix-client-jar.html


2016-09-06 14:21 GMT+09:00 Vikash Kumar <vikash.ku...@resilinc.com>:

> Hi,
>
> I am loading the library through UI in spark interpreter as:
>
>
>
> 1.   org.apache.phoenix:phoenix-spark:4.4.0-HBase-1.1
>
> Excluded :- org.scala-lang:scala-library, org.scala-lang:scala-compiler,
> org.scala-lang:scala-reflect, org.apache.phoenix:phoenix-core
>
>
>
> 2.   org.apache.phoenix:phoenix-core:4.4.0-HBase-1.1
>
> Excluded :- com.sun.jersey:jersey-core, com.sun.jersey:jersey-server,
> com.sun.jersey:jersey-client, org.ow2.asm:asm, io.netty:netty
>
>
>
> Thanks and Regard,
>
> Vikash Kumar
>
>
>
> *From:* astros...@gmail.com [mailto:astros...@gmail.com] *On Behalf Of *Hyung
> Sung Shim
> *Sent:* Tuesday, September 6, 2016 10:47 AM
> *To:* users <users@zeppelin.apache.org>
> *Subject:* Re: Spark error when loading phoenix-spark dependency
>
>
>
> Hello.
>
> How did you load library?
>
>
>
>
>
> 2016-09-06 13:49 GMT+09:00 Vikash Kumar <vikash.ku...@resilinc.com>:
>
> Hi ,
>
> Is there anyone who is getting the same errors?
>
>
>
> Thanks and Regard,
>
> Vikash Kumar
>
>
>
> *From:* Vikash Kumar [mailto:vikash.ku...@resilinc.com]
> *Sent:* Thursday, September 1, 2016 11:08 AM
> *To:* users@zeppelin.apache.org
> *Subject:* Spark error when loading phoenix-spark dependency
>
>
>
> Hi all,
>
> I am getting the following error when loading the
> org.apache.phoenix:phoenix-spark:4.4.0-HBase-1.1 dependency from spark
> interpreter. I am using Zeppelin *Version 0.6.2-SNAPSHOT* with spark
> 1.6.1 and hdp 2.7.1.
>
>
>
> The packages that I am inporting is:
>
> import org.apache.phoenix.spark._
>
> import org.apache.phoenix.spark.PhoenixRDD._
>
> import java.sql.{ Date, Timestamp}
>
> My build command is
>
> mvn clean package -DskipTests -Drat.ignoreErrors=true
> -Dcheckstyle.skip=true -Pspark-1.6 -Dspark.version=1.6.1 -Phadoop-2.6 –Pyarn
>
>
>
>
>
> java.lang.NoSuchMethodError: org.apache.spark.util.Utils$.
> resolveURIs(Ljava/lang/String;)Ljava/lang/String;
>
> at org.apache.spark.repl.SparkILoop$.getAddedJars(
> SparkILoop.scala:1079)
>
> at org.apache.spark.repl.SparkILoop.createInterpreter(
> SparkILoop.scala:210)
>
> at org.apache.zeppelin.spark.SparkInterpreter.open(
> SparkInterpreter.java:698)
>
> at org.apache.zeppelin.interpreter.
> LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
>
> at org.apache.zeppelin.interpreter.
> LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
>
> at org.apache.zeppelin.interpreter.remote.
> RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:
> 341)
>
> at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
>
> at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(
> FIFOScheduler.java:139)
>
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.run(FutureTask.
> java:266)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
>
>
>
>
>
>
> Thanks and Regard,
>
> Vikash Kumar
>
>
>

Re: Spark error when loading phoenix-spark dependency

[Hyung Sung Shim]

You can refer [1] to load dependency loading library.

[1]
http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/spark.html#dependencyloading


2016-09-06 14:16 GMT+09:00 Hyung Sung Shim <hss...@nflabs.com>:

> Hello.
> How did you load library?
>
>
> 2016-09-06 13:49 GMT+09:00 Vikash Kumar <vikash.ku...@resilinc.com>:
>
>> Hi ,
>>
>> Is there anyone who is getting the same errors?
>>
>>
>>
>> Thanks and Regard,
>>
>> Vikash Kumar
>>
>>
>>
>> *From:* Vikash Kumar [mailto:vikash.ku...@resilinc.com]
>> *Sent:* Thursday, September 1, 2016 11:08 AM
>> *To:* users@zeppelin.apache.org
>> *Subject:* Spark error when loading phoenix-spark dependency
>>
>>
>>
>> Hi all,
>>
>> I am getting the following error when loading the
>> org.apache.phoenix:phoenix-spark:4.4.0-HBase-1.1 dependency from spark
>> interpreter. I am using Zeppelin *Version 0.6.2-SNAPSHOT* with spark
>> 1.6.1 and hdp 2.7.1.
>>
>>
>>
>> The packages that I am inporting is:
>>
>> import org.apache.phoenix.spark._
>>
>> import org.apache.phoenix.spark.PhoenixRDD._
>>
>> import java.sql.{ Date, Timestamp}
>>
>> My build command is
>>
>> mvn clean package -DskipTests -Drat.ignoreErrors=true
>> -Dcheckstyle.skip=true -Pspark-1.6 -Dspark.version=1.6.1 -Phadoop-2.6 –Pyarn
>>
>>
>>
>>
>>
>> java.lang.NoSuchMethodError: org.apache.spark.util.Utils$.r
>> esolveURIs(Ljava/lang/String;)Ljava/lang/String;
>>
>> at org.apache.spark.repl.SparkILo
>> op$.getAddedJars(SparkILoop.scala:1079)
>>
>> at org.apache.spark.repl.SparkILo
>> op.createInterpreter(SparkILoop.scala:210)
>>
>> at org.apache.zeppelin.spark.Spar
>> kInterpreter.open(SparkInterpreter.java:698)
>>
>> at org.apache.zeppelin.interprete
>> r.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
>>
>> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.
>> interpret(LazyOpenInterpreter.java:93)
>>
>> at org.apache.zeppelin.interprete
>> r.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteI
>> nterpreterServer.java:341)
>>
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
>>
>> at org.apache.zeppelin.scheduler.
>> FIFOScheduler$1.run(FIFOScheduler.java:139)
>>
>> at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:511)
>>
>> at java.util.concurrent.FutureTas
>> k.run(FutureTask.java:266)
>>
>> at java.util.concurrent.Scheduled
>> ThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledT
>> hreadPoolExecutor.java:180)
>>
>> at java.util.concurrent.Scheduled
>> ThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPo
>> olExecutor.java:293)
>>
>> at java.util.concurrent.ThreadPoo
>> lExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>
>> at java.util.concurrent.ThreadPoo
>> lExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Thanks and Regard,
>>
>> Vikash Kumar
>>
>
>

[ANNOUNCE] Apache Zeppelin 0.6.1 released

[Hyung Sung Shim]

Congratulations for 0.6.1 release!
Thank you for everyone especially mina.

2016-08-16 16:57 GMT+09:00 Mina Lee >:

> The Apache Zeppelin community is pleased to announce the availability of
> the 0.6.1 release.
>
> Zeppelin is a collaborative data analytics and visualization tool for
> distributed, general-purpose data processing system such as Apache Spark,
> Apache Flink, etc.
>
> The community put significant effort into improving Apache Zeppelin since
> the last release, focusing on supporting Scala 2.11 and Spark 2.0.
> 22 contributors provided 80+ patches for new features, improvements
> and bug fixes. More than 60+ issues have been resolved.
>
> We encourage download the latest release from
> http://zeppelin.apache.org/download.html
>
> Release note is available at
> http://zeppelin.apache.org/releases/zeppelin-release-0.6.1.html
>
> We welcome your help and feedback. For more information on the project and
> how to get involved, visit our website at http://zeppelin.apache.org/
>
> Thanks to all users and contributors who have helped to improve Apache
> Zeppelin.
>
> Regards,
> The Apache Zeppelin community
>

Re: integrate authorization

[Hyung Sung Shim]

Hi.
Unfortunetely There is no way to solve your problem on zeppelin level but
it seems like interesting feature so could you create issue for this?

2016년 7월 20일 수요일, 仓库快满了님이 작성한 메시지:

> Hi,
> I have a question.
> Can I pass the authorization from one web system to this zeppelin web
> system?
> I have a web system,and it has its own authorization.I want to integrate
> it with zeppelin.
> For example,when I logged in my web system as user1, and I click a link
> and jump to zeppelin system,I want it is also logged as user1.
> As I know,the Shiro or nginx is recommended,but their users and roles are
> written in a config file.My web system's users and roles are stored in
> Mysql database and I want the zeppelin to use the same users and roles.
> Can you give me some advise or examples?
>
>
> Thanks.

java.lang.ClassNotFoundException

[Hyung Sung Shim]

Hi.
The problem of showing result basic table issue was fixed on the master.


2016년 7월 20일 수요일, Gilad Landau>님이 작성한 메시지:

> Hello All,
>
>
>
> I have another issue with zeppelin 0.6.0…
>
> When using %sql interpreter, I manage to get SQL results but not into the
> basic table data visualization… The data does appear on the pie-chart,
> scatter-plot etc’ but not in the basic table results.
>
>
>
> Any idea what could cause this behavior?
>
>
>
> Thanks,
>
>
>
> Gilad
>
>
>

Re: Order of paragraphs vs. different interpreters (spark vs. pyspark)

[Hyung Sung Shim]

hi
I think you can run the workflows that you defined just 'run' paragraph.
and I believe functionality of view are going to be better. :)

2016년 7월 14일 목요일, xiufeng liu<toxiuf...@gmail.com>님이 작성한 메시지:

> It is easy to change the code. I did myself and use it as an ETL tool. It
> is very powerful
>
> Afancy
>
> On Wednesday, July 13, 2016, Ahmed Sobhi <ahmed.so...@gmail.com
> <javascript:_e(%7B%7D,'cvml','ahmed.so...@gmail.com');>> wrote:
>
>> I think this pr addresses what I need. Case 2 seem to describe the issue
>> I'm having if I'm reading it correctly.
>>
>> The proposed solution, however, is not that clear to me.
>>
>> Is it that you define workflows where a work flow is a sequence of
>> (notebook, paragraph) pairs that are to be run in a specific order?
>> If that's the case, then this definitely solves my problem, but it's
>> really cumbersome from a usability point of view. I think a better solution
>> for my use case is to just have an option to run all paragraphs in the
>> order they appear in on the notebook, regardless of which interpreter they
>> use.
>>
>> On Wed, Jul 13, 2016 at 12:31 PM, Hyung Sung Shim <hss...@nflabs.com>
>> wrote:
>>
>>> hi.
>>> Maybe https://github.com/apache/zeppelin/pull/1176 is related what you
>>> want.
>>> Please check this pr.
>>>
>>> 2016년 7월 13일 수요일, xiufeng liu<toxiuf...@gmail.com>님이 작성한 메시지:
>>>
>>> You have to change the source codes to add the dependencies of running
>>>> paragraphs. I think it is a really interesting feature, for example, it can
>>>> be use as an ETL tool. But, unfortunately, there is no configuration option
>>>> right now.
>>>>
>>>> /afancy
>>>>
>>>> On Wed, Jul 13, 2016 at 12:27 PM, Ahmed Sobhi <ahmed.so...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I have been working on a large Spark Scala notebook. I recently had
>>>>> the requirement to produce graphs/plots out of these data. Python and
>>>>> PySpark seemed like a natural fit but since I've already invested a lot of
>>>>> time and effort into the Scala version, I want to restrict my usage of
>>>>> python to just plotting.
>>>>>
>>>>> I found a good workflow for where in the scala paragraphs I can use 
>>>>> *registerTempTable
>>>>> *and in python I can just use *sqlContext.table *to retrieve that
>>>>> table.
>>>>>
>>>>> The problem now is that if I try to run all paragraphs to get the
>>>>> notebook updated, the python paragraphs fail because they are running
>>>>> before the scala ones eventhough they are placed after them.
>>>>>
>>>>> It seems like the behavior in Zeppelin is that it attempts to run the
>>>>> paragraphs concurrently if they were running on different interpreters
>>>>> which might seem fine on the surface. But now that I want to introduce 
>>>>> some
>>>>> dependency between spark/pyspark paragraphs, is there any way to do that?
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> Ahmed
>>>>>
>>>>
>>>>
>>
>>
>> --
>> Cheers,
>> Ahmed
>> http://bit.ly/ahmed_abtme <http://about.me/humanzz>
>>
>

Re: Installing Zeppelin 0.6.0 on latest EMR (4.7.1)

[Hyung Sung Shim]

I think it seems possible but i don't have experience.
Could you upgrade version following [1] and share the results?
It'll be really a good case for users.

[1]
http://zeppelin.apache.org/docs/0.6.0/install/upgrade.html

2016-07-07 23:42 GMT+09:00 Panayotis Trapatsas <p.trapat...@pamediakopes.gr>
:

> Yes, I understand that, but is there a way to upgrade or to install Zeppelin
> 0.6.0 in a EMR cluster?
>
> On Thu, Jul 7, 2016 at 4:24 PM, Hyung Sung Shim <hss...@nflabs.com> wrote:
>
>> Hi.
>> Zeppelin 0.5.6 Sandbox is in the EMR-4.7.x
>> so If you want to use zeppelin 0.6 with EMR-4.7, I think you can use EC2
>> as your zeppelin machine.
>>
>>
>> 2016-07-07 22:08 GMT+09:00 Panayotis Trapatsas <
>> p.trapat...@pamediakopes.gr>:
>>
>>> Hi,
>>>
>>> Is there a way to install Zeppelin 0.6.0 on EMR version 4.7.x?
>>>
>>> --
>>>
>>> [image: e-Travel SA] <http://pamediakopes.gr/>
>>>
>>> Panayotis Trapatsas / Lead Data Engineer
>>> p.trapat...@pamediakopes.gr / +30 6979493921
>>>
>>> e-Travel SA
>>> Office: +30 213 0184000/ Fax: +30 211 8001181
>>> Leof. Kifisias 7, AmpelokipiAthens 115 23, Greece
>>> pamediakopes.gr |  trip.ru |  airtickets24.com |  fantasticgreece.com
>>>
>>> [image: Google +] <https://plus.google.com/u/0/+PanayotisTrapatsas>[image:
>>> Stack Overflow] <http://stackoverflow.com/users/2618541/panayotis>
>>>
>>
>>
>
>
> --
>
> [image: e-Travel SA] <http://pamediakopes.gr/>
>
> Panayotis Trapatsas / Lead Data Engineer
> p.trapat...@pamediakopes.gr / +30 6979493921
>
> e-Travel SA
> Office: +30 213 0184000/ Fax: +30 211 8001181
> Leof. Kifisias 7, AmpelokipiAthens 115 23, Greece
> pamediakopes.gr |  trip.ru |  airtickets24.com |  fantasticgreece.com
>
> [image: Google +] <https://plus.google.com/u/0/+PanayotisTrapatsas>[image:
> Stack Overflow] <http://stackoverflow.com/users/2618541/panayotis>
>

Re: classnotfoundexception using zeppelin with spark authentication

[Hyung Sung Shim]

Hi. Jonathan.
If it is possible, could you use the zeppelin master on the github?
Zeppelin is working well with spark authentication when i tested.



2016-07-02 3:05 GMT+09:00 Jonathan Esterhazy <jonathan.esterh...@gmail.com>:

> Hyung, thx for your help. I opened these:
>
> https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala problem)
> https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking
> python problem)
>
> LMK if I can provide more info or help in some way.
>
> On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim <hss...@nflabs.com> wrote:
>
>> Hi Jonathan.
>> Unfortunately I got same error in my test bed.
>> Do you mind create an jira issue for this?
>>
>> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com>:
>>
>>> I added this and still get the same exception. The same property is also
>>> set in spark-defaults.conf.
>>>
>>> After that didn't work, I also tried adding --conf
>>> spark.authenticate=true and --conf
>>> spark.authenticate.enableSaslEncryption=true, to match the other related
>>> settings in spark-defaults.conf. Still get the same classnotfoundexception.
>>>
>>>
>>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim <hss...@nflabs.com>
>>> wrote:
>>>
>>>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>>>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and restart
>>>> zeppelin and retry your code.
>>>>
>>>>
>>>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>>>> jonathan.esterh...@gmail.com>:
>>>>
>>>>> yes it does. i only see this problem in zeppelin.
>>>>>
>>>>> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <hss...@nflabs.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Jonathan.
>>>>>> It's not easy to build the test environments but I am working on this.
>>>>>> I have question for you.
>>>>>> Does your code working well on spark-shell in the spark.authenticate
>>>>>> mode?
>>>>>>
>>>>>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>>>>>> jonathan.esterh...@gmail.com>:
>>>>>>
>>>>>>> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm
>>>>>>> quite stumped.
>>>>>>>
>>>>>>> thx.
>>>>>>>
>>>>>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <hss...@nflabs.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thank you.
>>>>>>>> Let me try.
>>>>>>>>
>>>>>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>>>>>>> jonathan.esterh...@gmail.com>:
>>>>>>>>
>>>>>>>>> Hyung,
>>>>>>>>>
>>>>>>>>> Yes, here they are.
>>>>>>>>>
>>>>>>>>> zeppelin-env.sh:
>>>>>>>>>
>>>>>>>>> export ZEPPELIN_PORT=8890
>>>>>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>>>>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>>>>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>>>>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>>>>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>>>>>>>> export MASTER=yarn-client
>>>>>>>>> export SPARK_HOME=/usr/lib/spark
>>>>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>>>>> export
>>>>>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>>>>>>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>>>>>>>> export
>>>>>>>>> ZEPPELIN_NOTEB

Re: Installing Zeppelin 0.6.0 on latest EMR (4.7.1)

[Hyung Sung Shim]

Hi.
Zeppelin 0.5.6 Sandbox is in the EMR-4.7.x
so If you want to use zeppelin 0.6 with EMR-4.7, I think you can use EC2 as
your zeppelin machine.


2016-07-07 22:08 GMT+09:00 Panayotis Trapatsas 
:

> Hi,
>
> Is there a way to install Zeppelin 0.6.0 on EMR version 4.7.x?
>
> --
>
> [image: e-Travel SA] 
>
> Panayotis Trapatsas / Lead Data Engineer
> p.trapat...@pamediakopes.gr / +30 6979493921
>
> e-Travel SA
> Office: +30 213 0184000/ Fax: +30 211 8001181
> Leof. Kifisias 7, AmpelokipiAthens 115 23, Greece
> pamediakopes.gr |  trip.ru |  airtickets24.com |  fantasticgreece.com
>
> [image: Google +] [image:
> Stack Overflow] 
>

Re: Error installing Zeppelin 0.6.0 on Ubuntu 14.04 using pre build package

[Hyung Sung Shim]

Hi.
There are three types of zeppelin 6.0.
One is source that you should build to use and Two are binary version that
you don't need to build to use.
What did you get?



2016-07-07 21:20 GMT+09:00 B00083603 Michael O Brien <
b00083...@student.itb.ie>:

> Hi all,
>
>
> I was hoping to use Zeppelin for some data analysis but I'm having trouble
> installing it using the pre-built package.
>
>
> I download the tar and extract to /usr/share/zeppelin so its
> /usr/share/zeppelin/zeppelin0.6.0/
>
> I have default jdk installed and JAVA_HOME set as
>  /usr/lib/jvm/java-7-openjdk-amd64 by editing /etc/environment
>
>
> java version "1.7.0_101"
>
> OpenJDK Runtime Environment (IcedTea 2.6.6) (7u101-2.6.6-0ubuntu0.14.04.1)
> OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
>
> When I run /bin/zeppelin-daemon.sh start it initially creates the log
> folders and reports
> Zeppelin start [  OK  ]
> Zeppelin process died  [FAILED]
>
> The log file zeppelin-username-computername.out just contains *"**Error:
> Could not find or load main
> class org.apache.zeppelin.server.ZeppelinServer"*
>
> Any pointers on how I can get the pre-built zeppelin working?
>
> Michael
>
> ITB Email Disclaimer This is a confidential communication and is intended
> only for the addressee indicated in the message (or duly authorised to be
> responsible for the delivery of the message to such person). You are
> specifically prohibited from copying this message or delivering the same,
> or any part thereof, to any other person, whomsoever or howsoever, unless
> you receive written authorisation from us to do. If you are anyone other
> than the intended addressee, or person duly authorised and responsible for
> the delivery of this message to the intended addressee, you should destroy
> this message and notify us immediately.
>

Re: classnotfoundexception using zeppelin with spark authentication

[Hyung Sung Shim]

Hi Jonathan.
Unfortunately I got same error in my test bed.
Do you mind create an jira issue for this?

2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <jonathan.esterh...@gmail.com>:

> I added this and still get the same exception. The same property is also
> set in spark-defaults.conf.
>
> After that didn't work, I also tried adding --conf spark.authenticate=true
> and --conf spark.authenticate.enableSaslEncryption=true, to match the other
> related settings in spark-defaults.conf. Still get the same
> classnotfoundexception.
>
>
> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim <hss...@nflabs.com>
> wrote:
>
>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and restart
>> zeppelin and retry your code.
>>
>>
>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com>:
>>
>>> yes it does. i only see this problem in zeppelin.
>>>
>>> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <hss...@nflabs.com>
>>> wrote:
>>>
>>>> Hi Jonathan.
>>>> It's not easy to build the test environments but I am working on this.
>>>> I have question for you.
>>>> Does your code working well on spark-shell in the spark.authenticate
>>>> mode?
>>>>
>>>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>>>> jonathan.esterh...@gmail.com>:
>>>>
>>>>> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm
>>>>> quite stumped.
>>>>>
>>>>> thx.
>>>>>
>>>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <hss...@nflabs.com>
>>>>> wrote:
>>>>>
>>>>>> Thank you.
>>>>>> Let me try.
>>>>>>
>>>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>>>>> jonathan.esterh...@gmail.com>:
>>>>>>
>>>>>>> Hyung,
>>>>>>>
>>>>>>> Yes, here they are.
>>>>>>>
>>>>>>> zeppelin-env.sh:
>>>>>>>
>>>>>>> export ZEPPELIN_PORT=8890
>>>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>>>>>> export MASTER=yarn-client
>>>>>>> export SPARK_HOME=/usr/lib/spark
>>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>>> export
>>>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>>>>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>>>>>> export
>>>>>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>>>>>>
>>>>>>> spark-defaults.conf:
>>>>>>>
>>>>>>> spark.master yarn
>>>>>>> spark.driver.extraClassPath
>>>>>>>  
>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>> spark.driver.extraLibraryPath
>>>>>>>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>> spark.executor.extraClassPath
>>>>>>>  
>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>> spark.executor.extraLibraryPath
>>>>>>>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/nati

Re: classnotfoundexception using zeppelin with spark authentication

[Hyung Sung Shim]

Please add the *export SPARK_SUBMIT_OPTIONS="--conf
spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and restart
zeppelin and retry your code.


2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <jonathan.esterh...@gmail.com>
:

> yes it does. i only see this problem in zeppelin.
>
> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <hss...@nflabs.com>
> wrote:
>
>> Hi Jonathan.
>> It's not easy to build the test environments but I am working on this.
>> I have question for you.
>> Does your code working well on spark-shell in the spark.authenticate mode?
>>
>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com>:
>>
>>> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm
>>> quite stumped.
>>>
>>> thx.
>>>
>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <hss...@nflabs.com>
>>> wrote:
>>>
>>>> Thank you.
>>>> Let me try.
>>>>
>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>>> jonathan.esterh...@gmail.com>:
>>>>
>>>>> Hyung,
>>>>>
>>>>> Yes, here they are.
>>>>>
>>>>> zeppelin-env.sh:
>>>>>
>>>>> export ZEPPELIN_PORT=8890
>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>>>> export MASTER=yarn-client
>>>>> export SPARK_HOME=/usr/lib/spark
>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>> export
>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>>>> export
>>>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>>>>
>>>>> spark-defaults.conf:
>>>>>
>>>>> spark.master yarn
>>>>> spark.driver.extraClassPath
>>>>>  
>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>> spark.driver.extraLibraryPath
>>>>>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>> spark.executor.extraClassPath
>>>>>  
>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>> spark.executor.extraLibraryPath
>>>>>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>> spark.eventLog.enabled   true
>>>>> spark.eventLog.dir   hdfs:///var/log/spark/apps
>>>>> spark.history.fs.logDirectoryhdfs:///var/log/spark/apps
>>>>> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
>>>>> spark.history.ui.port18080
>>>>> spark.shuffle.service.enabledtrue
>>>>> spark.driver.extraJavaOptions
>>>>>  -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
>>>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
>>>>> -XX:MaxHeapFreeRatio=70
>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
>>>>> -XX:OnOutOfMemoryError='kill -9 %p'
>>>>> spark.dynamicAllocation.enabled  true
>>>>> spark.executor.extraJavaOptions  -verbose:gc -XX:+PrintGCDetails
>>>>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
>>>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
>>>>> spark.executor.memory8640m
>>>>> spark.exec

Re: classnotfoundexception using zeppelin with spark authentication

[Hyung Sung Shim]

Hi Jonathan.
It's not easy to build the test environments but I am working on this.
I have question for you.
Does your code working well on spark-shell in the spark.authenticate mode?

2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <jonathan.esterh...@gmail.com>
:

> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm quite
> stumped.
>
> thx.
>
> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <hss...@nflabs.com>
> wrote:
>
>> Thank you.
>> Let me try.
>>
>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com>:
>>
>>> Hyung,
>>>
>>> Yes, here they are.
>>>
>>> zeppelin-env.sh:
>>>
>>> export ZEPPELIN_PORT=8890
>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>> export MASTER=yarn-client
>>> export SPARK_HOME=/usr/lib/spark
>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>> export
>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>> export
>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>>
>>> spark-defaults.conf:
>>>
>>> spark.master yarn
>>> spark.driver.extraClassPath
>>>  
>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>> spark.driver.extraLibraryPath
>>>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>> spark.executor.extraClassPath
>>>  
>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>> spark.executor.extraLibraryPath
>>>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>> spark.eventLog.enabled   true
>>> spark.eventLog.dir   hdfs:///var/log/spark/apps
>>> spark.history.fs.logDirectoryhdfs:///var/log/spark/apps
>>> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
>>> spark.history.ui.port18080
>>> spark.shuffle.service.enabledtrue
>>> spark.driver.extraJavaOptions
>>>  -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
>>> -XX:MaxHeapFreeRatio=70
>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
>>> -XX:OnOutOfMemoryError='kill -9 %p'
>>> spark.dynamicAllocation.enabled  true
>>> spark.executor.extraJavaOptions  -verbose:gc -XX:+PrintGCDetails
>>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
>>> spark.executor.memory8640m
>>> spark.executor.cores 7
>>> spark.authenticate.enableSaslEncryption true
>>> spark.driver.memory  1g
>>> spark.network.sasl.serverAlwaysEncrypt true
>>> spark.driver.cores   1
>>> spark.ssl.protocol   TLSv1.2
>>> spark.ssl.keyStorePassword   password
>>> spark.yarn.maxAppAttempts1
>>> spark.ssl.keyStore   /etc/emr/security/keystore.jks
>>> spark.authenticate   true
>>> spark.ssl.keyPasswordpassword
>>> spark.ssl.enabledtrue
>>> spark.ssl.enabledAlgorithms  TLS_RSA_WITH_AES_256_CBC_SHA
>>> spark.ssl.trustStore /etc/emr/security/truststore.jks
>>> spark.authenticate.secretsecret
>>> spark.ssl.trustStorePassword password
>>>
>>>
>>>
>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim <hss...@nflabs.com>
>>> wrote:
>&

Re: Running notebook from terminal

[Hyung Sung Shim]

Hi.
You can do what you want using zeppelin REST API[1] .

[1]
http://zeppelin.apache.org/docs/0.6.0-SNAPSHOT/rest-api/rest-notebook.html

Thanks.

2016-06-28 11:45 GMT+09:00 Chandra Maloo :

> Hello,
> I have a notebook with multiple paragraphs. I have embedded the results of
> those paragraphs in a webpage as iframes. Whenever my data changes I have
> to open the notebook and run all paragraphs to re generate the graphs. Is
> there some way I could do this without UI so that maybe i can run a cron
> job for it.
> Looking forward to reply.
>
> Thanks,
> Chandra
>

Re: classnotfoundexception using zeppelin with spark authentication

[Hyung Sung Shim]

Hi.
Could you share your conf/zeppelin-env.sh and spark-defaults.conf ?

2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy :

> I am having trouble using zeppelin in a spark cluster that has spark node
> authentication turned on (e.g. with spark.authenticate=true,
> spark.authenticate.secret=...)
>
> Notebook code that calls built-in spark functions (or other things on
> executor classpath) work fine, but functions defined in the notebook
> (anonymous or named) throw ClassNotFoundExceptions when called from an
> executor.
>
> For example, this code works:
>
> val rdd = sc.textFile("hdfs://my-text-file")
> rdd.take(1).foreach(println)
>
> rdd.saveAsTextFile("hdfs:///my-other-text-file")
>
> but code like this...
>
> rdd.filter(_.contains("my data"))
>
> fails with
>
> Caused by: java.lang.ClassNotFoundException:
> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
> at
> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> ...
>
>
> I get the same kind of error if the filter function is defined as a named
> function in the notebook, or as a member of singleton object defined in the
> notebook.
>
> When I look at the executor's log output, I see this error:
>
> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to check
> existence of class
> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on REPL
> class server at https://172.30.54.30:34980
> java.lang.NullPointerException
> at
> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113)
> at
> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146)
> at
> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
>
> ...
>
>
> If I disable spark authentication, everything works as expected. I am
> running zeppelin 0.5.6 on spark 1.6.1 with yarn.
>
> Has anyone been able to get zeppelin working with spark authentication?
>
>
>

Re: Logout from Zeppelin

[Hyung Sung Shim]

Hi.
I couldn't reproduce your situation.
Do you mind create an jira issue?
Thanks.

2016-06-23 13:14 GMT+09:00 Abhisar Mohapatra <abhisar.mohapa...@inmobi.com>:

> Only when a browser is closed and again started, login is asked for.
> I tried crearing cookies and cached pages but doesn't seem to work.But
> when i open a new window it asks  for login
>
> On Thu, Jun 23, 2016 at 9:32 AM, Abhisar Mohapatra <
> abhisar.mohapa...@inmobi.com> wrote:
>
>> I have taken  master branch of zeppelin and then built it.it was 2 days
>> back when I built it.
>> I have tried clearing cookies and all.
>> Check the attached screen shots sorted based on time.It happens when i do
>> logout.
>>
>> On Thu, Jun 23, 2016 at 9:13 AM, Hyung Sung Shim <hss...@nflabs.com>
>> wrote:
>>
>>> Hi.
>>> What zeppelin version are you using?
>>> and Can you try after removing cookies in your browser?
>>> If it happens same, please share your screen shot.
>>>
>>> 2016-06-23 12:32 GMT+09:00 Abhisar Mohapatra <
>>> abhisar.mohapa...@inmobi.com>:
>>>
>>>> Hi,
>>>> I configured Zeppelin with basic Shiro authentication.Login and the all
>>>> the regular notebook creation and  code execution happen fine.
>>>> But when I log out ,it redirects me to the same home page without
>>>> logging me out.Is there any problem or am I missing some step ?
>>>>
>>>> _
>>>> The information contained in this communication is intended solely for
>>>> the use of the individual or entity to whom it is addressed and others
>>>> authorized to receive it. It may contain confidential or legally privileged
>>>> information. If you are not the intended recipient you are hereby notified
>>>> that any disclosure, copying, distribution or taking any action in reliance
>>>> on the contents of this information is strictly prohibited and may be
>>>> unlawful. If you have received this communication in error, please notify
>>>> us immediately by responding to this email and then delete it from your
>>>> system. The firm is neither liable for the proper and complete transmission
>>>> of the information contained in this communication nor for any delay in its
>>>> receipt.
>>>
>>>
>>>
>>
>
> _
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>

Re: Build error zengine

[Hyung Sung Shim]

good to hear that it worked anyway. 

2016년 6월 22일 수요일, Darren Govoni<dar...@ontrenet.com>님이 작성한 메시지:

> H. It worked the 3rd time. Not sure what the hiccup was.
>
> On 06/21/2016 11:46 AM, Hyung Sung Shim wrote:
>
> hi.
> I just got build success on my ubuntu machine as your build command.
> Did you install prerequisites things[1] to build zeppelin?
> And Can you share your build log?
>
> [1]
> https://github.com/apache/zeppelin
>
> 2016-06-21 23:47 GMT+09:00 Vinay Shukla <vinayshu...@gmail.com
> <javascript:_e(%7B%7D,'cvml','vinayshu...@gmail.com');>>:
>
>> What is the exact build failure?
>>
>>
>> On Tuesday, June 21, 2016, Darren Govoni <
>> <javascript:_e(%7B%7D,'cvml','dar...@ontrenet.com');>dar...@ontrenet.com
>> <javascript:_e(%7B%7D,'cvml','dar...@ontrenet.com');>> wrote:
>>
>>> Ubuntu 15.10
>>>
>>> mvn clean package -Pspark-1.6 -Phadoop-2.4 -Pyarn -Ppyspark
>>>
>>>
>>> On 06/21/2016 09:03 AM, Hyung Sung Shim wrote:
>>>
>>> hi.
>>> What is your build command and please tell me your environments.
>>>
>>> 2016-06-21 21:45 GMT+09:00 Darren Govoni <dar...@ontrenet.com
>>> <javascript:_e(%7B%7D,'cvml','dar...@ontrenet.com');>>:
>>>
>>>> Hi
>>>>
>>>> I am trying to build git repo but zengine fails. Any tips on this?
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>> Sent from my Verizon Wireless 4G LTE smartphone
>>>>
>>>
>>>
>>>
>
>

Re: Build error zengine

[Hyung Sung Shim]

hi.
I just got build success on my ubuntu machine as your build command.
Did you install prerequisites things[1] to build zeppelin?
And Can you share your build log?

[1]
https://github.com/apache/zeppelin

2016-06-21 23:47 GMT+09:00 Vinay Shukla <vinayshu...@gmail.com>:

> What is the exact build failure?
>
>
> On Tuesday, June 21, 2016, Darren Govoni <dar...@ontrenet.com> wrote:
>
>> Ubuntu 15.10
>>
>> mvn clean package -Pspark-1.6 -Phadoop-2.4 -Pyarn -Ppyspark
>>
>>
>> On 06/21/2016 09:03 AM, Hyung Sung Shim wrote:
>>
>> hi.
>> What is your build command and please tell me your environments.
>>
>> 2016-06-21 21:45 GMT+09:00 Darren Govoni <dar...@ontrenet.com>:
>>
>>> Hi
>>>
>>> I am trying to build git repo but zengine fails. Any tips on this?
>>>
>>> Thanks
>>>
>>>
>>>
>>> Sent from my Verizon Wireless 4G LTE smartphone
>>>
>>
>>
>>

Re: Build error zengine

[Hyung Sung Shim]

hi.
What is your build command and please tell me your environments.

2016-06-21 21:45 GMT+09:00 Darren Govoni :

> Hi
>
> I am trying to build git repo but zengine fails. Any tips on this?
>
> Thanks
>
>
>
> Sent from my Verizon Wireless 4G LTE smartphone
>

Re: Ask opinion regarding 0.6.0 release package

[Hyung Sung Shim]

I think making the 'zeppelin-bin-netinst' is greate idea and more make
sense than voting which interpreter should be included to zeppelin-bin-min.


2016-06-18 1:15 GMT+09:00 moon soo Lee :

> In case of no internet access, how about
>
> a. download 'zeppelin-bin-netinst' and run 'bin/install-interpreter.sh',
> and then copy the package to production env.
> b. download 'zeppelin-bin-all' and copy the package to production env.
>
> ?
>
> Thanks,
> moon
>
>
> On Fri, Jun 17, 2016 at 9:10 AM Mohit Jaggi  wrote:
>
>> Many production environments have no internet access. A script like  this
>> can be useful to some but it should not replace the proposed min binary.
>>
>> Sent from my iPhone
>>
>> On Jun 17, 2016, at 9:20 PM, moon soo Lee  wrote:
>>
>> Hi,
>>
>> Thanks for bringing this discussion.
>> it's great idea minimize binary package size.
>>
>> Can we set a policy to decide which interpreter goes to
>> 'zeppelin-bin-min', which is not?
>>
>> One alternative is, instead of making 'zeppelin-bin-min', we can make
>> 'zeppelin-bin-netinst'.
>> We can provide a shell script such as, 'bin/install-interpreter.sh' and
>> the script will download interpreters and their dependencies from maven
>> repository and store under /interpreter dir. By leveraging 
>> DependencyResolver[1],
>> i think we can make this feature in couple of hours.
>>
>> Only spark interpreter can not be installed in simple way, while it
>> requires some python and R packages under /interpreter dir and they're not
>> available on maven repository, so it'll need special treatment, but all
>> other interpreters can be installed in the simple way.
>>
>> Then, 'zeppelin-bin-netinst' version can have minimal package size, and
>> still gives easy way to install all the interpreters.
>> Also 'bin/install-interpreter.sh' will still useful even if we have
>> dynamic interpreter loading feature [2], to build offline package.
>>
>> what do you think?
>>
>> [1]
>> https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/dep/DependencyResolver.java
>> [2] https://issues.apache.org/jira/browse/ZEPPELIN-598
>>
>>
>> On Fri, Jun 17, 2016 at 1:02 AM mina lee  wrote:
>>
>>> Hi all!
>>>
>>> Zeppelin just started release process. Prior to creating release
>>> candidate I want to ask users' opinion about how you want it to be packaged.
>>>
>>> For the last release(0.5.6), we have released one binary package which
>>> includes all interpreters.
>>> The concern with providing one type of binary package is that package
>>> size will be quite big(~600MB).
>>> So I am planning to provide two binary packages:
>>>   - zeppelin-0.6.0-bin-all.tgz (includes all interpreters)
>>>   - zeppelin-0.6.0-bin-min.tgz (includes only most used interpreters)
>>>
>>> I am thinking about putting *spark(pyspark, sparkr, sql), python, jdbc,
>>> shell, markdown, angular* in minimized package.
>>> Could you give your opinion on whether these sets are enough, or some of
>>> them are ok to be excluded?
>>>
>>> Community's opinion will be helpful to make decision not only for 0.6.0
>>> but also for 0.7.0 release since we are planning to provide only minimized
>>> package from 0.7.0 release. From the 0.7.0 version, interpreters those are
>>> not included in binary package will be able to use dynamic interpreter
>>> feature [1] which is in progress under [2].
>>>
>>> Thanks,
>>> Mina
>>>
>>> [1]
>>> http://zeppelin.apache.org/docs/0.6.0-SNAPSHOT/manual/dynamicinterpreterload.html
>>> [2] https://github.com/apache/zeppelin/pull/908
>>>
>>