RE: Livy metrics

[Shanyu Zhao]

+1

I just created a JIRA to track this effort:
LIVY-743: Support Livy metrics based on dropwizard

Anhui, do you want to write a design doc for it? At lease you can attach your 
PR to the JIRA first.

Thanks
Shanyu


-Original Message-
From: Bikas Saha [mailto:bi...@apache.org] 
Sent: Saturday, January 11, 2020 5:39 AM
To: dev@livy.incubator.apache.org
Subject: [EXTERNAL] Re: Livy metrics

+1

Bikas


From: Marco Gaido 
Sent: Friday, January 10, 2020 2:35 AM
To: dev@livy.incubator.apache.org 
Subject: Re: Livy metrics

I think the best approach would be first preparing a design doc with your 
proposal so we can discuss on it and decide how to go about it.
You can go ahead with this if you want to contribute.

Thanks,
Marco
.

Il giorno ven 10 gen 2020 alle ore 03:00 安辉  ha scritto:

> In my company, We are maintaining a metrc version of livy, I am 
> willing to submit this patch, if you do not mind.
>
> -邮件原件-
> 发件人: Saisai Shao [mailto:sai.sai.s...@gmail.com]
> 发送时间: 2020年1月10日 9:39
> 收件人: dev@livy.incubator.apache.org
> 主题: Re: Livy metrics
>
> Currently we don't have concrete plan on this, it would be great if 
> you could contribute on this.
>
> Thanks
> Saisai
>
> Shanyu Zhao  于2020年1月10日周五 上午8:44写道：
>
> > Hi,
> >
> > Are there ongoing effort to add Livy metrics? I saw that dropwizard 
> > is added as dependency in pom.xml but didn't see it get used anywhere.
> >
> > Thanks,
> > Shanyu
> >
> 
> OPPO
>
>
> 本电子邮件及其附件含有OPPO公司的保密信息，仅限于邮件指明的收件人使用（包含个人及群组）。禁止任何人在未经授权的情况下以任何形式使用。如果
> 您错收了本邮件，请立即以电子邮件通知发件人并删除本邮件及其附件。
>
> This e-mail and its attachments contain confidential information from 
> OPPO, which is intended only for the person or entity whose address is 
> listed above. Any use of the information contained herein in any way 
> (including, but not limited to, total or partial disclosure, 
> reproduction, or dissemination) by persons other than the intended 
> recipient(s) is prohibited. If you receive this e-mail in error, 
> please notify the sender by phone or email immediately and delete it!
>

Re: .NET for Apache Spark interactive interpreter support

[Steve Suh]

@Saisai

This will not be a wrapper around the REST API.  The plan is to support a
Livy POST /sessions request where kind == "sparkdotnet".  Livy will
eventually push this request down to the ReplDriver, and in turn the
ReplDriver will create a new SparkDotnet Interpreter class.  This will be
similar to how the  PythonInterpreter, SparkRInterpreter, and
SQLInterpreter classes get instantiated and used.


Regards,
-Steve

On Sun, Jan 12, 2020 at 5:45 PM Saisai Shao  wrote:

> Is this just a wrapper of Livy REST API, or it could support Livy Job API?
>
> From my opinion, if it is just a wrapper of REST API, then it would be
> better to maintain out of Livy, since REST API is language independent, if
> we're going to have all the languages support in Livy, then it is hard to
> maintain.
>
> Just my two cents.
>
> Thanks
> Saisai
>
> Steve Suh  于2020年1月12日周日 下午3:49写道：
>
> > Hi,
> >
> > I contribute to the .NET for Apache Spark <
> https://github.com/dotnet/spark
> > >
> > project and we have seen *a lot* of *user interest* in providing first
> > class notebook support for *.NET for Apache Spark*.
> >
> > Livy currently supports *Scala*, *Python *and *R **interactive sessions*.
> > We have a working prototype
> > <
> >
> https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks
> > >
> > available
> > that adds support for a Spark *Dotnet **interactive session* and I would
> > like to know if the Livy community would be interested in adding this
> > feature to the main code base.  If so, please let me know and I can work
> on
> > creating this PR.
> >
> > For now, I've created a Jira item
> >  to track this.
> >
> >
> > Regards,
> > -Steve
> >
>

Re: Livy metrics

[Yiheng Wang]

+1

On Fri, Jan 10, 2020 at 10:00 AM 安辉  wrote:

> In my company, We are maintaining a metrc version of livy, I am willing to
> submit this patch, if you do not mind.
>
> -邮件原件-
> 发件人: Saisai Shao [mailto:sai.sai.s...@gmail.com]
> 发送时间: 2020年1月10日 9:39
> 收件人: dev@livy.incubator.apache.org
> 主题: Re: Livy metrics
>
> Currently we don't have concrete plan on this, it would be great if you
> could contribute on this.
>
> Thanks
> Saisai
>
> Shanyu Zhao  于2020年1月10日周五 上午8:44写道：
>
> > Hi,
> >
> > Are there ongoing effort to add Livy metrics? I saw that dropwizard is
> > added as dependency in pom.xml but didn't see it get used anywhere.
> >
> > Thanks,
> > Shanyu
> >
> 
> OPPO
>
>
> 本电子邮件及其附件含有OPPO公司的保密信息，仅限于邮件指明的收件人使用（包含个人及群组）。禁止任何人在未经授权的情况下以任何形式使用。如果您错收了本邮件，请立即以电子邮件通知发件人并删除本邮件及其附件。
>
> This e-mail and its attachments contain confidential information from
> OPPO, which is intended only for the person or entity whose address is
> listed above. Any use of the information contained herein in any way
> (including, but not limited to, total or partial disclosure, reproduction,
> or dissemination) by persons other than the intended recipient(s) is
> prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!
>

Re: .NET for Apache Spark interactive interpreter support

[Saisai Shao]

I see your point. Personally I don't have a strong opinion on this, I'm not
sure what others think about it. Why don't you start a design doc and call
for a vote about this feature.

Thanks
Saisai

Steve Suh  于2020年1月14日周二 上午10:12写道：

> @Saisai
>
> This will not be a wrapper around the REST API.  The plan is to support a
> Livy POST /sessions request where kind == "sparkdotnet".  Livy will
> eventually push this request down to the ReplDriver, and in turn the
> ReplDriver will create a new SparkDotnet Interpreter class.  This will be
> similar to how the  PythonInterpreter, SparkRInterpreter, and
> SQLInterpreter classes get instantiated and used.
>
>
> Regards,
> -Steve
>
> On Sun, Jan 12, 2020 at 5:45 PM Saisai Shao 
> wrote:
>
> > Is this just a wrapper of Livy REST API, or it could support Livy Job
> API?
> >
> > From my opinion, if it is just a wrapper of REST API, then it would be
> > better to maintain out of Livy, since REST API is language independent,
> if
> > we're going to have all the languages support in Livy, then it is hard to
> > maintain.
> >
> > Just my two cents.
> >
> > Thanks
> > Saisai
> >
> > Steve Suh  于2020年1月12日周日 下午3:49写道：
> >
> > > Hi,
> > >
> > > I contribute to the .NET for Apache Spark <
> > https://github.com/dotnet/spark
> > > >
> > > project and we have seen *a lot* of *user interest* in providing first
> > > class notebook support for *.NET for Apache Spark*.
> > >
> > > Livy currently supports *Scala*, *Python *and *R **interactive
> sessions*.
> > > We have a working prototype
> > > <
> > >
> >
> https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks
> > > >
> > > available
> > > that adds support for a Spark *Dotnet **interactive session* and I
> would
> > > like to know if the Livy community would be interested in adding this
> > > feature to the main code base.  If so, please let me know and I can
> work
> > on
> > > creating this PR.
> > >
> > > For now, I've created a Jira item
> > >  to track this.
> > >
> > >
> > > Regards,
> > > -Steve
> > >
> >
>

Re: [VOTE] Release Livy 0.7.0 based on RC4

[Jeff Zhang]

+1

Bikas Saha  于2020年1月11日周六 下午9:40写道：

> +1
>
> Bikas
> 
> From: mingchao zhao 
> Sent: Wednesday, January 8, 2020 6:16 PM
> To: dev@livy.incubator.apache.org 
> Subject: Re: [VOTE] Release Livy 0.7.0 based on RC4
>
> Tested the general functionality of apache-livy-0.7.0-incubat-bin.zip  and
> looks good.
>
> +1
>
> On Wed, Jan 8, 2020 at 8:07 PM Yiheng Wang  wrote:
>
> > Check the release files. Looks good.
> >
> > +1
> >
> > On Tue, Jan 7, 2020 at 9:25 PM Saisai Shao 
> wrote:
> >
> > > This vote is for releasing Livy 0.7.0 based on RC4.
> > >
> > > This RC mainly fixes the license issue.
> > >
> > > The vote will be open until Sunday  Jan 12, 23:59 UTC and
> > > will pass with a minimum of 3 +1 binding votes and a majority of
> positive
> > > votes.
> > >
> > > [+1] This release is ready to send to the Incubator PMC for approval
> > >
> > > [-1] This release is not ready because...
> > >
> > > This vote is held according to Apache Incubator release policy:
> > > https://incubator.apache.org/policy/incubation.html#releases
> > >
> > > The RC is based on tag v0.7.0-incubating-rc4:
> > >
> > >
> >
> https://github.com/apache/incubator-livy/commit/664503355d8bae763989ebac087122e306239d54
> > >
> > > The release files can be found here:
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc4/
> > >
> > > The staged maven artifacts can be found here:
> > > https://repository.apache.org/content/repositories/orgapachelivy-1013
> > >
> > > The list of resolved JIRAs in this release can be found here:
> > > https://issues.apache.org/jira/projects/LIVY/versions/12345179
> > >
> > > Thanks
> > > Saisai
> > >
> >
>


-- 
Best Regards

Jeff Zhang

Re: .NET for Apache Spark interactive interpreter support

[Jeff Zhang]

IIUC, donet spark is just a wrapper of spark. Besides adding
DotNetSparkInterpreter, what kind of other things user need to do ? For
now, user also need to specify SPARK_HOME at least, but to support donet,
what other things user need to do.


Saisai Shao  于2020年1月14日周二 上午11:01写道：

> I see your point. Personally I don't have a strong opinion on this, I'm not
> sure what others think about it. Why don't you start a design doc and call
> for a vote about this feature.
>
> Thanks
> Saisai
>
> Steve Suh  于2020年1月14日周二 上午10:12写道：
>
> > @Saisai
> >
> > This will not be a wrapper around the REST API.  The plan is to support a
> > Livy POST /sessions request where kind == "sparkdotnet".  Livy will
> > eventually push this request down to the ReplDriver, and in turn the
> > ReplDriver will create a new SparkDotnet Interpreter class.  This will be
> > similar to how the  PythonInterpreter, SparkRInterpreter, and
> > SQLInterpreter classes get instantiated and used.
> >
> >
> > Regards,
> > -Steve
> >
> > On Sun, Jan 12, 2020 at 5:45 PM Saisai Shao 
> > wrote:
> >
> > > Is this just a wrapper of Livy REST API, or it could support Livy Job
> > API?
> > >
> > > From my opinion, if it is just a wrapper of REST API, then it would be
> > > better to maintain out of Livy, since REST API is language independent,
> > if
> > > we're going to have all the languages support in Livy, then it is hard
> to
> > > maintain.
> > >
> > > Just my two cents.
> > >
> > > Thanks
> > > Saisai
> > >
> > > Steve Suh  于2020年1月12日周日 下午3:49写道：
> > >
> > > > Hi,
> > > >
> > > > I contribute to the .NET for Apache Spark <
> > > https://github.com/dotnet/spark
> > > > >
> > > > project and we have seen *a lot* of *user interest* in providing
> first
> > > > class notebook support for *.NET for Apache Spark*.
> > > >
> > > > Livy currently supports *Scala*, *Python *and *R **interactive
> > sessions*.
> > > > We have a working prototype
> > > > <
> > > >
> > >
> >
> https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks
> > > > >
> > > > available
> > > > that adds support for a Spark *Dotnet **interactive session* and I
> > would
> > > > like to know if the Livy community would be interested in adding this
> > > > feature to the main code base.  If so, please let me know and I can
> > work
> > > on
> > > > creating this PR.
> > > >
> > > > For now, I've created a Jira item
> > > >  to track this.
> > > >
> > > >
> > > > Regards,
> > > > -Steve
> > > >
> > >
> >
>


-- 
Best Regards

Jeff Zhang