Re: Next release

[Saisai Shao]

Hi,

I just merged the Spark 3 support yesterday, I would wait for most tests to
stabilize this feature. In the meanwhile, you can build the master branch
yourself to try this latest feature.

Thanks
Saisai

Aliaksandr Sasnouskikh  于2020年7月2日周四 下午10:50写道：

> Hi, thanks for notifying about the release, will check it today, shouldn’t
> be too complex. Will update you on the plans till the end of the week.
>
> чт, 2 июля 2020 г. в 14:41, murat migdisoglu :
>
> > Hello,
> >
> > When can we expect the next release for livy?
> >  Spark3.0 support (which has been merged today) is kind of critical for
> us.
> >
> >
> > Thank you
> >
> > --
> > "Talkers aren’t good doers. Rest assured that we’re going there to use
> our
> > hands, not our tongues."
> > W. Shakespeare
> >
> --
> Best regards,
> Aliaksandr Sasnouskikh
> https://github.com/jahstreet/
> https://www.linkedin.com/in/jahstreet/
> +31 (0) 6 51 29 58 39
>

Re: Review request

[Saisai Shao]

Sorry for the delay, I will take a review.

Thanks
Saisai

Shingo Furuyama  于2020年3月24日周二 下午12:45写道：

> Hello,
>
> I am Shingo Furuyama, working at Yahoo! JAPAN.
>
> We are trying to use livy 0.7.0-incubating and spark 2.4.5 with YARN
> HDP2.6.4 and have sent a few tiny patches related to the work.
> Can anyone review the patches?
>
> [LIVY-751]Livy server should allow to customize LIVY_CLASSPATH
> https://github.com/apache/incubator-livy/pull/282
>
> Add description of POST /batches
> https://github.com/apache/incubator-livy/pull/283
>
> modified the description of POST /sessions/{sessionId}/completion
> https://github.com/apache/incubator-livy/pull/285
>
> Regards,
> Shingo
>
>

Re: About building and releasing livy java/scala doc

[Saisai Shao]

Just figured out there's a doc about it. Thanks!

Best regards,
Saisai

Saisai Shao  于2020年2月2日周日 下午7:15写道：

> Hi Team,
>
> Do you know how to build and release java/scala doc for new Livy (0.7.0)
> release. I think document thing is the only remaining thing for 0.7.0
> release. Please help to guide, would be appreciated if there's doc about it.
>
> Thanks
> Saisai
>

About building and releasing livy java/scala doc

[Saisai Shao]

Hi Team,

Do you know how to build and release java/scala doc for new Livy (0.7.0)
release. I think document thing is the only remaining thing for 0.7.0
release. Please help to guide, would be appreciated if there's doc about it.

Thanks
Saisai

Re: python-api CI failure

[Saisai Shao]

Thanks for the fix. Would you please create a separate PR about this issue?
It should be enough to create a "minor" GH PR directly without JIRA issue.

Best regards,
Saisai

Wing Yew Poon  于2020年1月31日周五 上午6:33写道：

> I updated python-api/setup.py to include
>
> 'mock~=3.0.5',
>
> in requirements.
> That got the CI passing on my PR,
> https://github.com/apache/incubator-livy/pull/275.
>
> On Thu, Jan 30, 2020 at 12:02 PM Wing Yew Poon 
> wrote:
> >
> > Resending ...
> >
> > On Wed, Jan 29, 2020 at 9:26 PM Wing Yew Poon 
> wrote:
> > >
> > > Hi,
> > > I ran into a CI failure in the python-api unit tests. (See
> > > https://travis-ci.org/apache/incubator-livy/builds/643685980.)
> > > From what I can tell, something is pulling in mock 4.0.0b1 (in
> > > pre-release according to https://pypi.org/project/mock/#history)
> > > instead of 3.0.5, and a method signature has changed in mock.py.
> > > Any guidance on how I can get around this?
> > > Thanks,
> > > Wing Yew
> >
> > From https://pypi.org/project/mock/#history it appears that mock
> > 4.0.0b1 showed up Jan 29, 2020, which is why this failure only just
> > started happening.
>

Re: [VOTE] Release Livy 0.7.0 based on RC4

[Saisai Shao]

Thanks all,

We have 3 +1 (binding), 3 +1 (non-binding), vote is passed. I will start
another vote on incubating general.

Thanks
Saisai

Jeff Zhang  于2020年1月14日周二 下午12:07写道：

> +1
>
> Bikas Saha  于2020年1月11日周六 下午9:40写道：
>
> > +1
> >
> > Bikas
> > 
> > From: mingchao zhao 
> > Sent: Wednesday, January 8, 2020 6:16 PM
> > To: dev@livy.incubator.apache.org 
> > Subject: Re: [VOTE] Release Livy 0.7.0 based on RC4
> >
> > Tested the general functionality of apache-livy-0.7.0-incubat-bin.zip
> and
> > looks good.
> >
> > +1
> >
> > On Wed, Jan 8, 2020 at 8:07 PM Yiheng Wang  wrote:
> >
> > > Check the release files. Looks good.
> > >
> > > +1
> > >
> > > On Tue, Jan 7, 2020 at 9:25 PM Saisai Shao 
> > wrote:
> > >
> > > > This vote is for releasing Livy 0.7.0 based on RC4.
> > > >
> > > > This RC mainly fixes the license issue.
> > > >
> > > > The vote will be open until Sunday  Jan 12, 23:59 UTC and
> > > > will pass with a minimum of 3 +1 binding votes and a majority of
> > positive
> > > > votes.
> > > >
> > > > [+1] This release is ready to send to the Incubator PMC for approval
> > > >
> > > > [-1] This release is not ready because...
> > > >
> > > > This vote is held according to Apache Incubator release policy:
> > > > https://incubator.apache.org/policy/incubation.html#releases
> > > >
> > > > The RC is based on tag v0.7.0-incubating-rc4:
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-livy/commit/664503355d8bae763989ebac087122e306239d54
> > > >
> > > > The release files can be found here:
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc4/
> > > >
> > > > The staged maven artifacts can be found here:
> > > >
> https://repository.apache.org/content/repositories/orgapachelivy-1013
> > > >
> > > > The list of resolved JIRAs in this release can be found here:
> > > > https://issues.apache.org/jira/projects/LIVY/versions/12345179
> > > >
> > > > Thanks
> > > > Saisai
> > > >
> > >
> >
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: .NET for Apache Spark interactive interpreter support

[Saisai Shao]

I see your point. Personally I don't have a strong opinion on this, I'm not
sure what others think about it. Why don't you start a design doc and call
for a vote about this feature.

Thanks
Saisai

Steve Suh  于2020年1月14日周二 上午10:12写道：

> @Saisai
>
> This will not be a wrapper around the REST API.  The plan is to support a
> Livy POST /sessions request where kind == "sparkdotnet".  Livy will
> eventually push this request down to the ReplDriver, and in turn the
> ReplDriver will create a new SparkDotnet Interpreter class.  This will be
> similar to how the  PythonInterpreter, SparkRInterpreter, and
> SQLInterpreter classes get instantiated and used.
>
>
> Regards,
> -Steve
>
> On Sun, Jan 12, 2020 at 5:45 PM Saisai Shao 
> wrote:
>
> > Is this just a wrapper of Livy REST API, or it could support Livy Job
> API?
> >
> > From my opinion, if it is just a wrapper of REST API, then it would be
> > better to maintain out of Livy, since REST API is language independent,
> if
> > we're going to have all the languages support in Livy, then it is hard to
> > maintain.
> >
> > Just my two cents.
> >
> > Thanks
> > Saisai
> >
> > Steve Suh  于2020年1月12日周日 下午3:49写道：
> >
> > > Hi,
> > >
> > > I contribute to the .NET for Apache Spark <
> > https://github.com/dotnet/spark
> > > >
> > > project and we have seen *a lot* of *user interest* in providing first
> > > class notebook support for *.NET for Apache Spark*.
> > >
> > > Livy currently supports *Scala*, *Python *and *R **interactive
> sessions*.
> > > We have a working prototype
> > > <
> > >
> >
> https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks
> > > >
> > > available
> > > that adds support for a Spark *Dotnet **interactive session* and I
> would
> > > like to know if the Livy community would be interested in adding this
> > > feature to the main code base.  If so, please let me know and I can
> work
> > on
> > > creating this PR.
> > >
> > > For now, I've created a Jira item
> > > <https://issues.apache.org/jira/browse/LIVY-742> to track this.
> > >
> > >
> > > Regards,
> > > -Steve
> > >
> >
>

Re: .NET for Apache Spark interactive interpreter support

[Saisai Shao]

Is this just a wrapper of Livy REST API, or it could support Livy Job API?

>From my opinion, if it is just a wrapper of REST API, then it would be
better to maintain out of Livy, since REST API is language independent, if
we're going to have all the languages support in Livy, then it is hard to
maintain.

Just my two cents.

Thanks
Saisai

Steve Suh  于2020年1月12日周日 下午3:49写道：

> Hi,
>
> I contribute to the .NET for Apache Spark  >
> project and we have seen *a lot* of *user interest* in providing first
> class notebook support for *.NET for Apache Spark*.
>
> Livy currently supports *Scala*, *Python *and *R **interactive sessions*.
> We have a working prototype
> <
> https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks
> >
> available
> that adds support for a Spark *Dotnet **interactive session* and I would
> like to know if the Livy community would be interested in adding this
> feature to the main code base.  If so, please let me know and I can work on
> creating this PR.
>
> For now, I've created a Jira item
>  to track this.
>
>
> Regards,
> -Steve
>

Re: Livy metrics

[Saisai Shao]

Currently we don't have concrete plan on this, it would be great if you
could contribute on this.

Thanks
Saisai

Shanyu Zhao  于2020年1月10日周五 上午8:44写道：

> Hi,
>
> Are there ongoing effort to add Livy metrics? I saw that dropwizard is
> added as dependency in pom.xml but didn't see it get used anywhere.
>
> Thanks,
> Shanyu
>

Re: [VOTE] Release Livy 0.7.0 based on RC4

[Saisai Shao]

My +1 of course.

Saisai Shao  于2020年1月7日周二 下午9:25写道：

> This vote is for releasing Livy 0.7.0 based on RC4.
>
> This RC mainly fixes the license issue.
>
> The vote will be open until Sunday  Jan 12, 23:59 UTC and
> will pass with a minimum of 3 +1 binding votes and a majority of positive
> votes.
>
> [+1] This release is ready to send to the Incubator PMC for approval
>
> [-1] This release is not ready because...
>
> This vote is held according to Apache Incubator release policy:
> https://incubator.apache.org/policy/incubation.html#releases
>
> The RC is based on tag v0.7.0-incubating-rc4:
>
> https://github.com/apache/incubator-livy/commit/664503355d8bae763989ebac087122e306239d54
>
> The release files can be found here:
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc4/
>
> The staged maven artifacts can be found here:
> https://repository.apache.org/content/repositories/orgapachelivy-1013
>
> The list of resolved JIRAs in this release can be found here:
> https://issues.apache.org/jira/projects/LIVY/versions/12345179
>
> Thanks
> Saisai
>

[VOTE] Release Livy 0.7.0 based on RC4

[Saisai Shao]

This vote is for releasing Livy 0.7.0 based on RC4.

This RC mainly fixes the license issue.

The vote will be open until Sunday  Jan 12, 23:59 UTC and
will pass with a minimum of 3 +1 binding votes and a majority of positive
votes.

[+1] This release is ready to send to the Incubator PMC for approval

[-1] This release is not ready because...

This vote is held according to Apache Incubator release policy:
https://incubator.apache.org/policy/incubation.html#releases

The RC is based on tag v0.7.0-incubating-rc4:
https://github.com/apache/incubator-livy/commit/664503355d8bae763989ebac087122e306239d54

The release files can be found here:
https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc4/

The staged maven artifacts can be found here:
https://repository.apache.org/content/repositories/orgapachelivy-1013

The list of resolved JIRAs in this release can be found here:
https://issues.apache.org/jira/projects/LIVY/versions/12345179

Thanks
Saisai

Re: Fwd: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3

[Saisai Shao]

Thanks Alex for your check. I'm not sure if we have a proper place to add
this reference in the document. I'm fine with the current status, and will
add the comment during release vote.

Best regards,
Saisai

Alex Bozarth  于2020年1月7日周二 上午4:04写道：

> On the note of the "missing" licenses:
>
> Since I added those I did a bit of research to refresh my memory. Based on
> https://www.glyphicons.com/license/ we do not need to directly mention
> the licensing for those fonts. This is again backed up by the comment at
> the top of the bootstrap page
> https://getbootstrap.com/docs/3.3/components/ which is where our usage of
> the fonts/icons comes from. Based on these links we are fine as is but
> could also add a mention somewhere in our docs/licenses to the glyphicons
> website. Another reference would be this PR for the retired Apache project
> Falcon which also used these fonts
> https://issues.apache.org/jira/browse/FALCON-2110
>
> tl:dr we're fine as we are, but it might be kind to add a reference
> somewhere in our docs
>
>
> *Alex Bozarth*
> Software Engineer
> Center for Open-Source Data & AI Technologies
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Saisai Shao ---01/05/2020 10:22:40
> PM---Does anyone knows how to handle license issue for fonts like]Saisai
> Shao ---01/05/2020 10:22:40 PM---Does anyone knows how to handle license
> issue for fonts like below? Thanks
>
> From: Saisai Shao 
> To: dev@livy.incubator.apache.org
> Date: 01/05/2020 10:22 PM
> Subject: [EXTERNAL] Fwd: [VOTE] Release Apache Livy 0.7.0 (incubating)
> based on RC3
>
> --
>
>
>
> Does anyone knows how to handle license issue for fonts like below?
>
> Thanks
> Saisai
>
> -- Forwarded message -
> 发件人： Justin Mclean 
> Date: 2020年1月3日周五 上午9:59
> Subject: Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3
> To: 
>
>
> Hi,
>
> +1 (binding). There’s a license issue that needs to be fixed before the
> next release.
>
> I checked:
> - incubating in name
> - DISCLAIMER exists
> - LICENSE is missing a few things
> - NOTICE year needs updating (Is it still 2018?)
> - No unexpended binary files
> - all source files have ASF headers
> - can compile from source
>
> License is missing license for [1][2]
>
> Thanks,
> Justin
>
> 1.
>
> ./docs/assets/themes/apache/bootstrap/fonts/glyphicons-halflings-regular.svg
> 2.
>
> ./server/src/main/resources/org/apache/livy/server/ui/static/fonts/glyphicons-halflings-regular.*
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>
>
>

Fwd: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3

[Saisai Shao]

Does anyone knows how to handle license issue for fonts like below?

Thanks
Saisai

-- Forwarded message -
发件人： Justin Mclean 
Date: 2020年1月3日周五 上午9:59
Subject: Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3
To: 


Hi,

+1 (binding). There’s a license issue that needs to be fixed before the
next release.

I checked:
- incubating in name
- DISCLAIMER exists
- LICENSE is missing a few things
- NOTICE year needs updating (Is it still 2018?)
- No unexpended binary files
- all source files have ASF headers
- can compile from source

License is missing license for [1][2]

Thanks,
Justin

1.
./docs/assets/themes/apache/bootstrap/fonts/glyphicons-halflings-regular.svg
2.
./server/src/main/resources/org/apache/livy/server/ui/static/fonts/glyphicons-halflings-regular.*


-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: Podling Livy Report Reminder - January 2020

[Saisai Shao]

I will work on this.

Thanks
Saisai

Justin Mclean  于2020年1月1日周三 下午4:06写道：

> Hi,
> Just a friendly reminder the report is due today is anyone working on it?
> Thanks,
> Justin
>

Re: [VOTE] Release Livy 0.7.0 based on RC3

[Saisai Shao]

We have 3 binding +1, no 0s or -1s. Vote passes, I'll follow up on the
general@ list.

Thanks
Saisai

Bikas Saha  于2019年12月29日周日 上午11:25写道：

> +1
>
> Bikas
>
> 
> From: Aliaksandr Sasnouskikh 
> Sent: Monday, December 23, 2019 11:58 PM
> To: dev@livy.incubator.apache.org 
> Subject: Re: [VOTE] Release Livy 0.7.0 based on RC3
>
> +1
>
> вт, 24 дек. 2019 г. в 8:11, Saisai Shao :
>
> > The date is overdue, but we still didn't receive enough votes, let me
> > extend the due date to Dec 29, 23:59 UTC. Please help to vote.
> >
> > Best regards,
> > Saisai
> >
> > Saisai Shao  于2019年12月20日周五 下午4:02写道：
> >
> > > +1 myself.
> > >
> > > Yiheng Wang  于2019年12月19日周四 上午11:41写道：
> > >
> > >> Check the release files. Looks good.
> > >>
> > >> +1
> > >>
> > >> On Wed, Dec 18, 2019 at 5:25 PM runzhiwang 
> > >> wrote:
> > >>
> > >> > 
> > >> >   +1
> > >> >  
> > >> >  ** release files*
> > >> > I verify the apache-livy-0.7.0-incubating-bin.zip by creating
> session,
> > >> > statement, and everything is ok.
> > >> >  I also verify the sha512 and asc file. Both look good.
> > >> >
> > >> >   ** staged maven artifacts **
> > >> >  It's ok.
> > >> >  
> > >> >
> > >> >  ** resolved JIRAs **
> > >> >  It's ok. I check all the jiras in the commits.
> > >> >  
> > >> >
> > >> > Thanks
> > >> > RunzhiWang
> > >> >
> > >> >
> > >> >
> > >> >  --原始邮件--
> > >> >   发件人:"Saisai Shao" > >> >  发送时间:2019年12月18日(星期三) 中午1:58
> > >> >  收件人:"dev" > >> >
> > >> >  主题:[VOTE] Release Livy 0.7.0 based on RC3
> > >> >
> > >> >
> > >> >
> > >> > This vote is for releasing Livy 0.7.0 based on RC3.
> > >> >
> > >> > The vote will be open until Friday Dec 23, 23:59 UTC and
> > >> > will pass with a minimum of 3 +1 binding votes and a majority of
> > >> positive
> > >> > votes.
> > >> >
> > >> > [+1] This release is ready to send to the Incubator PMC for approval
> > >> >
> > >> > [-1] This release is not ready because...
> > >> >
> > >> > This vote is held according to Apache Incubator release policy:
> > >> > https://incubator.apache.org/policy/incubation.html#releases
> > >> >
> > >> > The RC is based on tag v0.7.0-incubating-rc3:
> > >> >
> > >> >
> > >>
> >
> https://github.com/apache/incubator-livy/commit/0cc431a7e9236e313f9904ef121cad5df444d4da
> > >> >
> > >> > The release files can be found here:
> > >> >
> > >>
> >
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc3/
> > >> >
> > >> > The staged maven artifacts can be found here:
> > >> >
> https://repository.apache.org/content/repositories/orgapachelivy-1012
> > >> >
> > >> > The list of resolved JIRAs in this release can be found here:
> > >> > https://issues.apache.org/jira/projects/LIVY/versions/12345179
> > >> >
> > >> > Thanks
> > >> > Saisai
> > >>
> > >
> >
> --
> Отправлено с мобильного устройства
>

Re: [VOTE] Release Livy 0.7.0 based on RC3

[Saisai Shao]

The date is overdue, but we still didn't receive enough votes, let me
extend the due date to Dec 29, 23:59 UTC. Please help to vote.

Best regards,
Saisai

Saisai Shao  于2019年12月20日周五 下午4:02写道：

> +1 myself.
>
> Yiheng Wang  于2019年12月19日周四 上午11:41写道：
>
>> Check the release files. Looks good.
>>
>> +1
>>
>> On Wed, Dec 18, 2019 at 5:25 PM runzhiwang 
>> wrote:
>>
>> > 
>> >   +1
>> >  
>> >  ** release files*
>> > I verify the apache-livy-0.7.0-incubating-bin.zip by creating session,
>> > statement, and everything is ok.
>> >  I also verify the sha512 and asc file. Both look good.
>> >
>> >   ** staged maven artifacts **
>> >  It's ok.
>> >  
>> >
>> >  ** resolved JIRAs **
>> >  It's ok. I check all the jiras in the commits.
>> >  
>> >
>> > Thanks
>> > RunzhiWang
>> >
>> >
>> >
>> >  --原始邮件--
>> >   发件人:"Saisai Shao"> >  发送时间:2019年12月18日(星期三) 中午1:58
>> >  收件人:"dev"> >
>> >  主题:[VOTE] Release Livy 0.7.0 based on RC3
>> >
>> >
>> >
>> > This vote is for releasing Livy 0.7.0 based on RC3.
>> >
>> > The vote will be open until Friday Dec 23, 23:59 UTC and
>> > will pass with a minimum of 3 +1 binding votes and a majority of
>> positive
>> > votes.
>> >
>> > [+1] This release is ready to send to the Incubator PMC for approval
>> >
>> > [-1] This release is not ready because...
>> >
>> > This vote is held according to Apache Incubator release policy:
>> > https://incubator.apache.org/policy/incubation.html#releases
>> >
>> > The RC is based on tag v0.7.0-incubating-rc3:
>> >
>> >
>> https://github.com/apache/incubator-livy/commit/0cc431a7e9236e313f9904ef121cad5df444d4da
>> >
>> > The release files can be found here:
>> >
>> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc3/
>> >
>> > The staged maven artifacts can be found here:
>> > https://repository.apache.org/content/repositories/orgapachelivy-1012
>> >
>> > The list of resolved JIRAs in this release can be found here:
>> > https://issues.apache.org/jira/projects/LIVY/versions/12345179
>> >
>> > Thanks
>> > Saisai
>>
>

Re: [VOTE] Release Livy 0.7.0 based on RC3

[Saisai Shao]

+1 myself.

Yiheng Wang  于2019年12月19日周四 上午11:41写道：

> Check the release files. Looks good.
>
> +1
>
> On Wed, Dec 18, 2019 at 5:25 PM runzhiwang 
> wrote:
>
> > 
> >   +1
> >  
> >  ** release files*
> > I verify the apache-livy-0.7.0-incubating-bin.zip by creating session,
> > statement, and everything is ok.
> >  I also verify the sha512 and asc file. Both look good.
> >
> >   ** staged maven artifacts **
> >  It's ok.
> >  
> >
> >  ** resolved JIRAs **
> >  It's ok. I check all the jiras in the commits.
> >  
> >
> > Thanks
> > RunzhiWang
> >
> >
> >
> >  --原始邮件--
> >   发件人:"Saisai Shao" >  发送时间:2019年12月18日(星期三) 中午1:58
> >  收件人:"dev" >
> >  主题:[VOTE] Release Livy 0.7.0 based on RC3
> >
> >
> >
> > This vote is for releasing Livy 0.7.0 based on RC3.
> >
> > The vote will be open until Friday Dec 23, 23:59 UTC and
> > will pass with a minimum of 3 +1 binding votes and a majority of positive
> > votes.
> >
> > [+1] This release is ready to send to the Incubator PMC for approval
> >
> > [-1] This release is not ready because...
> >
> > This vote is held according to Apache Incubator release policy:
> > https://incubator.apache.org/policy/incubation.html#releases
> >
> > The RC is based on tag v0.7.0-incubating-rc3:
> >
> >
> https://github.com/apache/incubator-livy/commit/0cc431a7e9236e313f9904ef121cad5df444d4da
> >
> > The release files can be found here:
> >
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc3/
> >
> > The staged maven artifacts can be found here:
> > https://repository.apache.org/content/repositories/orgapachelivy-1012
> >
> > The list of resolved JIRAs in this release can be found here:
> > https://issues.apache.org/jira/projects/LIVY/versions/12345179
> >
> > Thanks
> > Saisai
>

Re: [VOTE] Release Livy 0.7.0 based on RC2

[Saisai Shao]

Thanks guys,

I didn't add thrift profile into building, so the thrift related part are
missing. For 2.10 related poms/jars, because we removed the support of
2.10, so they're not existed anymore, also for minicluster related profile.

So this vote is failed, and I will start a new build with thrift included.

Thanks
Saisai


Yiheng Wang  于2019年12月16日周一 下午8:32写道：

> Hi
>
> ** maven staging artifacts*
> Compare the staging maven artifacts with 0.6.0, looks like some folders are
> missing:
> livy-beeline/
> livy-core_2.10/
> livy-repl_2.10/
> livy-scala-api_2.10/
> livy-thriftserver/
> livy-thriftserver-session/
> minicluster-dependencies-parent/
> minicluster-dependencies_2.10/
> minicluster-dependencies_2.11/
>
> ** release files*
> I verify the sha512 and asc file. Both look good.
>
> Thanks
> Yiheng
>
> On Mon, Dec 16, 2019 at 4:07 PM Saisai Shao 
> wrote:
>
> > This vote is for releasing Livy 0.7.0 based on RC2 (RC1 has one
> unnecessary
> > commit due to my bad).
> >
> > The vote will be open until Friday  Dec 20, 23:59 UTC and
> > will pass with a minimum of 3 +1 binding votes and a majority of positive
> > votes.
> >
> > [+1] This release is ready to send to the Incubator PMC for approval
> >
> > [-1] This release is not ready because...
> >
> > This vote is held according to Apache Incubator release policy:
> > https://incubator.apache.org/policy/incubation.html#releases
> >
> > The RC is based on tag v0.7.0-incubating-rc2:
> >
> >
> https://github.com/apache/incubator-livy/commit/eebb6ecfe5ec021244c69096fa240e84f933b2ca
> >
> > The release files can be found here:
> >
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc2/
> >
> > The staged maven artifacts can be found here:
> > https://repository.apache.org/content/repositories/orgapachelivy-1009
> >
> > The list of resolved JIRAs in this release can be found here:
> > https://issues.apache.org/jira/projects/LIVY/versions/12345179
> >
> > Thanks
> > Saisai
> >
>

Re: [Vote][LIVY-718] Support multi-active high availability in Livy

[Saisai Shao]

+1

rei sivan  于2019年12月12日周四 下午2:09写道：

> +1
>
> On Thu, Dec 12, 2019, 05:30 Yiheng Wang  wrote:
>
> > Dear Community
> >
> > I'd like to call a vote on LIVY-718
> >  "Support
> > multi-active high availability in Livy".
> >
> > Currently, Livy only supports single node recovery. This is not
> sufficient
> > in some production environments. In our scenario, the Livy server serves
> > many notebook and JDBC services. We want to make Livy service more
> > fault-tolerant and scalable.
> >
> > There're already some proposals in the community for high availability.
> But
> > they're not so complete or just for active-standby high availability. So
> we
> > propose a multi-active high availability design to achieve the following
> > goals:
> >
> >- One or more servers will serve the client requests at the same time.
> >- Sessions are allocated among different servers.
> >- When one node crashes, the affected sessions will be moved to other
> >active services.
> >
> > Please find the design doc here
> > <
> >
> https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing
> > >,
> > which has been reviewed for several weeks.
> >
> > This vote is open until next Wednesday (Dec. 18).
> >
> > [] +1: Accept the proposal
> > [] +0
> > [] -1: I don't think this is a good idea because ...
> >
> > Thank you
> >
> > Yiheng
> >
>

Re: Support multi-active high availability in Livy

[Saisai Shao]

As I remembered we don't have such requirement in Livy community, but I'm
OK to start a vote. Yiheng, would you please start a vote?

Thanks
Saisai

Marco Gaido  于2019年12月12日周四 上午2:50写道：

> Shall we start an official vote before creating the subtasks?
>
> Thanks,
> Marco
>
> Il giorno mer 11 dic 2019 alle ore 14:50 Saisai Shao <
> sai.sai.s...@gmail.com>
> ha scritto:
>
> > I think it is feasible to move to next step. Let's create subtasks to
> track
> > all the works.
> >
> > Thanks
> > Saisai
> >
> > Yiheng Wang  于2019年12月10日周二 下午3:18写道：
> >
> > > Hi Community
> > >
> > > We have received some feedback for this design. Here're the open
> > questions
> > > for discussion
> > > 1. Session allocation algorithm
> > > We agree to make it pluggable. There're two allocation algorithms:
> > > server-session mapping and consistent hashing. We haven't agreed on
> which
> > > one is the default.
> > > 2. The semantic change of getAllSessions
> > > We agree to not change its semantics. However, we haven't achieved
> > > an agreement on the implementation.
> > >
> > > We think the open questions are not a blocker. So can we have this
> design
> > > doc approved?
> > >
> > > Thanks
> > > Yiheng
> > >
> > > On Thu, Nov 28, 2019 at 6:37 PM Yiheng Wang  wrote:
> > >
> > > > Hi Community
> > > >
> > > > We propose a multi-active high availability design for Livy. We think
> > > this
> > > > feature is valuable for many users. Here're the JIRA and the design
> > doc.
> > > If
> > > > you're interested, please take a look and comment :-)
> > > >
> > > > JIRA:
> > > > https://issues.apache.org/jira/projects/LIVY/issues/LIVY-718
> > > >
> > > > Design doc:
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing
> > > >
> > > > Best regards
> > > > Yiheng
> > > >
> > >
> >
>

Re: Support multi-active high availability in Livy

[Saisai Shao]

I think it is feasible to move to next step. Let's create subtasks to track
all the works.

Thanks
Saisai

Yiheng Wang  于2019年12月10日周二 下午3:18写道：

> Hi Community
>
> We have received some feedback for this design. Here're the open questions
> for discussion
> 1. Session allocation algorithm
> We agree to make it pluggable. There're two allocation algorithms:
> server-session mapping and consistent hashing. We haven't agreed on which
> one is the default.
> 2. The semantic change of getAllSessions
> We agree to not change its semantics. However, we haven't achieved
> an agreement on the implementation.
>
> We think the open questions are not a blocker. So can we have this design
> doc approved?
>
> Thanks
> Yiheng
>
> On Thu, Nov 28, 2019 at 6:37 PM Yiheng Wang  wrote:
>
> > Hi Community
> >
> > We propose a multi-active high availability design for Livy. We think
> this
> > feature is valuable for many users. Here're the JIRA and the design doc.
> If
> > you're interested, please take a look and comment :-)
> >
> > JIRA:
> > https://issues.apache.org/jira/projects/LIVY/issues/LIVY-718
> >
> > Design doc:
> >
> >
> https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing
> >
> > Best regards
> > Yiheng
> >
>

Re: Plan to cut new branch and prepare to release 0.7

[Saisai Shao]

Hi all,

I've cut the branch-0.7
https://github.com/apache/incubator-livy/tree/branch-0.7 and prepare for
the release. Now master branch is bumped to 0.8.0-incubating-SNAPSHOT.

Best regards,
Saisai

yihengw  于2019年10月11日周五 上午10:31写道：

> Yes, the binary mode has this problem.
>
>
> In http mode, jdbc clients talk to thrift server not using a long
> connection. It should be fit the recovery.
>
>
>  原始邮件
> 发件人: Marco Gaido
> 收件人: dev
> 发送时间: 2019年10月10日(周四) 21:59
> 主题: Re: Plan to cut new branch and prepare to release 0.7
>
>
> I think it is harder to do that for the thriftserver case. By its nature,
> the REST API allows multiple single requests, while in the thrifserver case
> the connection is always the same. I think without automatic failover, that
> feature is not so critical. Il giorno gio 10 ott 2019 alle ore 11:12
> yihengw  ha scritto: > Regarding the thrift server GA,
> should we consider supporting recovery in > thrift server? I open a JIRA
> for it > https://issues.apache.org/jira/browse/LIVY-696 > > > welcome for
> discussion. > > > 原始邮件 > 发件人: Saisai Shao > 收件人:
> dev > 发送时间: 2019年10月10日(周四) 16:24 > 主题:
> Re: Plan to cut new branch and prepare to release 0.7 > > > I'm hesitant to
> include service discovery into 0.7.0. Because all of the > JIRAs only
> addressed a small piece of high availability, but lack a whole > picture of
> HA design, I would suggest to have a whole HA design, then > splitting into
> small pieces, rather than opposite. So currently I'm not > confident if the
> JIRAs are good enough to merge. Thanks Saisai Marco Gaido < >
> marcogaid...@gmail.com> 于2019年10月10日周四 下午4:19写道： > Hi Saisai, > > thanks
> > for starting this discussion. I agree starting talking about a 0.7.0 > >
> release. Just a couple of notes: > > [LIVY-692][THRIFT] Use configured >
> principal for Kerberos authentication: as > I mentioned in the PR, I don't
> > think this is an actual problem. > > What about including in the scope of
> > this release also service discovery? > I've seen a couple of PRs and it
> may > be worth to have it in this release. > Any thoughts on this? > >
> Thanks, > > Marco > > Il giorno gio 10 ott 2019 alle ore 06:08 Jeff Zhang <
> > zjf...@gmail.com> ha > scritto: > > > Make sense to me > > > > Saisai >
> Shao  于2019年10月10日周四 下午12:04写道： > > > > > Hi all,
> > > > > > > > I'm planning to cut a new branch and prepare for the 0.7 >
> release, which > > did > > > lots of improvements to thrift module, I think
> > we could make thrift > > module > > > GA as for this release. > > > > > >
> > Currently I'm waiting for this JIRAs to be merged before cutting the >
> new > > > > branch. > > > > > > [LIVY-356][SERVER]Add LDAP authentication
> for > livy-server. > > > [LIVY-678] Thrift ldap authentication, based on
> ldapurl, > basedn, domain > > > [LIVY-692][THRIFT] Use configured principal
> for > Kerberos authentication > > > [LIVY-689] Deliver stage process
> message to > the end user using > > thriftserver > > > > > > Any thought? >
> > > > > > > Thanks > > > Saisai > > > > > > > > > -- > > Best Regards > > >
> > Jeff > Zhang > > >

Re: Plan to cut new branch and prepare to release 0.7

[Saisai Shao]

I'm hesitant to include service discovery into 0.7.0. Because all of the
JIRAs only addressed a small piece of high availability, but lack a whole
picture of HA design, I would suggest to have a whole HA design, then
splitting into small pieces, rather than opposite. So currently I'm not
confident if the JIRAs are good enough to merge.

Thanks
Saisai

Marco Gaido  于2019年10月10日周四 下午4:19写道：

> Hi Saisai,
>
> thanks for starting this discussion. I agree starting talking about a 0.7.0
> release. Just a couple of notes:
>
> [LIVY-692][THRIFT] Use configured principal for Kerberos authentication: as
> I mentioned in the PR, I don't think this is an actual problem.
>
> What about including in the scope of this release also service discovery?
> I've seen a couple of PRs and it may be worth to have it in this release.
> Any thoughts on this?
>
> Thanks,
> Marco
>
> Il giorno gio 10 ott 2019 alle ore 06:08 Jeff Zhang  ha
> scritto:
>
> > Make sense to me
> >
> > Saisai Shao  于2019年10月10日周四 下午12:04写道：
> >
> > > Hi all,
> > >
> > > I'm planning to cut a new branch and prepare for the 0.7 release, which
> > did
> > > lots of improvements to thrift module, I think we could make thrift
> > module
> > > GA as for this release.
> > >
> > > Currently I'm waiting for this JIRAs to be merged before cutting the
> new
> > > branch.
> > >
> > > [LIVY-356][SERVER]Add LDAP authentication for livy-server.
> > > [LIVY-678] Thrift ldap authentication, based on ldapurl, basedn, domain
> > > [LIVY-692][THRIFT] Use configured principal for Kerberos authentication
> > > [LIVY-689] Deliver stage process message to the end user using
> > thriftserver
> > >
> > > Any thought?
> > >
> > > Thanks
> > > Saisai
> > >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>

Plan to cut new branch and prepare to release 0.7

[Saisai Shao]

Hi all,

I'm planning to cut a new branch and prepare for the 0.7 release, which did
lots of improvements to thrift module, I think we could make thrift module
GA as for this release.

Currently I'm waiting for this JIRAs to be merged before cutting the new
branch.

[LIVY-356][SERVER]Add LDAP authentication for livy-server.
[LIVY-678] Thrift ldap authentication, based on ldapurl, basedn, domain
[LIVY-692][THRIFT] Use configured principal for Kerberos authentication
[LIVY-689] Deliver stage process message to the end user using thriftserver

Any thought?

Thanks
Saisai

Re: Possible missing mentor(s)

[Saisai Shao]

First, we're scaling the PR review, but we only have few active committers,
so the merge may not be fast.
Second, if someone has a *good* and *large* contribution history, and
actively participates in community, we will add him without doubt.
Third, 2-year old open PRs doesn't stand anything, some reviewers left the
community and PRs get staled, it is quite common, especially in large
community.

Sid Anand  于2019年9月1日周日 下午4:46写道：

> Apache projects promote contributors to committers based on contributions
> made, not on an expectation of future activity. That's the Apache way per
> my understanding. Over time, folks become inactive and busy -- life
> happens, I get it.  May I ask what are you folks doing to scale PR review
> and merging? Are you adding new committers?  Do you feel that 2-year old
> open PRs is where you wish to be and is the right way to grow a community?
>
> On Sun, Sep 1, 2019 at 1:46 AM Sid Anand  wrote:
>
> > Apache projects promote contributors to committers based on contributions
> > made, not on an expectation of future activity. That's the Apache way per
> > my understanding. Over time, folks become inactive and busy -- life
> > happens, I get it.  May I ask what are you folks doing to scale PR review
> > and merging? Are you adding new committers?  Do you feel that 2-year old
> > open PRs is where you wish to be and is the right way to grow a
> community?
> >
> > -s
> >
> > On Sun, Sep 1, 2019 at 12:59 AM Saisai Shao 
> > wrote:
> >
> >> It's unfair to say there's underlying bias. Livy project is a small
> >> project, the contributor diversity may not be as rich as popular project
> >> like Spark, it is not fair to say that the contributions only limits to
> >> someones, so project is biased. There're many small Apache project which
> >> has only few contributors, can we say those projects are biased? Also
> for
> >> years the committers have joined and left the community, it is hard to
> >> track every contribution in time, as we're not a full-time Livy open
> >> source
> >> contributors. I also have several PRs left unreviewed for years. It's
> >> quite
> >> common even for large project like Spark, Hadoop, there're so many
> >> un-merged PRs left for several years. It's unfair to say the project is
> >> biased, unhealthy because of some un-merged PRs.
> >>
> >> The community is small but free and open, I would deny that the
> community
> >> is unhealthy especially biased, this is an irresponsible and subjective
> >> word.
> >>
> >> Sid Anand  于2019年9月1日周日 上午4:20写道：
> >>
> >> > Folks!
> >> > We've (several devs, myself included) contacted the livy dev list and
> >> the
> >> > owners DL several times. Our PRs stagnated over a few years. Livy is a
> >> > central component in PayPal's Data Infra (Our data footprint is 80+
> PB).
> >> > The project seems pretty unhealthy. After a few years, this dev moved
> on
> >> > and the state of our PR may be harder to define, with both absentee
> >> > mentors/PMC/committers and PR author.
> >> >
> >> > I see only a narrow band of contributors being merged. I hope there is
> >> no
> >> > underlying bias, given that this would not be the Apache way. As
> >> mentors, I
> >> > hope the goal is to watch for such bias and eliminate it to promote
> >> > community health, engagement, and integrity.
> >> >
> >> > -s
> >> >
> >> > On Fri, Aug 30, 2019 at 10:22 PM Jean-Baptiste Onofré <
> j...@nanthrax.net>
> >> > wrote:
> >> >
> >> > > Hi Justin,
> >> > >
> >> > > Like Luciano, I'm also around. I've proposed couple of new features
> >> that
> >> > > I plan to work on and I try to review/verify the releases.
> >> > >
> >> > > Regards
> >> > > JB
> >> > >
> >> > > On 21/10/2018 01:50, Justin Mclean wrote:
> >> > > > Hi,
> >> > > >
> >> > > > We have discussed missing mentors on the incubator general list,
> >> > > identified those who may be missing and over a couple of months
> tried
> >> to
> >> > > contact your mentor in several ways but have got no reply so they
> have
> >> > been
> >> > > removed from the roster.
> >> > > >
> >> > > > If this is error, and your mentor is actually active, please
> >> contact me
> >> > > and I'll add them back to the roster.
> >> > > >
> >> > > > The IPMC will also do what it can to find extra mentors for those
> >> > > podling that need them.
> >> > > >
> >> > > > Thanks,
> >> > > > Justin
> >> > > > (V.P. Incubator)
> >> > > >
> >> > >
> >> > > --
> >> > > Jean-Baptiste Onofré
> >> > > jbono...@apache.org
> >> > > http://blog.nanthrax.net
> >> > > Talend - http://www.talend.com
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] Getting rid of old stuff

[Saisai Shao]

+1，

To support J8, Spark 2.2+ might be the only option. I'm not sure if those
vendors will continue to support Spark 2.2-, but IMHO for the community
release, I think moving forward would be a better choice.

Thanks
Saisai


Jeff Zhang  于2018年9月14日周五 上午9:39写道：

> +1
>
>
> Alex Bozarth 于2018年9月14日周五 上午6:26写道：
>
> > I agree with all of Marcelo's points. The last time we discussed this was
> > when Spark 2.2 was new and it was decided that it was probably too soon,
> > but that was awhile ago now. I've been in support of deprecating and
> > removing support for older versions of Java/Scala/Spark for a while and I
> > believe it will allow us to clean up and unify large portions of our
> code.
> >
> >
> > *Alex Bozarth*
> > Software Engineer
> > Center for Open-Source Data & AI Technologies
> > --
> > *E-mail:* *ajboz...@us.ibm.com* 
> > *GitHub: **github.com/ajbozarth* 
> >
> >
> > 505 Howard Street
> > <
> https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g
> >
> > San Francisco, CA 94105
> > <
> https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g
> >
> > United States
> > <
> https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g
> >
> >
> >
> >
> > [image: Inactive hide details for Marcelo Vanzin ---09/13/2018 03:10:28
> > PM---Hey all, I'd like to gauge people's reaction to some propo]Marcelo
> > Vanzin ---09/13/2018 03:10:28 PM---Hey all, I'd like to gauge people's
> > reaction to some proposals regarding what
> >
> > From: Marcelo Vanzin 
> > To: dev@livy.incubator.apache.org
> > Date: 09/13/2018 03:10 PM
> > Subject: [DISCUSS] Getting rid of old stuff
> > --
> >
> >
> >
> >
> > Hey all,
> >
> > I'd like to gauge people's reaction to some proposals regarding what
> > is supported in Livy.
> >
> > #1: Java versions
> >
> > I propose dropping support for Java 7. Even J8 is already EOL,
> > although it's pretty obvious nobody is getting rid of it anytime soon.
> > But I don't see a good reason to support J7. Even testing it is a
> > nuisance, since most people only have jdk8 around...
> >
> > #2: Spark versions
> >
> > I think we should drop 1.6. At least. This would clean up some code
> > that currently uses reflection, and fix some parts of the API (like
> > the JobContext method to retrieve a "SparkSession" instance).
> >
> > Changing the API is well, not backwards compatible, but I think it's
> > better to do that sort of cleanup earlier than later when the project
> > is more mature.
> >
> > Separately, we could consider dropping 2.0 and 2.1 also. There was
> > talk in the Spark list about making 2.1 EOL - I don't remember a final
> > verdict, but I don't imagine there will be a lot of new deployments of
> > 2.1 going forward, since the only reason to use it is if you're stuck
> > with J7.
> >
> > #3: Scala versions
> >
> > If we decide to only support Spark 2.2+, then the decision is easy.
> > But if we drop 1.6, we should consider dropping Scala 2.10 support.
> > Spark does not ship official builds with 2.10 support in the 2.x line,
> > and it was dropped altogether in 2.2.
> >
> > We shouldn't remove support for multiple versions of Scala, though,
> > since 2.12 will be beta in Spark 2.4, and there will be a 2.13 at some
> > point.
> >
> > Thoughts?
> >
> >
> > --
> > Marcelo
> >
> >
> >
> >
> >
>

Re: Podling report this month

[Saisai Shao]

Updated. Please review and sign off.

Thanks

Saisai Shao  于2018年7月9日周一 上午8:27写道：

> Sorry I forgot about this, will draft this today.
>
> Thanks
> Saisai
>
> Jean-Baptiste Onofré  于2018年7月8日周日 下午1:17写道：
>
>> Hi guys,
>>
>> we have to report this month:
>>
>> https://wiki.apache.org/incubator/July2018
>>
>> Did someone already start a report ? I can bootstrap one if you want.
>>
>> Thanks,
>> Regards
>> JB
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>

Re: Podling report this month

[Saisai Shao]

Sorry I forgot about this, will draft this today.

Thanks
Saisai

Jean-Baptiste Onofré  于2018年7月8日周日 下午1:17写道：

> Hi guys,
>
> we have to report this month:
>
> https://wiki.apache.org/incubator/July2018
>
> Did someone already start a report ? I can bootstrap one if you want.
>
> Thanks,
> Regards
> JB
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: A new link request to my project and one question

[Saisai Shao]

I think Livy super user is similar to Hadoop's proxy user, it allows this
user to impersonate others, but it doesn't check whether other users is
allowed to be impersonated.

In the meantime, Livy has ACL mechanisms, which allows only ACL verified
users to connect to LivyServer, so I think with ACL, we can do a more
fine-grained control.

For other missing point, I think we can improve the Livy code.

Marcelo Vanzin  于2018年6月26日周二 上午9:53写道：

> Superusers are a little more than "allowed to impersonate others". I
> don't remember exactly what are the things that it allows, but it
> would be better to add finer grained permissions.
>
> On Mon, Jun 25, 2018 at 6:30 PM, Saisai Shao 
> wrote:
> > Yes, has a configuration "livy.superusers". Here in this case, the sql
> > server user should be added as a superuser, who can impersonate other
> > different users.
> >
> > Marcelo Vanzin  于2018年6月26日周二 上午9:12写道：
> >
> >> You're talking about another service between the user and the
> application.
> >>
> >> In that case a parameter probably makes sense. But then you'd need to
> >> add those config options, because this is a dangerous feature, and
> >> Livy should know who is allowed to impersonate who. In this case the
> >> service needs to authenticate to Livy as a privileged user, and Livy's
> >> configuration would say that the service's user is allowed to
> >> impersonate certain users or groups (same as the other services that
> >> allow impersonation like YARN).
> >>
> >>
> >> On Mon, Jun 25, 2018 at 5:41 PM, Takeshi Yamamuro <
> linguin@gmail.com>
> >> wrote:
> >> > Yea, I know the Livy supports impersonation.
> >> > I assume a case blow
> >> > [different users] ---Some protocols---> [the server applications
> managing
> >> > multiple sessions for users] ---REST---> [Livy server]
> >> > In this case, Livy already has a way to pass proxyUser from the
> >> application
> >> > to Livy?
> >> > Sorry, but I'm not familiar with Livy internal logic.
> >> >
> >> >
> >> > On Tue, Jun 26, 2018 at 9:14 AM Marcelo Vanzin
> >> 
> >> > wrote:
> >> >
> >> >> On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro <
> >> linguin@gmail.com>
> >> >> wrote:
> >> >> > In that case, I think Livy is useful; the application can pass
> >> proxyUser
> >> >> to
> >> >> > build LivyClient for each user
> >> >> > and run spark queries as each user authorization.
> >> >>
> >> >> But Livy already supports impersonation. It can impersonate the
> >> >> authenticated user.
> >> >>
> >> >> You're suggesting adding a parameter so the user can request
> >> >> impersonation of some specific user, which is a different thing. What
> >> >> is the use case for that?
> >> >>
> >> >> --
> >> >> Marcelo
> >> >>
> >> >
> >> >
> >> > --
> >> > ---
> >> > Takeshi Yamamuro
> >>
> >>
> >>
> >> --
> >> Marcelo
> >>
>
>
>
> --
> Marcelo
>

Re: A new link request to my project and one question

[Saisai Shao]

Yes, has a configuration "livy.superusers". Here in this case, the sql
server user should be added as a superuser, who can impersonate other
different users.

Marcelo Vanzin  于2018年6月26日周二 上午9:12写道：

> You're talking about another service between the user and the application.
>
> In that case a parameter probably makes sense. But then you'd need to
> add those config options, because this is a dangerous feature, and
> Livy should know who is allowed to impersonate who. In this case the
> service needs to authenticate to Livy as a privileged user, and Livy's
> configuration would say that the service's user is allowed to
> impersonate certain users or groups (same as the other services that
> allow impersonation like YARN).
>
>
> On Mon, Jun 25, 2018 at 5:41 PM, Takeshi Yamamuro 
> wrote:
> > Yea, I know the Livy supports impersonation.
> > I assume a case blow
> > [different users] ---Some protocols---> [the server applications managing
> > multiple sessions for users] ---REST---> [Livy server]
> > In this case, Livy already has a way to pass proxyUser from the
> application
> > to Livy?
> > Sorry, but I'm not familiar with Livy internal logic.
> >
> >
> > On Tue, Jun 26, 2018 at 9:14 AM Marcelo Vanzin
> 
> > wrote:
> >
> >> On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro <
> linguin@gmail.com>
> >> wrote:
> >> > In that case, I think Livy is useful; the application can pass
> proxyUser
> >> to
> >> > build LivyClient for each user
> >> > and run spark queries as each user authorization.
> >>
> >> But Livy already supports impersonation. It can impersonate the
> >> authenticated user.
> >>
> >> You're suggesting adding a parameter so the user can request
> >> impersonation of some specific user, which is a different thing. What
> >> is the use case for that?
> >>
> >> --
> >> Marcelo
> >>
> >
> >
> > --
> > ---
> > Takeshi Yamamuro
>
>
>
> --
> Marcelo
>

Re: A new link request to my project and one question

[Saisai Shao]

Sure, I will merge the website code, thanks!

For proxyUser thing, I think there's no particular reason not adding it,
maybe we just forgot to add the proxyUser support.

It would be better if you could create a JIRA to track this issue. If
you're familiar with Livy code, you can also submit a PR about it.

Thanks
Jerry

Takeshi Yamamuro  于2018年6月15日周五 上午7:33写道：

> Hi, Livy dev,
>
> I opened a new pr in incubator-livy-website to add a new link in
> third-party-projects.md. It'd be great if you could check this;
> https://github.com/apache/incubator-livy-website/pull/23
>
> Btw, I have one question; currently, we cannot pass proxyUser
> in LivyClientBuilder. Any reason not to add code for that?
> I know we can handle this in an application side by adding a bit code like
>
> https://github.com/maropu/spark-sql-server/blob/master/sql/sql-server/src/main/java/org/apache/livyclient/common/CreateClientRequestWithProxyUser.java
> But, If Livy itself supported this, it'd be nice to me.
>
> Best,
> takeshi
>
> --
> ---
> Takeshi Yamamuro
>

About new API set to create interactive and batch session

[Saisai Shao]

Hi Team,

In our current API design to create interactive / batch session, we assume
end user could upload jars, pyFiles and related dependencies to HDFS before
creating the session, and we use one POST request to create session. But
usually end user may not have the permission to access the HDFS in their
submission machine, so it makes them hard to creating new sessions. So the
requirement here is that if Livy could offer APIs to upload resources
during session creation.

One implementation is proposed in
https://github.com/apache/incubator-livy/pull/91. This add a field in
session creation request to delay the session creation, then adding a bunch
of APIs to support resource upload, finally adding an API to start creating
the session. This seems a feasible solution, but also a little hack to
support such scenario. So I was thinking if we could a set of new APIs to
support such scenarios, rather than hack the existing APIs.

To borrow the concept from yarn application submission, we can have 3 APIs
to create session.

1. requesting a new session id from Livy Server.
2. uploading resources associate with this session id.
3. submitting request to create session.

This is similar to YARN's process to submit application, and we can bump
the supported API version for newly added APIs.

What do you think about it? Any suggestion is greatly appreciated!

Best regards

Re: Query on creating multiple livy sessions in parallel

[Saisai Shao]

This might be a BUG. If possible can you please create a JIRA to track this
issue. Thanks!

Best,
Jerry

2018-03-19 20:18 GMT+08:00 Rao, Abhishek (Nokia - IN/Bangalore) <
abhishek@nokia.com>:

> Hi,
>
> We're trying to create multiple livy sessions in parallel and then using
> them. But when we try to create the sessions continuously, we're seeing
> that few sessions are entering to dead state. We see the below exception in
> the logs.
>
> 18/02/27 10:30:20 WARN RSCClient: Client RPC channel closed unexpectedly.
> 18/02/27 10:30:20 WARN RSCClient: Error stopping RPC.
> io.netty.util.concurrent.BlockingOperationException:
> DefaultChannelPromise@7a828ea3(uncancellable)
>at io.netty.util.concurrent.DefaultPromise.checkDeadLock(
> DefaultPromise.java:390)
>at io.netty.channel.DefaultChannelPromise.checkDeadLock(
> DefaultChannelPromise.java:157)
>at io.netty.util.concurrent.DefaultPromise.await(
> DefaultPromise.java:251)
>at io.netty.channel.DefaultChannelPromise.await(
> DefaultChannelPromise.java:129)
>at io.netty.channel.DefaultChannelPromise.await(
> DefaultChannelPromise.java:28)
>at io.netty.util.concurrent.DefaultPromise.sync(
> DefaultPromise.java:218)
>at io.netty.channel.DefaultChannelPromise.sync(
> DefaultChannelPromise.java:117)
>at io.netty.channel.DefaultChannelPromise.sync(
> DefaultChannelPromise.java:28)
>at com.cloudera.livy.rsc.rpc.Rpc.close(Rpc.java:307)
>at com.cloudera.livy.rsc.RSCClient.stop(RSCClient.java:225)
>at com.cloudera.livy.rsc.RSCClient$2$1.onSuccess(
> RSCClient.java:122)
>at com.cloudera.livy.rsc.RSCClient$2$1.onSuccess(
> RSCClient.java:116)
>at com.cloudera.livy.rsc.Utils$2.
> operationComplete(Utils.java:108)
>at io.netty.util.concurrent.DefaultPromise.notifyListener0(
> DefaultPromise.java:680)
>at io.netty.util.concurrent.DefaultPromise.notifyListeners(
> DefaultPromise.java:567)
>at io.netty.util.concurrent.DefaultPromise.trySuccess(
> DefaultPromise.java:406)
>at io.netty.channel.DefaultChannelPromise.trySuccess(
> DefaultChannelPromise.java:82)
>at io.netty.channel.AbstractChannel$CloseFuture.
> setClosed(AbstractChannel.java:956)
>at io.netty.channel.AbstractChannel$
> AbstractUnsafe.doClose0(AbstractChannel.java:608)
>at io.netty.channel.AbstractChannel$AbstractUnsafe.close(
> AbstractChannel.java:586)
>at io.netty.channel.nio.AbstractNioByteChannel$
> NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
>at io.netty.channel.nio.AbstractNioByteChannel$
> NioByteUnsafe.read(AbstractNioByteChannel.java:158)
>at io.netty.channel.nio.NioEventLoop.processSelectedKey(
> NioEventLoop.java:511)
>at io.netty.channel.nio.NioEventLoop.
> processSelectedKeysOptimized(NioEventLoop.java:468)
>at io.netty.channel.nio.NioEventLoop.processSelectedKeys(
> NioEventLoop.java:382)
>at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.
> java:354)
>at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:111)
>at java.lang.Thread.run(Thread.java:748)
> 18/02/27 10:30:20 DEBUG RSCClient: Disconnected from context
> dad7c668-3c09-4ad2-9810-28f684c5ec49, shutdown = false.
>
> However, when we create the sessions one after the other (Create session 1
> after session 0 is in Idle state), it works fine.
> We wanted to know if there is any known restriction in livy for creating
> multiple sessions in parallel.
>
> Thanks & Regards,
> Abhishek
>
>

Fwd: [apache/incubator-livy] One of your dependencies may have a security vulnerability

[Saisai Shao]

Hi Alex,

Would you please check again? Thanks!

Best regards

-- Forwarded message --
From: Apache Security Team <secur...@apache.org>
Date: 2018-03-06 16:39 GMT+08:00
Subject: Re: [apache/incubator-livy] One of your dependencies may have a
security vulnerability
To: Saisai Shao <sai.sai.s...@gmail.com>
Cc: priv...@livy.incubator.apache.org, Apache Security Team <
secur...@apache.org>


Hi, no the gitlab notification states it needs to be nokogiri > 1.8.1 and
your current Gemfile.lock specifies = 1.8.0

Cheers, Mark J Cox

On Tue, Mar 6, 2018 at 12:13 AM, Saisai Shao <sai.sai.s...@gmail.com> wrote:

> I think it was fixed by Alex (https://github.com/apache/inc
> ubator-livy/commit/26428c56f20ba5ea608038ed8c2e11d8f04665d4).
>
>
> 2018-03-06 2:29 GMT+08:00 Marcelo Vanzin <van...@cloudera.com>:
>
>> Hey Alex / Saisai,
>>
>> This was fixed, right?
>>
>> If so you need to update the guys at security@ saying this was fixed (or
>> what needs to be done to fix it).
>>
>>
>> On Mon, Mar 5, 2018 at 1:50 AM, Apache Security Team <secur...@apache.org
>> > wrote:
>>
>>> On Mon, Feb 19, 2018 at 8:55 AM, Apache Security Team <
>>> secur...@apache.org> wrote:
>>>
>>>> Hi Livy team, making sure you saw this and will action it.
>>>>
>>>> Regards, Mark J Cox
>>>>
>>>> On Tue, Jan 23, 2018 at 10:14 PM, Greg Stein <gst...@gmail.com> wrote:
>>>>
>>>>> Livy PPMC: FYI
>>>>>
>>>>> -- Forwarded message --
>>>>> From: GitHub <notificati...@github.com>
>>>>> Date: Tue, Jan 23, 2018 at 2:22 PM
>>>>> Subject: [apache/incubator-livy] One of your dependencies may have a
>>>>> security vulnerability
>>>>> To: apache/incubator-livy <incubator-l...@noreply.github.com>
>>>>> Cc: Security alert <security_al...@noreply.github.com>
>>>>>
>>>>>
>>>>> We found a potential security vulnerabilty in one of your dependencies
>>>>> [image: GitHub]
>>>>> <http://sgmail.githubmail.com/wf/click?upn=lYxq-2FYU7yocrdKNILYalBlaoUQ7ZnNSfaod-2BRPoWgKQ-3D_w6S5n3vrKqGS7A36Z0jQnv0H94jgQYM8GX7TqkbHsZJGWdsYqQFjxwriEF8ZmW1sZ8ttXgzgS3BVWKu3VBqXOMpSzW2VEkJKKe2e9uTex9Q7Z9UijIWv0RRYA-2Fdc2r546s6eSy8HZocDFla36b4iDH-2B3aDT4HLjIh-2Fo9vK3qWDuW00SPllrHUyE-2F7oUepVlho6xRLLFnygiZnALZqGXTakYwTsw7U1i0kOz8YTJZN0atv-2B6Wb8Vsz97NI2noXzGt>
>>>>>  Sign
>>>>> in
>>>>> <http://sgmail.githubmail.com/wf/click?upn=lYxq-2FYU7yocrdKNILYalBluE-2FGrtUQ7WwbM8S6nEaj0-3D_w6S5n3vrKqGS7A36Z0jQnv0H94jgQYM8GX7TqkbHsZJGWdsYqQFjxwriEF8ZmW1sJEB6DZ3WcL-2F4II6g4nOXtSt18YBqIm8t9ln67kM2qPU7-2BwIp1OhBha1A2HxxgKMyX40eU0B-2BxCoEbAUvsw0AB0X9T5UGmnA4C-2BYrM2D-2B3MDuTZhqAqaXY6Ippc5CRnN3usIzrFwtPWH1tKIk-2FIapGBC7Y2Lsyw7S4QWTtwqE8U67-2FuDGyxs1Fd0tvqdx9gIQ>
>>>>> *gstein,*
>>>>>
>>>>> We found a potential security vulnerability in a repository for which
>>>>> you have been granted security alert access.
>>>>> [image: @apache] apache/incubator-livy
>>>>> <http://sgmail.githubmail.com/wf/click?upn=lYxq-2FYU7yocrdKNILYalBg5kFs28ucWJkBdd8Thfp23Ag8-2FxhdvxK9GAMrvp8gUC_w6S5n3vrKqGS7A36Z0jQnv0H94jgQYM8GX7TqkbHsZJGWdsYqQFjxwriEF8ZmW1sqJPyhuVFWI7a-2BCvW4tyXVGKVBZY13BEvr-2Bq0IaZU-2BUr9JXtZ-2FwPj4cV2z3v3QVEOiwfg1cPLVN93lsgJ8m5RMCdkFQBaHX-2Bxc-2B-2BIRsFowmpW0QyMBlxuDLaxDM4JwxNhXI3BIM7nyaHpSS-2FYq6xcOzCY2u-2B-2B2GH1SAI3PmsjyEjQqdMIARNgBMpvoIRbrRgp>
>>>>> Known * critical severity* security vulnerability detected in nokogiri
>>>>> < 1.8.1 defined in Gemfile.lock
>>>>> <http://sgmail.githubmail.com/wf/click?upn=lYxq-2FYU7yocrdKNILYalBg5kFs28ucWJkBdd8Thfp210wIho9lAyQVafDi7j-2Bh1B6kbDR-2FojhEUYkAYcdbN0VSnoCf19MxCRvx0tyoloYkc-3D_w6S5n3vrKqGS7A36Z0jQnv0H94jgQYM8GX7TqkbHsZJGWdsYqQFjxwriEF8ZmW1sKmS-2B4Jr20quYqSULJfJpwhzNFCYuG-2Fcp-2BZ53NXhvxtDb6uQlhPVD-2BWhPS-2F8KvYfjoJvoxxa-2B8fGggIKzvNEAZq3ghOpKRdYfXiWO7PMcJMkpxyPF1lBYdww4rR2mqKtRCh8hbW8Pikyiij0abzMoZOe5IhuZhuCtVolZWuydD9MOHFlbZZ085iiui59TrE6Z>.
>>>>>
>>>>> Gemfile.lock
>>>>> <http://sgmail.githubmail.com/wf/click?upn=lYxq-2FYU7yocrdKNILYalBg5kFs28ucWJkBdd8Thfp210wIho9lAyQVafDi7j-2Bh1B6kbDR-2FojhEUYkAYcdbN0VSnoCf19MxCRvx0tyoloYkc-3D_w6S5n3vrKqGS7A36Z0jQnv0H94jgQYM8GX7TqkbHsZJGWdsYqQFjxwriEF8ZmW1sZ5YHwUqPmspFEs5FGzvBtkT-2BGxTgoMX32p1A30L7XZl9ba1BQ6kIc1Ju5KJnc9UFc9YhoObi9S7D6j4K4Kd-2BPNMLSjQYMDdw1Ok22ar0ELvfe0GIC8Kr6L3-2BcuFd4h134bTAF-2BE4BkAZkEJ09z-2FBOw8UEmNbvbW47WusN6PUaa-2BpC4X2-2BAl0DkEaPeDdIX4p>
>>>>> update suggested: nokog

Re: Podling Report Reminder - February 2018

[Saisai Shao]

Great, thanks Alex!

Best regards,
Jerry

2018-02-06 7:56 GMT+08:00 Alex Bozarth :

> I've updated the report if a mentor could take a look and sign off (or
> give feedback).
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* 
>
>
> 505 Howard Street
> 
> San Francisco, CA 94105
> 
> United States
> 
>
>
>
> [image: Inactive hide details for johndament---02/05/2018 03:39:37
> PM---Dear podling, This email was sent by an automated system on 
> beh]johndament---02/05/2018
> 03:39:37 PM---Dear podling, This email was sent by an automated system on
> behalf of the Apache
>
> From: johndam...@apache.org
> To: dev@livy.incubator.apache.org
> Date: 02/05/2018 03:39 PM
> Subject: Podling Report Reminder - February 2018
> --
>
>
>
> Dear podling,
>
> This email was sent by an automated system on behalf of the Apache
> Incubator PMC. It is an initial reminder to give you plenty of time to
> prepare your quarterly board report.
>
> The board meeting is scheduled for Wed, 21 February 2018, 10:30 am PDT.
> The report for your podling will form a part of the Incubator PMC
> report. The Incubator PMC requires your report to be submitted 2 weeks
> before the board meeting, to allow sufficient time for review and
> submission (Wed, February 07).
>
> Please submit your report with sufficient time to allow the Incubator
> PMC, and subsequently board members to review and digest. Again, the
> very latest you should submit your report is 2 weeks prior to the board
> meeting.
>
> Thanks,
>
> The Apache Incubator PMC
>
> Submitting your Report
>
> --
>
> Your report should contain the following:
>
> *   Your project name
> *   A brief description of your project, which assumes no knowledge of
>the project or necessarily of its field
> *   A list of the three most important issues to address in the move
>towards graduation.
> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
>aware of
> *   How has the community developed since the last report
> *   How has the project developed since the last report.
> *   How does the podling rate their own maturity.
>
> This should be appended to the Incubator Wiki page at:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.
> apache.org_incubator_February2018=DwICAg=jf_iaSHvJObTbx-siA1ZOg=S1_
> S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=9w4xtXyRYOT11wkHHK2PmZpGws3TV-
> cfS06hWIz5_Ts=bk0pXyivBbYVNYwGkWrBcayuHCm00XM3YMlLps3E8yI=
>
> Note: This is manually populated. You may need to wait a little before
> this page is created from a template.
>
> Mentors
> ---
>
> Mentors should review reports for their project(s) and sign them off on
> the Incubator wiki page. Signing off reports shows that you are
> following the project - projects that are not signed may raise alarms
> for the Incubator PMC.
>
> Incubator PMC
>
>
>
>
>

Re: [VOTE] Livy 0.5.0-incubating (RC2)

[Saisai Shao]

asc, sha512 and md5 LGTM. Also checked the package, seems everything is
landed.

Alex, did you also publish Livy artifacts to staging repo?

2018-01-26 8:03 GMT+08:00 Alex Bozarth :

> That works for me, voting will be open an extra day for a total of 96
> hours ending on Tuesday January 30th at 12:00AM (UTC)
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* 
>
>
> 505 Howard Street
> 
> San Francisco, CA 94105
> 
> United States
> 
>
>
>
> [image: Inactive hide details for Marcelo Vanzin ---01/25/2018 04:00:28
> PM---On Thu, Jan 25, 2018 at 3:57 PM, Alex Bozarth  Vanzin ---01/25/2018 04:00:28 PM---On Thu, Jan 25, 2018 at 3:57 PM, Alex
> Bozarth  wrote: > The vote will be open 7
>
> From: Marcelo Vanzin 
> To: dev@livy.incubator.apache.org
> Date: 01/25/2018 04:00 PM
> Subject: Re: [VOTE] Livy 0.5.0-incubating (RC2)
> --
>
>
>
> On Thu, Jan 25, 2018 at 3:57 PM, Alex Bozarth  wrote:
> > The vote will be open 72 hours until Monday January 29th at 12:00AM UTC
> and
> > will pass with a minimum of 3 +1 votes and a majority of positive votes.
>
> That means most of the voting period will be on a weekend. Maybe leave
> it open a little longer?
>
> --
> Marcelo
>
>
>
>
>

Fwd: [apache/incubator-livy] One of your dependencies may have a security vulnerability

[Saisai Shao]

Hi Alex,

Is it due to your recent changes to add ruby file?

Thanks
Jerry

-- Forwarded message --
From: Greg Stein 
Date: 2018-01-24 6:14 GMT+08:00
Subject: Fwd: [apache/incubator-livy] One of your dependencies may have a
security vulnerability
To: priv...@livy.incubator.apache.org
Cc: priv...@incubator.apache.org, priv...@infra.apache.org,
secur...@apache.org


Livy PPMC: FYI

-- Forwarded message --
From: GitHub 
Date: Tue, Jan 23, 2018 at 2:22 PM
Subject: [apache/incubator-livy] One of your dependencies may have a
security vulnerability
To: apache/incubator-livy 
Cc: Security alert 


We found a potential security vulnerabilty in one of your dependencies
[image: GitHub]

Sign
in

*gstein,*

We found a potential security vulnerability in a repository for which you
have been granted security alert access.
[image: @apache] apache/incubator-livy

Known * critical severity* security vulnerability detected in nokogiri <
1.8.1 defined in Gemfile.lock
.

Gemfile.lock

update suggested: nokogiri ~> 1.8.1.
Always verify the validity and compatibility of suggestions with your
codebase.
Review vulnerable dependency

--

Only users who have been assigned access to security alerts will receive
these notifications.
Unsubscribe

· Email preferences

· Terms

Re: [DRAFT] Incubator PMC Board Report - January 2018

[Saisai Shao]

Sorry I was on paternity leave last two weeks, didn't have time to check
it. Maybe you can also help on this next time.

Thanks
Jerry

2018-01-09 4:32 GMT+08:00 Alex Bozarth :

> Saw that we didn't report this month, any reason why, or was everyone just
> on holiday?
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* 
>
>
> 505 Howard Street
> 
> San Francisco, CA 94105
> 
> United States
> 
>
>
>
> [image: Inactive hide details for "John D. Ament" ---01/08/2018 11:06:22
> AM---All, Below is the current draft on the incubator report.]"John D.
> Ament" ---01/08/2018 11:06:22 AM---All, Below is the current draft on the
> incubator report. We have a high number
>
> From: "John D. Ament" 
> To: "gene...@incubator.apache.org" 
> Cc: "d...@annotator.apache.org" ,
> d...@crail.apache.org, "d...@htrace.incubator.apache.org" <
> d...@htrace.incubator.apache.org>, "dev@livy.incubator.apache.org" <
> dev@livy.incubator.apache.org>, "d...@milagro.apache.org" <
> d...@milagro.apache.org>, "d...@myriad.incubator.apache.org" <
> d...@myriad.incubator.apache.org>, d...@sdap.apache.org,
> d...@senssoft.apache.org, "d...@spot.incubator.apache.org" <
> d...@spot.incubator.apache.org>, d...@airflow.apache.org, "
> odf-...@incubator.apache.org" 
> Date: 01/08/2018 11:06 AM
> Subject: [DRAFT] Incubator PMC Board Report - January 2018
> --
>
>
>
> All,
>
> Below is the current draft on the incubator report.  We have a high number
> of podlings not reporting.  They are CC'd in hopes they can report.
>
> Incubator PMC report for January 2018
>
> The Apache Incubator is the entry path into the ASF for projects and
> codebases wishing to become part of the Foundation's efforts.
>
> There are currently 53 podlings in the inubator.  We added two new podlings
> in December, and executed four podling releases.  Two new PMC members
> joined, in support of mentoring podlings.
>
> * Community
>
>  New IPMC members:
>
>  - Stefan Bodewig
>  - Carl Johan Erik Edstrom
>
>  People who left the IPMC:
>
>
>
> * New Podlings
>
>  - PLC4X
>  - SkyWalking
>
> * Podlings that failed to report, expected next month
>
>  - Annotator
>  - Crail
>  - HTrace
>  - Livy
>  - Milagro
>  - Myriad
>  - SDAP
>  - SensSoft
>  - Spot
>  - Wave
>
> * Podlings missing sign off
>
>  - Airflow
>  - ODF Toolkit
>
> * Graduations
>
>  The board has motions for the following:
>
>  - Your podling here?
>  - Your podling here?
>
> * Releases
>
>  The following releases entered distribution during the month of
>  December:
>
>  - 2017-12-03 Apache MXNet 1.0.0
>  - 2017-12-13 Apache BatchEE 0.5
>  - 2017-12-14 Apache Edgent 1.2.0
>  - 2017-12-17 Apache Pulsar 1.21.0
>
> * IP Clearance
>
>
>
> * Legal / Trademarks
>
>
>
> * Infrastructure
>
>
>
> * Miscellaneous
>
>
>
> * Credits
>
> --
>   Table of Contents
> Airflow
> Amaterasu
> Annotator
> BatchEE
> Crail
> DataFu
> FreeMarker
> Gobblin
> Gossip
> HAWQ
> HTrace
> Livy
> Milagro
> MXNet
> Myriad
> NetBeans
> ODF Toolkit
> PageSpeed
> PLC4X
> Pony Mail
> Rya
> SDAP
> SensSoft
> ServiceComb
> SkyWalking
> Spot
> Traffic Control
> Wave
> Weex
>
> --
>
> 
> Airflow
>
> Airflow is a workflow automation and scheduling system that can be used to
> author and manage data pipelines.
>
> Airflow has been incubating since 2016-03-31.
>
> Three most important issues to address in the move towards graduation:
>
>  1. We have completed 4 apache releases. We are nearing graduation!
>  2.
>  3.
>
> Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> aware of?
> None
>
>
> How has the community developed since the last report?
>  1. Since our last podling report 3 months ago (i.e. between Sept 25 & Dec
> 26, inclusive), we grew our contributors from 315 to 360
>  2. Since our last podling report 3 months ago (i.e. between Sept 25 & Dec
> 26, inclusive), we resolved 221 pull requests (currently at 2018 closed
> PRs)
>  3. Since being accepted into the incubator, the number of companies
> officially using Apache Airflow has risen from 30 to 123, 9 new from
> the last podling report.
>
>
> How has the project developed since the last report?
> See above : 221 PR resolved, 45 new contributors, & 9 new companies
> officially using it.
>
>
> How 

Re: How to get the test logs from travis

[Saisai Shao]

Thanks Luciano,

I also googled the approach, looks like cat is the most recommended way to
handle this. Another way is to upload to some public storage service.

Best regards,
Jerry


On Thu, Sep 21, 2017 at 11:30 AM, Luciano Resende <luckbr1...@gmail.com>
wrote:

> Take a look at what Zeppelin does, they kind cat a lot of the logs and
> related files which then get appended to the build results.
>
> https://github.com/apache/zeppelin/blob/master/.travis.yml
>
> On Wed, Sep 20, 2017 at 8:22 PM, Saisai Shao <sai.sai.s...@gmail.com>
> wrote:
>
> > Hi Team,
> >
> > Currently it is quite painful to figure out the cause of failure on
> travis
> > test. Do you know how to get test logs from travis, is there a way
> > supported by travis to either upload environment to some places, or log
> on
> > to travis to dig the files.
> >
> > Previously we uploaded logs to azure when test is failed, I'm not sure it
> > is still worked, shall we figure out a stable way to address this issue?
> >
> > Thanks
> > Jerry
> >
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

How to get the test logs from travis

[Saisai Shao]

Hi Team,

Currently it is quite painful to figure out the cause of failure on travis
test. Do you know how to get test logs from travis, is there a way
supported by travis to either upload environment to some places, or log on
to travis to dig the files.

Previously we uploaded logs to azure when test is failed, I'm not sure it
is still worked, shall we figure out a stable way to address this issue?

Thanks
Jerry

Re: user defined sessionId / URI for Livy sessions

[Saisai Shao]

I see. So based on this, we should manage a data structure in Livy Server
to keep all the live sessions' name. Also regarding to session recovery, we
should persist this structure to the reliable storage and recover after
restart.

I'm not pretty sure if it is a good feature or not. First because we
usually programmatically manage the session id, so from code level to
manager a session id or a session name there's no much difference; second,
usually it is hard for user to pick a unique name if one Livy Server has
many live sessions, by large chance the name will be conflicted, people
always like short, simple name.

Since I'm so familiar how people really use it, so it is just my two cents.

Thanks
Jerry


On Tue, Sep 12, 2017 at 8:46 AM, Meisam Fathi 
wrote:

> > If we're using session name, how do we guarantee the uniqueness of this
> > name?
> >
>
> If the requested session name already exist, Livy returns an error and does
> not create the session.
>
> Thanks,
> Meisam
>

Re: user defined sessionId / URI for Livy sessions

[Saisai Shao]

If we're using session name, how do we guarantee the uniqueness of this
name?

Thanks
Jerry

On Tue, Sep 12, 2017 at 4:51 AM, Alex Bozarth  wrote:

> I would agree with Marcelo's comment the JIRA that this isn't a good
> feature for livy, but I'll take a look at your impl if you open a PR and
> see if it changes my mind.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* 
>
>
> 505 Howard Street
> 
> San Francisco, CA 94105
> 
> United States
> 
>
>
>
> [image: Inactive hide details for Meisam Fathi ---09/11/2017 10:23:49
> AM---+ dev Is there any interest in adding this feature to Livy?]Meisam
> Fathi ---09/11/2017 10:23:49 AM---+ dev Is there any interest in adding
> this feature to Livy? I can send a PR
>
> From: Meisam Fathi 
> To: "u...@livy.incubator.apache.org" , "
> dev@livy.incubator.apache.org" 
> Date: 09/11/2017 10:23 AM
> Subject: Re: user defined sessionId / URI for Livy sessions
> --
>
>
>
> + dev
> Is there any interest in adding this feature to Livy? I can send a PR
>
> Ideally, it would be helpful if we could mint a session ID with a PUT
> > request, something like PUT /sessions/foobar, where "foobar" is the newly
> > created sessionId.
> >
> > I suggest we make session names unique and nonnumeric values (to
> guarantee
> a session name does not clash with another session name or session ID).
>
> Design doc:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_meisam_incubator-2Dlivy_wiki_Design-2Ddoc-2Dfor-
> 2DLivy-2D41-3A-2DAccessing-2Dsessions-2Dby-2Dname=
> DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-
> cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=
> xTiY52FMWMdTRgCmiNRWe6yEoCchxKNxQrYPEkPupbw=
> JIRA ticket: https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_LIVY-2D41=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_
> S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-
> juQkf5Q2FMYtA5kv5sqiMM=lFed2hYlDA_wUo94RWUAw7N01lSN368P-ABmP_npWrM=
>
>
> Thanks,
> Meisam
>
>
>
>

[ANNOUNCE] Apache Livy 0.4.0-incubating released

[Saisai Shao]

The Apache Livy team is proud to announce Apache Livy version
0.4.0-incubating.

This is the first Livy release after entering the Apache Incubator.

Livy is web service that exposes a REST interface for managing
long running Apache Spark contexts in your cluster. With Livy, new
applications can be built on top of Apache Spark that require fine grained
interaction with many Spark contexts.

For Livy release details and downloads, visit:
http://livy.incubator.apache.org/download/


We would like to thank the contributors that made the release possible.


Regards,
The Livy Team

Re: The process of Livy 0.4.0-incubating release

[Saisai Shao]

Hi all,

All the stuffs related to 0.4.0-incubating release are done, please check.

If you don't mind I'm going to announce the release of Livy
0.4.0-incubating in the incubation mail list.

Thanks
Jerry



On Sat, Sep 2, 2017 at 2:11 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:

> The website PRs are updated (with a 9/1/17 release date) and ready to
> merge.
>
> @jerry if you want to merge them and update the website when you send the
> announcement email.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Saisai Shao ---09/01/2017 01:11:03
> AM---Hi all, I just published maven artifacts to repository.apache]Saisai
> Shao ---09/01/2017 01:11:03 AM---Hi all, I just published maven artifacts
> to repository.apache.org, please check (https://urldefense.
>
> From: Saisai Shao <sai.sai.s...@gmail.com>
> To: dev@livy.incubator.apache.org
> Date: 09/01/2017 01:11 AM
> Subject: Re: The process of Livy 0.4.0-incubating release
> --
>
>
>
> Hi all, I just published maven artifacts to repository.apache.org, please
> check (https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__repository.apache.org_-23nexus-2Dsearch-3Bquick-
> 7Eorg.apache.livy=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=S1_
> S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=fMV22rxtZ56xBOsS_
> 08nAtWN0INQGs6rwMRoj1ot8js=PDM5YGd_nH5qlIketLl_
> Igr0Fhdqyf9IsI1QIwwEq4M= ).
>
> I think once related doc is updated, all the release work should be done.
>
> Thanks
> Jerry
>
>
>
> On Thu, Aug 31, 2017 at 9:01 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:
>
> > I agree with Jeff on the announcement timing, the links on the website PR
> > are already updated and working, I'll just have to push an update to the
> > release date once we know when we'll announce the release. And you can
> > delete the gh-pages branch, it's been moved to the old-site branch on the
> > website repo.
> >
> >
> > *Alex Bozarth*
> > Software Engineer
> > Spark Technology Center
> > --
> > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> > *GitHub: **github.com/ajbozarth* <https://urldefense.
> proofpoint.com/v2/url?u=https-3A__github.com_ajbozarth=
> DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-
> cx37DPYDyo=fMV22rxtZ56xBOsS_08nAtWN0INQGs6rwMRoj1ot8js=
> CzCDePcP9HxXc4FCCima2ThJ0zgF6K5oHcRKWJEegMg= >
> >
> >
> > 505 Howard Street
> > San Francisco, CA 94105
> > United States
> >
> >
> >
> > [image: Inactive hide details for Jeff Zhang ---08/30/2017 05:53:15
> PM---I
> > think we'd better to announce after the artifacts are publis]Jeff Zhang
> > ---08/30/2017 05:53:15 PM---I think we'd better to announce after the
> > artifacts are published. Saisai Shao <sai.sai.shao@gmail.c
> >
> > From: Jeff Zhang <zjf...@gmail.com>
> > To: dev@livy.incubator.apache.org
> > Date: 08/30/2017 05:53 PM
> > Subject: Re: The process of Livy 0.4.0-incubating release
> > --
>
> >
> >
> >
> > I think we'd better to announce after the artifacts are published.
> >
> > Saisai Shao <sai.sai.s...@gmail.com>于2017年8月31日周四 上午8:35写道：
> >
> > > Hi Alex,
> > >
> > > I think you can update the website PR firstly to link to the correct
> > > download URL (since the package is already available) and doc.
> > >
> > > I'm working on publish jars to nexus repository, currently I'm waiting
> > for
> > > infra team to create a Livy nexus profile, so I push push jars to
> staging
> > > repo.
> > >
> > > Is it OK to announce now? Since we haven't yet pushed jars, technically
> > the
> > > release process is not finished.
> > >
> > > I will cleanup some unnecessary branches and tags, one thing is that is
> > it
> > > OK to remove gh-pages branch?
> > >
> > > Thanks
> > > Jerry
> > >
> > >
> > >
> > > On Thu, Aug 31, 2017 at 2:59 AM, Alex Bozarth <ajboz...@us.ibm.com>
> > wrote:
> > >
> > > > So is the release ready to announce then? The apache bin/src download
> > > > links are live so all I need for the website update is an o

Re: The process of Livy 0.4.0-incubating release

[Saisai Shao]

Hi all, I just published maven artifacts to repository.apache.org, please
check (https://repository.apache.org/#nexus-search;quick~org.apache.livy).

I think once related doc is updated, all the release work should be done.

Thanks
Jerry



On Thu, Aug 31, 2017 at 9:01 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:

> I agree with Jeff on the announcement timing, the links on the website PR
> are already updated and working, I'll just have to push an update to the
> release date once we know when we'll announce the release. And you can
> delete the gh-pages branch, it's been moved to the old-site branch on the
> website repo.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Jeff Zhang ---08/30/2017 05:53:15 PM---I
> think we'd better to announce after the artifacts are publis]Jeff Zhang
> ---08/30/2017 05:53:15 PM---I think we'd better to announce after the
> artifacts are published. Saisai Shao <sai.sai.shao@gmail.c
>
> From: Jeff Zhang <zjf...@gmail.com>
> To: dev@livy.incubator.apache.org
> Date: 08/30/2017 05:53 PM
> Subject: Re: The process of Livy 0.4.0-incubating release
> --
>
>
>
> I think we'd better to announce after the artifacts are published.
>
> Saisai Shao <sai.sai.s...@gmail.com>于2017年8月31日周四 上午8:35写道：
>
> > Hi Alex,
> >
> > I think you can update the website PR firstly to link to the correct
> > download URL (since the package is already available) and doc.
> >
> > I'm working on publish jars to nexus repository, currently I'm waiting
> for
> > infra team to create a Livy nexus profile, so I push push jars to staging
> > repo.
> >
> > Is it OK to announce now? Since we haven't yet pushed jars, technically
> the
> > release process is not finished.
> >
> > I will cleanup some unnecessary branches and tags, one thing is that is
> it
> > OK to remove gh-pages branch?
> >
> > Thanks
> > Jerry
> >
> >
> >
> > On Thu, Aug 31, 2017 at 2:59 AM, Alex Bozarth <ajboz...@us.ibm.com>
> wrote:
> >
> > > So is the release ready to announce then? The apache bin/src download
> > > links are live so all I need for the website update is an official
> > release
> > > date, would that be today or the day we announce to the user list? On a
> > > related note, should we clean up our branches and tags on the git repo?
> > It
> > > makes sense to remove the rc tags, but should we also get rid of the
> > extra
> > > branches other than "branch-0.x"
> > >
> > >
> > > *Alex Bozarth*
> > > Software Engineer
> > > Spark Technology Center
> > > --
> > > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> > > *GitHub: **github.com/ajbozarth* <https://urldefense.
> proofpoint.com/v2/url?u=https-3A__github.com_ajbozarth=
> DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-
> cx37DPYDyo=zDjACJJqa08AnZmjSeGhW07H5m36H1Jm-wbbwP_BFX8=
> xlygKgJEYxQqGARDhM7RSr8ZPQEW7fo1xuhNihFXR68= >
>
> > >
> > >
> > > 505 Howard Street
> > > San Francisco, CA 94105
> > > United States
> > >
> > >
> > >
> > > [image: Inactive hide details for Saisai Shao ---08/30/2017 12:05:24
> > > AM---Looks like I wronged the releasing process, I should
> publish]Saisai
> > > Shao ---08/30/2017 12:05:24 AM---Looks like I wronged the releasing
> > > process, I should publish jars to staging repo according with RC
> > >
> > > From: Saisai Shao <sai.sai.s...@gmail.com>
> > > To: dev@livy.incubator.apache.org
> > > Date: 08/30/2017 12:05 AM
> > > Subject: Re: The process of Livy 0.4.0-incubating release
> > > --
> > >
> > >
> > >
> > > Looks like I wronged the releasing process, I should publish jars to
> > > staging repo according with RC vote, and after vote passed click
> release
> > to
> > > publish jars. But what I currently do is to publish jars to staging
> repo
> > > after vote passed. I'm really sorry about not familiar with the
> process.
> > I
> > > will write a doc as well as release script for the next release.
> > >
> > > Sorry

Re: The process of Livy 0.4.0-incubating release

[Saisai Shao]

Hi Alex,

I think you can update the website PR firstly to link to the correct
download URL (since the package is already available) and doc.

I'm working on publish jars to nexus repository, currently I'm waiting for
infra team to create a Livy nexus profile, so I push push jars to staging
repo.

Is it OK to announce now? Since we haven't yet pushed jars, technically the
release process is not finished.

I will cleanup some unnecessary branches and tags, one thing is that is it
OK to remove gh-pages branch?

Thanks
Jerry



On Thu, Aug 31, 2017 at 2:59 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:

> So is the release ready to announce then? The apache bin/src download
> links are live so all I need for the website update is an official release
> date, would that be today or the day we announce to the user list? On a
> related note, should we clean up our branches and tags on the git repo? It
> makes sense to remove the rc tags, but should we also get rid of the extra
> branches other than "branch-0.x"
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Saisai Shao ---08/30/2017 12:05:24
> AM---Looks like I wronged the releasing process, I should publish]Saisai
> Shao ---08/30/2017 12:05:24 AM---Looks like I wronged the releasing
> process, I should publish jars to staging repo according with RC
>
> From: Saisai Shao <sai.sai.s...@gmail.com>
> To: dev@livy.incubator.apache.org
> Date: 08/30/2017 12:05 AM
> Subject: Re: The process of Livy 0.4.0-incubating release
> --
>
>
>
> Looks like I wronged the releasing process, I should publish jars to
> staging repo according with RC vote, and after vote passed click release to
> publish jars. But what I currently do is to publish jars to staging repo
> after vote passed. I'm really sorry about not familiar with the process. I
> will write a doc as well as release script for the next release.
>
> Sorry about it.
>
> Best regards,
> Jerry
>
> On Wed, Aug 30, 2017 at 2:10 PM, Saisai Shao <sai.sai.s...@gmail.com>
> wrote:
>
> > Hi Luciano,
> >
> > I'm trying to push maven artifacts to maven staging repository at
> > repository.apache.org, seems we need a org.apache.livy staging profile
> to
> > use to create a staging repo, I checked Spark release script, it has a
> > profile "d63f592e7eac0" used to create repo, do we need this profile for
> > Livy? Also how can I create this profile?
> >
> > Thanks
> > Jerry
> >
> >
> > On Mon, Aug 28, 2017 at 3:39 PM, Luciano Resende <luckbr1...@gmail.com>
> > wrote:
> >
> >> On Sun, Aug 27, 2017 at 11:53 PM, Saisai Shao <sai.sai.s...@gmail.com>
> >> wrote:
> >>
> >> > Hi mentors,
> >> >
> >> > Would you please guide us on how to release the Livy 0.4.0-incubating,
> >> > including artifact publish.
> >> >
> >>
> >> The artifacts move from :
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__dist.
> apache.org_repos_dist_dev_incubator_livy_=DwIBaQ=jf_
> iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=
> 73AoTw0B3DG5OXNjZxX9MzTFe42Dx59uljW55vuT2R0=
> 7tRxDlukXRLudKDtlpeeok3lzvCDHHYw-7oMx8fwz-I=
> >>
> >> to
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__dist.
> apache.org_repos_dist_release_incubator_livy_=DwIBaQ=jf_
> iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=
> 73AoTw0B3DG5OXNjZxX9MzTFe42Dx59uljW55vuT2R0=
> fKrHDzEReoqNRDrYzvoBKQ0hN7wHdFsSgJ0ka3vNDJk=
> >>
> >>
> >> >
> >> > Also regarding push artifacts to repo, are we inclining to change to
> >> Maven
> >> > central repo, or we still use Cloudera repo, any suggestion?
> >> >
> >> >
> >> You should have a maven staging repository at repository.apache.org,
> >> after
> >> a release is approved, that repository should just be released.
> >>
> >>
> >> > Besides, for Python package release, do we need to publish this python
> >> > client package to PIP or we don't need to do this now, any comment?
> >> >
> >> >
> >> +0, I would say do what you have been doing in the past
> >>
> >>
> >> > Thanks for your help and suggestion.
> >> >
> >> > Best regards,
> >> > Saisai (Jerry)
> >> >
> >>
> >>
> >>
> >> --
> >> Luciano Resende
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.
> com_lresende1975=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_
> S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=73AoTw0B3DG5OXNjZxX9MzTFe42Dx5
> 9uljW55vuT2R0=ooMsKS7OJms1O05-O2L786_xRXZLHYTvQ1aVPDEy_9s=
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__
> lresende.blogspot.com_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_
> S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=73AoTw0B3DG5OXNjZxX9MzTFe42Dx5
> 9uljW55vuT2R0=IWI3eFgaDV6GPzGVufAJAQioCPr7ydFrZhfYZEKPhvo=
> >>
> >
> >
>
>
>
>

Re: The process of Livy 0.4.0-incubating release

[Saisai Shao]

Looks like I wronged the releasing process, I should publish jars to
staging repo according with RC vote, and after vote passed click release to
publish jars. But what I currently do is to publish jars to staging repo
after vote passed. I'm really sorry about not familiar with the process. I
will write a doc as well as release script for the next release.

Sorry about it.

Best regards,
Jerry

On Wed, Aug 30, 2017 at 2:10 PM, Saisai Shao <sai.sai.s...@gmail.com> wrote:

> Hi Luciano,
>
> I'm trying to push maven artifacts to maven staging repository at
> repository.apache.org, seems we need a org.apache.livy staging profile to
> use to create a staging repo, I checked Spark release script, it has a
> profile "d63f592e7eac0" used to create repo, do we need this profile for
> Livy? Also how can I create this profile?
>
> Thanks
> Jerry
>
>
> On Mon, Aug 28, 2017 at 3:39 PM, Luciano Resende <luckbr1...@gmail.com>
> wrote:
>
>> On Sun, Aug 27, 2017 at 11:53 PM, Saisai Shao <sai.sai.s...@gmail.com>
>> wrote:
>>
>> > Hi mentors,
>> >
>> > Would you please guide us on how to release the Livy 0.4.0-incubating,
>> > including artifact publish.
>> >
>>
>> The artifacts move from :
>> https://dist.apache.org/repos/dist/dev/incubator/livy/
>>
>> to
>> https://dist.apache.org/repos/dist/release/incubator/livy/
>>
>>
>> >
>> > Also regarding push artifacts to repo, are we inclining to change to
>> Maven
>> > central repo, or we still use Cloudera repo, any suggestion?
>> >
>> >
>> You should have a maven staging repository at repository.apache.org,
>> after
>> a release is approved, that repository should just be released.
>>
>>
>> > Besides, for Python package release, do we need to publish this python
>> > client package to PIP or we don't need to do this now, any comment?
>> >
>> >
>> +0, I would say do what you have been doing in the past
>>
>>
>> > Thanks for your help and suggestion.
>> >
>> > Best regards,
>> > Saisai (Jerry)
>> >
>>
>>
>>
>> --
>> Luciano Resende
>> http://twitter.com/lresende1975
>> http://lresende.blogspot.com/
>>
>
>

Re: The process of Livy 0.4.0-incubating release

[Saisai Shao]

Hi Luciano,

I'm trying to push maven artifacts to maven staging repository at
repository.apache.org, seems we need a org.apache.livy staging profile to
use to create a staging repo, I checked Spark release script, it has a
profile "d63f592e7eac0" used to create repo, do we need this profile for
Livy? Also how can I create this profile?

Thanks
Jerry


On Mon, Aug 28, 2017 at 3:39 PM, Luciano Resende <luckbr1...@gmail.com>
wrote:

> On Sun, Aug 27, 2017 at 11:53 PM, Saisai Shao <sai.sai.s...@gmail.com>
> wrote:
>
> > Hi mentors,
> >
> > Would you please guide us on how to release the Livy 0.4.0-incubating,
> > including artifact publish.
> >
>
> The artifacts move from :
> https://dist.apache.org/repos/dist/dev/incubator/livy/
>
> to
> https://dist.apache.org/repos/dist/release/incubator/livy/
>
>
> >
> > Also regarding push artifacts to repo, are we inclining to change to
> Maven
> > central repo, or we still use Cloudera repo, any suggestion?
> >
> >
> You should have a maven staging repository at repository.apache.org, after
> a release is approved, that repository should just be released.
>
>
> > Besides, for Python package release, do we need to publish this python
> > client package to PIP or we don't need to do this now, any comment?
> >
> >
> +0, I would say do what you have been doing in the past
>
>
> > Thanks for your help and suggestion.
> >
> > Best regards,
> > Saisai (Jerry)
> >
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Update ASF GitHub Bot Settings?

[Saisai Shao]

I saw Zeppelin also mirror everything from github to JIRA, if INFRA is hard
to change to what you expected, I'm fine with current way. Since Livy is
not so active as Spark, we may not suffer from reviewing pain.

On Tue, Aug 29, 2017 at 10:47 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:

> Spark does use a script to do it, but that script requires a special admin
> JIRA account and is run via a cron job as part of the pr dashboard website
> by Databricks (https://spark-prs.appspot.com). The website source code is
> open source and I've actually done quick a bit of work with it, but it
> requires a paid Google CloudPlat account to run in production. Overall I
> like how Spark does it but it's not as simple to reproduce as it seems. If
> someone want to pay for a Google CloudPlat account and set up a ASF JIRA
> Bot like Spark's then I can easily set up a LIVY PR Dash that will run the
> JIRA update cron job.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Saisai Shao ---08/28/2017 06:53:28
> PM---Hi Alex, Do you want to achieve what Spark did in the JIRA? A]Saisai
> Shao ---08/28/2017 06:53:28 PM---Hi Alex, Do you want to achieve what Spark
> did in the JIRA? AFAIK Spark uses a
>
> From: Saisai Shao <sai.sai.s...@gmail.com>
> To: dev@livy.incubator.apache.org
> Date: 08/28/2017 06:53 PM
> Subject: Re: Update ASF GitHub Bot Settings?
> --
>
>
>
> Hi Alex,
>
> Do you want to achieve what Spark did in the JIRA? AFAIK Spark uses a
> script to sync between github and jira, it doesn't enable gihub robot to
> mirror everything from github.
>
> Thanks
> Jerry
>
> On Tue, Aug 29, 2017 at 5:00 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:
>
> > Opened an INFRA JIRA https://urldefense.proofpoint.
> com/v2/url?u=https-3A__issues.apache.org_jira_browse_INFRA-
> 2D14973=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_
> S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=HecI9o5Yex3-t-
> KdkHKD8f-oItRwZk_z7jkZ9L_Rp9s=aKdBIISGMsMO4cfi6VP9M2s4SNcUCq
> 2kpHytTTTjekg=
> >
> >
> > *Alex Bozarth*
> > Software Engineer
> > Spark Technology Center
> > --
> > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> > *GitHub: **github.com/ajbozarth* <https://urldefense.
> proofpoint.com/v2/url?u=https-3A__github.com_ajbozarth=
> DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-
> cx37DPYDyo=HecI9o5Yex3-t-KdkHKD8f-oItRwZk_z7jkZ9L_Rp9s=
> l0AeGk6t8q38vabAkHUe5zM4IkEfHujrrL2FROo2WbM= >
> >
> >
> > 505 Howard Street
> > San Francisco, CA 94105
> > United States
> >
> >
> >
> > [image: Inactive hide details for "Alex Bozarth" ---08/28/2017 12:08:36
> > PM---Is there a place where it describes the default and/or the]"Alex
> > Bozarth" ---08/28/2017 12:08:36 PM---Is there a place where it describes
> > the default and/or the options that I could use as reference whe
> >
> > From: "Alex Bozarth" <ajboz...@us.ibm.com>
> > To: dev@livy.incubator.apache.org
> > Date: 08/28/2017 12:08 PM
> > Subject: Re: Update ASF GitHub Bot Settings?
> > --
> >
> >
> >
> > Is there a place where it describes the default and/or the options that I
> > could use as reference when filing the JIRA?
> >
> > *Alex Bozarth*
> > Software Engineer
> > Spark Technology Center
> > --
> > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> > *GitHub: **github.com/ajbozarth*
> > <https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__github.com_ajbozarth=DwMFAg=jf_iaSHvJObTbx-siA1ZOg=S1_
> S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=flEt_
> Q08XzmKdKaGAHO3RTe5JvMKgEVjZKBtN1_TGiA=Ul0Q03YqdRvD21qPYqtc4jejt08ThO
> zUqaKEiaJlgjE=>
> >
> >
> > 505 Howard Street
> > San Francisco, CA 94105
> > United States
> >
> >
> >
> > Luciano Resende ---08/28/2017 12:06:37 PM---Please file an INFRA jira
> > describing you want to change the GitHub workflow. On Mon, Aug 28, 2017
> at
> >
> > From: Luciano Resende <luckbr1...@gmail.com>
> > To: dev@livy.incubator.apache.org
> > Date: 08/28/2017 12:06 PM
> > Subject: Re: Update ASF GitHub Bot Settings?
> > --
> >
>

Re: Update ASF GitHub Bot Settings?

[Saisai Shao]

Hi Alex,

Do you want to achieve what Spark did in the JIRA? AFAIK Spark uses a
script to sync between github and jira, it doesn't enable gihub robot to
mirror everything from github.

Thanks
Jerry

On Tue, Aug 29, 2017 at 5:00 AM, Alex Bozarth  wrote:

> Opened an INFRA JIRA https://issues.apache.org/jira/browse/INFRA-14973
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* 
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for "Alex Bozarth" ---08/28/2017 12:08:36
> PM---Is there a place where it describes the default and/or the]"Alex
> Bozarth" ---08/28/2017 12:08:36 PM---Is there a place where it describes
> the default and/or the options that I could use as reference whe
>
> From: "Alex Bozarth" 
> To: dev@livy.incubator.apache.org
> Date: 08/28/2017 12:08 PM
> Subject: Re: Update ASF GitHub Bot Settings?
> --
>
>
>
> Is there a place where it describes the default and/or the options that I
> could use as reference when filing the JIRA?
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth*
> 
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> Luciano Resende ---08/28/2017 12:06:37 PM---Please file an INFRA jira
> describing you want to change the GitHub workflow. On Mon, Aug 28, 2017 at
>
> From: Luciano Resende 
> To: dev@livy.incubator.apache.org
> Date: 08/28/2017 12:06 PM
> Subject: Re: Update ASF GitHub Bot Settings?
> --
>
>
>
> Please file an INFRA jira describing you want to change the GitHub
> workflow.
>
> On Mon, Aug 28, 2017 at 11:22 AM, Alex Bozarth 
> wrote:
>
> >
> >
> > Is there a way to edit the default ASF GitHub Bot settings? I think it's
> > great that it auto-comments on our JIRAs with links to new PRs, but it's
> > also auto-commenting on the JIRAs for every single comment on those PRs,
> > which clutters up both the JIRA and the Activity Stream (which is how I
> > personally keep up on the Livy JIRAs.
> >
> >
> >  Alex Bozarth
> >  Software Engineer
> >  Spark Technology Center
> >
> >
> >
> >
> >  E-mail: ajboz...@us.ibm.com
> >  GitHub: github.com/ajbozarth
> >505
> > Howard Street
> >  San
> > Francisco, CA 94105
> >
> >  United States
> >
> >
> >
> >
> >
> >
>
>
> --
> Luciano Resende
>
> *https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_lresende1975=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=K1uiNBsM4Wi-52dZ-0QOOhg_AUEd4fcpd9FOpKarXcQ=Hs8Np4fmjmGtSQm-ie9fPs579ISVlLeGjDUtLqyCaI8=*
> 
>
>
> *https://urldefense.proofpoint.com/v2/url?u=http-3A__lresende.blogspot.com_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=K1uiNBsM4Wi-52dZ-0QOOhg_AUEd4fcpd9FOpKarXcQ=_-mxqeV83wkvel4Y367vbHlPkKqkEM_xY6R1KfyysoE=*
> 
>
>
>
>
>
>
>

The process of Livy 0.4.0-incubating release

[Saisai Shao]

Hi mentors,

Would you please guide us on how to release the Livy 0.4.0-incubating,
including artifact publish.

Also regarding push artifacts to repo, are we inclining to change to Maven
central repo, or we still use Cloudera repo, any suggestion?

Besides, for Python package release, do we need to publish this python
client package to PIP or we don't need to do this now, any comment?

Thanks for your help and suggestion.

Best regards,
Saisai (Jerry)

Please vote for Apache Livy 0.4.0-incubating release

[Saisai Shao]

Hi teams,

Would you please help to vote for the Apache Livy release in general
incubator mail list. Thanks a lot.

http://mail-archives.apache.org/mod_mbox/incubator-general/201708.mbox/%3CCANvfmP8sAoofvRcs9AT5S-VW4LfiWeGfRXP2rhvEP4zyTag%2BYQ%40mail.gmail.com%3E

Thanks

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Saisai Shao]

Nan, I think Meisam already had a PR about this this, maybe you can discuss
with him on the github based on the proposed code.

Sorry I didn't follow the long discussion thread, but I think Paypal's
solution sounds simpler.

On Wed, Aug 23, 2017 at 12:07 AM, Nan Zhu  wrote:

> based on this result, I think we should follow the bulk operation pattern
>
> Shall we move forward with the PR from Paypal?
>
> Best,
>
> Nan
>
> On Mon, Aug 21, 2017 at 12:21 PM, Meisam Fathi 
> wrote:
>
> > Bottom line up front:
> > 1. The cost of calling 1 individual REST calls is about two order of
> > magnitude higher than calling a single batch REST call (1 * 0.05
> > seconds vs. 1.4 seconds)
> > 2. Time to complete a batch REST call plateaus at about 10,000
> application
> > reports per call.
> >
> > Full story:
> > I experimented and measure how long it takes to fetch Application Reports
> > from YARN with the REST API. My objective was to compare doing a batch
> REST
> > call to get all ApplicationReports vs doing individual REST calls for
> each
> > Application Report.
> >
> > I did the tests on 4 different cluster: 1) a test cluster, 2) a
> moderately
> > used dev cluster, 3) a lightly used production cluster, and 4) a heavily
> > used production cluster. For each cluster I made 7 REST call to get 1,
> 10,
> > 100, 1000, 1, 10, 100 application reports respectively. I
> > repeated each call 200 times to count for variations and I reported the
> > median time.
> > To measure the time, I used the following curl command:
> >
> > $ curl -o /dev/null -s -w "@curl-output-fromat.json" "http://
> > $rm_http_address:$rm_port/ws/v1/cluster/apps?applicationTypes=$
> > applicationTypes=$limit"
> >
> > The attached charts show the results. In all the charts, the x axis show
> > the number of results that were request in the call.
> > The bar chart show the time it takes to complete a REST call on each
> > cluster.
> > The first line plot also shows the same results as the bar chart on a log
> > scale (it is easier to see that the time to complete the REST call
> plateaus
> > at 10,000
> > The last chart shows the size of data that is being downloaded on each
> > REST call, which explains why the time plateaus  at 10,000.
> >
> >
> > [image: transfer_time_bar_plot.png][image: transfer_time_line_plot.png][
> image:
> > size_downloaded_line_plot.png]
> >
> >>
> >>
> > Thanks,
> > Meisam
> >
>

Re: [VOTE] Release Livy 0.4.0-incubating based on Livy 0.4.0 RC2

[Saisai Shao]

OK, sure, I will remove RC1 from the directory.

Thanks
Jerry

On Tue, Aug 22, 2017 at 7:24 PM, John D. Ament 
wrote:

> Hi,
>
> Looking at your release, it's confusing what we are voting on.  If RC2 is
> under vote, please remove RC1 from this directory.
>
> John
>
> On Tue, Aug 22, 2017 at 3:33 AM Jerry Shao  wrote:
>
> > Hello Incubator PMC’ers,
> >
> > The Apache Livy community has decided to release Apache Livy
> > 0.4.0-incubating based on 0.4.0-incubating Release Candidate 2. We now
> > kindly request the Incubator PMC members to review and vote on this
> > incubator
> > release.
> >
> > Livy is web service that exposes a REST interface for managing long
> running
> > Apache Spark contexts in your cluster. With Livy, new applications can be
> > built on top of Apache Spark that require fine grained interaction with
> > many Spark contexts.
> >
> > Artifacts are available at
> > https://dist.apache.org/repos/dist/dev/incubator/livy/, public keys are
> > available at https://dist.apache.org/repos/dist/dev/incubator/livy/KEYS.
> >
> > livy-0.4.0-incubating-src.zip <
> >
> > https://dist.apache.org/repos/dist/dev/incubator/livy/0.4.0-
> incubating/livy-0.4.0-incubating-src-RC2.zip
> > > is a source release. Along with it, for convenience, please find the
> > binary release as livy-0.4.0-incubating-bin-RC2.zip <
> >
> > https://dist.apache.org/repos/dist/dev/incubator/livy/0.4.0-
> incubating/livy-0.4.0-incubating-bin-RC2.zip
> > >.
> >
> >
> > Git tag:
> > *
> > https://github.com/apache/incubator-livy/releases/tag/
> v0.4.0-incubating-rc2
> > <
> > https://github.com/apache/incubator-livy/releases/tag/
> v0.4.0-incubating-rc2
> > >*
> >
> > The vote will be open for at least 72 hours or until necessary number of
> > votes are reached.
> >
> > Members please be sure to indicate "(Binding)" with your vote which will
> > help in tallying the vote(s).
> >
> > * Here is my +1 (non-binding) *
> >
> > Cheers,
> > Jerry
> >
>

Help to verify Apache Livy 0.4.0-incubating release

[Saisai Shao]

Hi all,

We're under progress to make a first Apache release of Livy
(0.4.0-incubating), we really hope you could verify the RC2[1] release
(binary and source) locally and return us the feedbacks.

We will call for an incubation vote next week if everything is fine.

Thanks a lot for your help.

[1]https://dist.apache.org/repos/dist/dev/incubator/livy/0.4.0-incubating/

Best regards,
Saisai (Jerry) Shao

Re: To release a first Apache version Livy

[Saisai Shao]

Thanks Marcelo for your comments. I checked create_release.sh in Spark, I
think I will adopt this after this first release, for this first release I
would like to do each step in person.

On Tue, Aug 8, 2017 at 8:34 AM, Marcelo Vanzin <van...@cloudera.com> wrote:

> Spark has a "create_release.sh" script, I wonder if that can be reused
> / adapted for Livy to make this easier in the future.
>
> I tracked all the dependencies' licenses for the incubation proposal,
> if that helps; although I didn't have the actual text of the licenses
> there.
>
> On Mon, Aug 7, 2017 at 5:29 PM, Luciano Resende <luckbr1...@gmail.com>
> wrote:
> > Just took a quick look at it, and here are some comments:
> >
> > Apache release requires source releases as described below
> > http://www.apache.org/legal/release-policy.html#source-packages
> >
> > Binary releases can also be provided as a convenience for users:
> > http://www.apache.org/legal/release-policy.html#compiled-packages
> >
> >
> > Also, for the binary artifact, I would list each included jar and it's
> > license type in the license file (see below as an example)
> > http://svn.apache.org/repos/asf/tuscany/sca-java-2.x/
> trunk/distribution/all/src/main/release/bin/LICENSE
> >
> > But note that, the source release artifact, which should not include any
> > dependency jars, should have the LICENSE file similar to what is in the
> > root of git today.
> >
> >
> >
> >
> >
> > On Mon, Aug 7, 2017 at 2:16 AM, Saisai Shao <sai.sai.s...@gmail.com>
> wrote:
> >
> >> Hi Team,
> >>
> >> Today I cut a RC1 release based on branch-0.4, here is the link (
> >> https://github.com/apache/incubator-livy/releases/tag/
> >> v0.4.0-incubating-rc1
> >> and https://dist.apache.org/repos/dist/dev/incubator/livy/), would you
> >> please help to test and verify. Thanks a lot and appreciate your help.
> >>
> >> Best regards,
> >> Saisai
> >>
> >> On Sat, Aug 5, 2017 at 10:44 AM, Alex Bozarth <ajboz...@us.ibm.com>
> wrote:
> >>
> >> > Last comment on the INFRA JIRA seemed to indicate that they hit a snag
> >> > with the import over a week ago and he never got back to us after. He
> >> told
> >> > us to keep using the Cloudera JIRA until he successfully completed a
> test
> >> > import then we could re-export for him.
> >> >
> >> >
> >> > *Alex Bozarth*
> >> > Software Engineer
> >> > Spark Technology Center
> >> > --
> >> > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> >> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
> >> >
> >> >
> >> > 505 Howard Street
> >> > San Francisco, CA 94105
> >> > United States
> >> >
> >> >
> >> >
> >> > [image: Inactive hide details for Bikas Saha ---08/04/2017 07:22:50
> >> > PM---Hi, Most of those jiras looked like bug fixes to me. Hence I
> t]Bikas
> >> > Saha ---08/04/2017 07:22:50 PM---Hi, Most of those jiras looked like
> bug
> >> > fixes to me. Hence I thought 0.4 could be a bug fix release.
> >> >
> >> > From: Bikas Saha <bi...@apache.org>
> >> > To: "dev@livy.incubator.apache.org" <dev@livy.incubator.apache.org>
> >> > Date: 08/04/2017 07:22 PM
> >> > Subject: Re: To release a first Apache version Livy
> >> > --
> >> >
> >> >
> >> >
> >> > Hi,
> >> >
> >> >
> >> > Most of those jiras looked like bug fixes to me. Hence I thought 0.4
> >> could
> >> > be a bug fix release. But I am ok releasing the current state so users
> >> can
> >> > gets an Apache release to transition to.
> >> >
> >> >
> >> > Given that its still a new project, a shorter cadence would help make
> bug
> >> > fix releases available.
> >> >
> >> >
> >> > Btw, does anyone know whats holding up the Apache jira process? If
> not, I
> >> > can follow up on that.
> >> >
> >> >
> >> > Bikas
> >> >
> >> > 
> >> > From: Saisai Shao <sai.sai.s...@gmail.com>
> >> > Sent: Thursday, August 3, 2017 7:31:10 PM
> >> > To: dev@livy.inc

Re: To release a first Apache version Livy

[Saisai Shao]

Thanks Luciano for your comments, I will do it today.

On Tue, Aug 8, 2017 at 8:29 AM, Luciano Resende <luckbr1...@gmail.com>
wrote:

> Just took a quick look at it, and here are some comments:
>
> Apache release requires source releases as described below
> http://www.apache.org/legal/release-policy.html#source-packages
>
> Binary releases can also be provided as a convenience for users:
> http://www.apache.org/legal/release-policy.html#compiled-packages
>
>
> Also, for the binary artifact, I would list each included jar and it's
> license type in the license file (see below as an example)
> http://svn.apache.org/repos/asf/tuscany/sca-java-2.x/
> trunk/distribution/all/src/main/release/bin/LICENSE
>
> But note that, the source release artifact, which should not include any
> dependency jars, should have the LICENSE file similar to what is in the
> root of git today.
>
>
>
>
>
> On Mon, Aug 7, 2017 at 2:16 AM, Saisai Shao <sai.sai.s...@gmail.com>
> wrote:
>
> > Hi Team,
> >
> > Today I cut a RC1 release based on branch-0.4, here is the link (
> > https://github.com/apache/incubator-livy/releases/tag/
> > v0.4.0-incubating-rc1
> > and https://dist.apache.org/repos/dist/dev/incubator/livy/), would you
> > please help to test and verify. Thanks a lot and appreciate your help.
> >
> > Best regards,
> > Saisai
> >
> > On Sat, Aug 5, 2017 at 10:44 AM, Alex Bozarth <ajboz...@us.ibm.com>
> wrote:
> >
> > > Last comment on the INFRA JIRA seemed to indicate that they hit a snag
> > > with the import over a week ago and he never got back to us after. He
> > told
> > > us to keep using the Cloudera JIRA until he successfully completed a
> test
> > > import then we could re-export for him.
> > >
> > >
> > > *Alex Bozarth*
> > > Software Engineer
> > > Spark Technology Center
> > > --
> > > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> > > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
> > >
> > >
> > > 505 Howard Street
> > > San Francisco, CA 94105
> > > United States
> > >
> > >
> > >
> > > [image: Inactive hide details for Bikas Saha ---08/04/2017 07:22:50
> > > PM---Hi, Most of those jiras looked like bug fixes to me. Hence I
> t]Bikas
> > > Saha ---08/04/2017 07:22:50 PM---Hi, Most of those jiras looked like
> bug
> > > fixes to me. Hence I thought 0.4 could be a bug fix release.
> > >
> > > From: Bikas Saha <bi...@apache.org>
> > > To: "dev@livy.incubator.apache.org" <dev@livy.incubator.apache.org>
> > > Date: 08/04/2017 07:22 PM
> > > Subject: Re: To release a first Apache version Livy
> > > --
> > >
> > >
> > >
> > > Hi,
> > >
> > >
> > > Most of those jiras looked like bug fixes to me. Hence I thought 0.4
> > could
> > > be a bug fix release. But I am ok releasing the current state so users
> > can
> > > gets an Apache release to transition to.
> > >
> > >
> > > Given that its still a new project, a shorter cadence would help make
> bug
> > > fix releases available.
> > >
> > >
> > > Btw, does anyone know whats holding up the Apache jira process? If
> not, I
> > > can follow up on that.
> > >
> > >
> > > Bikas
> > >
> > > 
> > > From: Saisai Shao <sai.sai.s...@gmail.com>
> > > Sent: Thursday, August 3, 2017 7:31:10 PM
> > > To: dev@livy.incubator.apache.org
> > > Subject: Re: To release a first Apache version Livy
> > >
> > > From my side 0.4 might be a feasible choice to release as for Apache.
> > > Reverting all the features and release 0.3.1 is too time-consuming and
> > not
> > > so necessary.
> > >
> > > On Fri, Aug 4, 2017 at 4:16 AM, Alex Bozarth <ajboz...@us.ibm.com>
> > wrote:
> > >
> > > > @Bikas The list of JIRAs fixed in 0.4 is 50 long (
> > > > https://issues.cloudera.org/issues/?jql=project%20%3D%
> > > > 20LIVY%20AND%20fixVersion%20%3D%200.4) so I'm wondering what you
> mean
> > by
> > > > not including feature work. Are you suggesting we revert some of the
> > work
> > > > for this release and the re-merge it later, or just that you would've
> > > > preferred i

Re: To release a first Apache version Livy

[Saisai Shao]

Hi Team,

Today I cut a RC1 release based on branch-0.4, here is the link (
https://github.com/apache/incubator-livy/releases/tag/v0.4.0-incubating-rc1
and https://dist.apache.org/repos/dist/dev/incubator/livy/), would you
please help to test and verify. Thanks a lot and appreciate your help.

Best regards,
Saisai

On Sat, Aug 5, 2017 at 10:44 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:

> Last comment on the INFRA JIRA seemed to indicate that they hit a snag
> with the import over a week ago and he never got back to us after. He told
> us to keep using the Cloudera JIRA until he successfully completed a test
> import then we could re-export for him.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Bikas Saha ---08/04/2017 07:22:50
> PM---Hi, Most of those jiras looked like bug fixes to me. Hence I t]Bikas
> Saha ---08/04/2017 07:22:50 PM---Hi, Most of those jiras looked like bug
> fixes to me. Hence I thought 0.4 could be a bug fix release.
>
> From: Bikas Saha <bi...@apache.org>
> To: "dev@livy.incubator.apache.org" <dev@livy.incubator.apache.org>
> Date: 08/04/2017 07:22 PM
> Subject: Re: To release a first Apache version Livy
> --
>
>
>
> Hi,
>
>
> Most of those jiras looked like bug fixes to me. Hence I thought 0.4 could
> be a bug fix release. But I am ok releasing the current state so users can
> gets an Apache release to transition to.
>
>
> Given that its still a new project, a shorter cadence would help make bug
> fix releases available.
>
>
> Btw, does anyone know whats holding up the Apache jira process? If not, I
> can follow up on that.
>
>
> Bikas
>
> 
> From: Saisai Shao <sai.sai.s...@gmail.com>
> Sent: Thursday, August 3, 2017 7:31:10 PM
> To: dev@livy.incubator.apache.org
> Subject: Re: To release a first Apache version Livy
>
> From my side 0.4 might be a feasible choice to release as for Apache.
> Reverting all the features and release 0.3.1 is too time-consuming and not
> so necessary.
>
> On Fri, Aug 4, 2017 at 4:16 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:
>
> > @Bikas The list of JIRAs fixed in 0.4 is 50 long (
> > https://issues.cloudera.org/issues/?jql=project%20%3D%
> > 20LIVY%20AND%20fixVersion%20%3D%200.4) so I'm wondering what you mean by
> > not including feature work. Are you suggesting we revert some of the work
> > for this release and the re-merge it later, or just that you would've
> > preferred it that was released without new features, but are okay with
> how
> > it is anyway? If you're worried about adding features in the first Apache
> > release should we also look at re-releasing 0.3.0 as 0.3.1-incubating or
> > back support, personally I think it's not worth it.
> >
> > As for release cadence, so far we've seemed to shoot for a 6 month
> cadence
> > (was longer this time with the move to Apache) but I'd be fine moving to
> a
> > 4 month cadence. I also prefer a time based release.
> >
> >
> > *Alex Bozarth*
> > Software Engineer
> > Spark Technology Center
> > --
> > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
> >
> >
> > 505 Howard Street
> > San Francisco, CA 94105
> > United States
> >
> >
> >
> > [image: Inactive hide details for Bikas Saha ---08/03/2017 12:56:59
> > PM---Hi, My preference would have been to not do a feature bearing]Bikas
> > Saha ---08/03/2017 12:56:59 PM---Hi, My preference would have been to not
> > do a feature bearing first release so that users could safe
> >
> > From: Bikas Saha <bi...@apache.org>
> > To: "dev@livy.incubator.apache.org" <dev@livy.incubator.apache.org>
> > Date: 08/03/2017 12:56 PM
> > Subject: Re: To release a first Apache version Livy
> > --
> >
> >
> >
> > Hi,
> >
> >
> > My preference would have been to not do a feature bearing first release
> so
> > that users could safely and painlessly migrate to the Apache release.
> > Adding features increases the risk of regressions etc.
> >
> >
> > However it seems like the web UI would be a relatively independent
> f

Re: To release a first Apache version Livy

[Saisai Shao]

>From my side 0.4 might be a feasible choice to release as for Apache.
Reverting all the features and release 0.3.1 is too time-consuming and not
so necessary.

On Fri, Aug 4, 2017 at 4:16 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:

> @Bikas The list of JIRAs fixed in 0.4 is 50 long (
> https://issues.cloudera.org/issues/?jql=project%20%3D%
> 20LIVY%20AND%20fixVersion%20%3D%200.4) so I'm wondering what you mean by
> not including feature work. Are you suggesting we revert some of the work
> for this release and the re-merge it later, or just that you would've
> preferred it that was released without new features, but are okay with how
> it is anyway? If you're worried about adding features in the first Apache
> release should we also look at re-releasing 0.3.0 as 0.3.1-incubating or
> back support, personally I think it's not worth it.
>
> As for release cadence, so far we've seemed to shoot for a 6 month cadence
> (was longer this time with the move to Apache) but I'd be fine moving to a
> 4 month cadence. I also prefer a time based release.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Bikas Saha ---08/03/2017 12:56:59
> PM---Hi, My preference would have been to not do a feature bearing]Bikas
> Saha ---08/03/2017 12:56:59 PM---Hi, My preference would have been to not
> do a feature bearing first release so that users could safe
>
> From: Bikas Saha <bi...@apache.org>
> To: "dev@livy.incubator.apache.org" <dev@livy.incubator.apache.org>
> Date: 08/03/2017 12:56 PM
> Subject: Re: To release a first Apache version Livy
> --
>
>
>
> Hi,
>
>
> My preference would have been to not do a feature bearing first release so
> that users could safely and painlessly migrate to the Apache release.
> Adding features increases the risk of regressions etc.
>
>
> However it seems like the web UI would be a relatively independent feature
> that would not affect the core stability. So it may be fine to include that
> in the first release as a new feature. In some ways, it gives users an
> incentive to move to the Apache release.
>
>
> +1 for getting the first release out as soon as feasible. And doing core
> feature work in follow up release.
>
>
> On this note, it would be good to consider the question of release
> cadence. Should we move to a 3 month or 4 month release cadence such that
> release trains are available for features to out when the features are
> ready. Or should do feature releases such that releases come out when major
> new functionality has added. My preference is a time based release cadence
> because it provides regular bug and security related fixes available in a
> released form for end users.
>
>
> Thanks!
>
> Bikas
>
>
> 
> From: Saisai Shao <sai.sai.s...@gmail.com>
> Sent: Monday, July 31, 2017 8:34:03 PM
> To: dev@livy.incubator.apache.org
> Subject: Re: To release a first Apache version Livy
>
> OK, thanks! Let's get this merged then prepare to the first Apache release.
>
> On Tue, Aug 1, 2017 at 11:30 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:
>
> > There's two open PRs on the livy repo with a follow-up PR on the website
> > repo:
> >
> > https://github.com/apache/incubator-livy/pull/25 <- Web UI Log Page
> > https://github.com/apache/incubator-livy/pull/26 <- Ability to build
> Livy
> > Docs
> > https://github.com/apache/incubator-livy-website/pull/7 <- Add Livy Docs
> > to Website (it may actually be better to update this and merge it after
> the
> > release)
> >
> > Once the Log Page PR is merged the basic Web UI is complete, the
> remaining
> > UI JIRAs are feature adds and tests that can come in the next release.
> >
> > *Alex Bozarth*
> > Software Engineer
> > Spark Technology Center
> > --
> > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
> >
> >
> > 505 Howard Street
> > San Francisco, CA 94105
> > United States
> >
> >
> >
> > [image: Inactive hide details for Saisai Shao ---07/31/2017 07:12:11
> > PM---Hi Alex, can you please list the JIRAs for UI related works y]Saisai
> > Shao ---07/31/2017 07:12:11 PM---Hi Alex, can you pleas

Re: To release a first Apache version Livy

[Saisai Shao]

Hi Team,

We're planning to release a first Apache version of Livy. Would you please
guide us the process of Apache incubating release, is there any doc to
follow. Thanks a lot!

Best regards
Saisai

On Tue, Aug 1, 2017 at 11:34 AM, Saisai Shao <sai.sai.s...@gmail.com> wrote:

> OK, thanks! Let's get this merged then prepare to the first Apache release.
>
> On Tue, Aug 1, 2017 at 11:30 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:
>
>> There's two open PRs on the livy repo with a follow-up PR on the website
>> repo:
>>
>> https://github.com/apache/incubator-livy/pull/25 <- Web UI Log Page
>> https://github.com/apache/incubator-livy/pull/26 <- Ability to build
>> Livy Docs
>> https://github.com/apache/incubator-livy-website/pull/7 <- Add Livy Docs
>> to Website (it may actually be better to update this and merge it after the
>> release)
>>
>> Once the Log Page PR is merged the basic Web UI is complete, the
>> remaining UI JIRAs are feature adds and tests that can come in the next
>> release.
>>
>> *Alex Bozarth*
>> Software Engineer
>> Spark Technology Center
>> --
>> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
>> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>>
>>
>> 505 Howard Street
>> San Francisco, CA 94105
>> United States
>>
>>
>>
>> [image: Inactive hide details for Saisai Shao ---07/31/2017 07:12:11
>> PM---Hi Alex, can you please list the JIRAs for UI related works y]Saisai
>> Shao ---07/31/2017 07:12:11 PM---Hi Alex, can you please list the JIRAs for
>> UI related works you want to merge in 0.4 release?
>>
>> From: Saisai Shao <sai.sai.s...@gmail.com>
>> To: dev@livy.incubator.apache.org
>> Date: 07/31/2017 07:12 PM
>> Subject: Re: To release a first Apache version Livy
>> --
>>
>>
>>
>> Hi Alex, can you please list the JIRAs for UI related works you want to
>> merge in 0.4 release?
>>
>> Thanks
>>
>> On Sat, Jul 29, 2017 at 7:59 AM, Alex Bozarth <ajboz...@us.ibm.com>
>> wrote:
>>
>> > After some tweaking and opening of PRs today, I'll change my vote to a
>> +1
>> > with the inclusion of my currently open PRs.
>> >
>> >
>> > *Alex Bozarth*
>> > Software Engineer
>> > Spark Technology Center
>> > --
>> > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
>> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>> >
>> >
>> > 505 Howard Street
>> > San Francisco, CA 94105
>> > United States
>> >
>> >
>> >
>> > [image: Inactive hide details for "Alex Bozarth" ---07/28/2017 01:01:16
>> > PM---I'm a +0 on this, I think we should get in a release soon,]"Alex
>> > Bozarth" ---07/28/2017 01:01:16 PM---I'm a +0 on this, I think we should
>> > get in a release soon, but I'm not sure if we should wait until
>> >
>> > From: "Alex Bozarth" <ajboz...@us.ibm.com>
>> > To: dev@livy.incubator.apache.org
>> > Date: 07/28/2017 01:01 PM
>> > Subject: Re: To release a first Apache version Livy
>> > --
>> >
>> >
>> >
>> > I'm a +0 on this, I think we should get in a release soon, but I'm not
>> > sure if we should wait until the Web UI and Documentation are finished
>> > (tracking in *LIVY-87* <https://issues.cloudera.org/browse/LIVY-87> and
>> > *LIVY-384* <https://issues.cloudera.org/browse/LIVY-384>). If others
>> are
>> > fine releasing with these feature partially complete then I'll be okay
>> as
>> > well.
>> >
>> > *Alex Bozarth*
>> > Software Engineer
>> > Spark Technology Center
>> > --
>> > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
>> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>> >
>> >
>> > 505 Howard Street
>> > San Francisco, CA 94105
>> > United States
>> >
>> >
>> >
>> > Jeff Zhang ---07/28/2017 01:12:24 AM---+1 for making the first apache
>> > release. Saisai Shao <sai.sai.s...@gmail.com>于2017年7月28日周五 下午3:42写道：
>> >
>> > From: Jeff Zhang <zjf...@gmail.com>
>> > To: dev@livy.incubator.apache.org
>> > Date: 07/28/2017 01:12 AM
>> > Subject: Re: To release a first Apache version Livy
>> > --
>> >
>> >
>> >
>> > +1 for making the first apache release.
>> >
>> >
>> > Saisai Shao <sai.sai.s...@gmail.com>于2017年7月28日周五 下午3:42写道：
>> >
>> > > Hi Team,
>> > >
>> > > We have already done most of the migration works to Apache, I think it
>> > > would be better to have a first Apache release based on the current
>> code.
>> > > What do you think?
>> > >
>> > > Thanks
>> > > Saisai
>> > >
>> >
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
>

Re: To release a first Apache version Livy

[Saisai Shao]

OK, thanks! Let's get this merged then prepare to the first Apache release.

On Tue, Aug 1, 2017 at 11:30 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:

> There's two open PRs on the livy repo with a follow-up PR on the website
> repo:
>
> https://github.com/apache/incubator-livy/pull/25 <- Web UI Log Page
> https://github.com/apache/incubator-livy/pull/26 <- Ability to build Livy
> Docs
> https://github.com/apache/incubator-livy-website/pull/7 <- Add Livy Docs
> to Website (it may actually be better to update this and merge it after the
> release)
>
> Once the Log Page PR is merged the basic Web UI is complete, the remaining
> UI JIRAs are feature adds and tests that can come in the next release.
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Saisai Shao ---07/31/2017 07:12:11
> PM---Hi Alex, can you please list the JIRAs for UI related works y]Saisai
> Shao ---07/31/2017 07:12:11 PM---Hi Alex, can you please list the JIRAs for
> UI related works you want to merge in 0.4 release?
>
> From: Saisai Shao <sai.sai.s...@gmail.com>
> To: dev@livy.incubator.apache.org
> Date: 07/31/2017 07:12 PM
> Subject: Re: To release a first Apache version Livy
> --
>
>
>
> Hi Alex, can you please list the JIRAs for UI related works you want to
> merge in 0.4 release?
>
> Thanks
>
> On Sat, Jul 29, 2017 at 7:59 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:
>
> > After some tweaking and opening of PRs today, I'll change my vote to a +1
> > with the inclusion of my currently open PRs.
> >
> >
> > *Alex Bozarth*
> > Software Engineer
> > Spark Technology Center
> > --
> > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
> >
> >
> > 505 Howard Street
> > San Francisco, CA 94105
> > United States
> >
> >
> >
> > [image: Inactive hide details for "Alex Bozarth" ---07/28/2017 01:01:16
> > PM---I'm a +0 on this, I think we should get in a release soon,]"Alex
> > Bozarth" ---07/28/2017 01:01:16 PM---I'm a +0 on this, I think we should
> > get in a release soon, but I'm not sure if we should wait until
> >
> > From: "Alex Bozarth" <ajboz...@us.ibm.com>
> > To: dev@livy.incubator.apache.org
> > Date: 07/28/2017 01:01 PM
> > Subject: Re: To release a first Apache version Livy
> > --
> >
> >
> >
> > I'm a +0 on this, I think we should get in a release soon, but I'm not
> > sure if we should wait until the Web UI and Documentation are finished
> > (tracking in *LIVY-87* <https://issues.cloudera.org/browse/LIVY-87> and
> > *LIVY-384* <https://issues.cloudera.org/browse/LIVY-384>). If others are
> > fine releasing with these feature partially complete then I'll be okay as
> > well.
> >
> > *Alex Bozarth*
> > Software Engineer
> > Spark Technology Center
> > --
> > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
> >
> >
> > 505 Howard Street
> > San Francisco, CA 94105
> > United States
> >
> >
> >
> > Jeff Zhang ---07/28/2017 01:12:24 AM---+1 for making the first apache
> > release. Saisai Shao <sai.sai.s...@gmail.com>于2017年7月28日周五 下午3:42写道：
> >
> > From: Jeff Zhang <zjf...@gmail.com>
> > To: dev@livy.incubator.apache.org
> > Date: 07/28/2017 01:12 AM
> > Subject: Re: To release a first Apache version Livy
> > --
> >
> >
> >
> > +1 for making the first apache release.
> >
> >
> > Saisai Shao <sai.sai.s...@gmail.com>于2017年7月28日周五 下午3:42写道：
> >
> > > Hi Team,
> > >
> > > We have already done most of the migration works to Apache, I think it
> > > would be better to have a first Apache release based on the current
> code.
> > > What do you think?
> > >
> > > Thanks
> > > Saisai
> > >
> >
> >
> >
> >
> >
> >
>
>
>
>

Re: To release a first Apache version Livy

[Saisai Shao]

Hi Alex, can you please list the JIRAs for UI related works you want to
merge in 0.4 release?

Thanks

On Sat, Jul 29, 2017 at 7:59 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:

> After some tweaking and opening of PRs today, I'll change my vote to a +1
> with the inclusion of my currently open PRs.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for "Alex Bozarth" ---07/28/2017 01:01:16
> PM---I'm a +0 on this, I think we should get in a release soon,]"Alex
> Bozarth" ---07/28/2017 01:01:16 PM---I'm a +0 on this, I think we should
> get in a release soon, but I'm not sure if we should wait until
>
> From: "Alex Bozarth" <ajboz...@us.ibm.com>
> To: dev@livy.incubator.apache.org
> Date: 07/28/2017 01:01 PM
> Subject: Re: To release a first Apache version Livy
> --
>
>
>
> I'm a +0 on this, I think we should get in a release soon, but I'm not
> sure if we should wait until the Web UI and Documentation are finished
> (tracking in *LIVY-87* <https://issues.cloudera.org/browse/LIVY-87> and
> *LIVY-384* <https://issues.cloudera.org/browse/LIVY-384>). If others are
> fine releasing with these feature partially complete then I'll be okay as
> well.
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> Jeff Zhang ---07/28/2017 01:12:24 AM---+1 for making the first apache
> release. Saisai Shao <sai.sai.s...@gmail.com>于2017年7月28日周五 下午3:42写道：
>
> From: Jeff Zhang <zjf...@gmail.com>
> To: dev@livy.incubator.apache.org
> Date: 07/28/2017 01:12 AM
> Subject: Re: To release a first Apache version Livy
> --
>
>
>
> +1 for making the first apache release.
>
>
> Saisai Shao <sai.sai.s...@gmail.com>于2017年7月28日周五 下午3:42写道：
>
> > Hi Team,
> >
> > We have already done most of the migration works to Apache, I think it
> > would be better to have a first Apache release based on the current code.
> > What do you think?
> >
> > Thanks
> > Saisai
> >
>
>
>
>
>
>

Re: Need input: Location of Livy Documentation

[Saisai Shao]

I'm in favor of store the docs in main repo, while put seldom changed
frameworks in website repo. I think you already mentioned the reason here (as
the Documentation grows and changes from version to version it would be
better to store it in the more regularly updated repo).

What I saw is that Spark put all the docs with version separated in here (
https://github.com/apache/spark-website/tree/asf-site/site/docs). So that
user could check docs with right version. I think in Livy we could also
maintain a such folder to put each release version's doc here (either
manually or by script). As for now (because we don't have a release) we
could create an "unreleased" folder to maintain docs in master branch, also
sync the docs periodically if anything changed.


On Wed, Jul 26, 2017 at 4:48 AM, Alex Bozarth  wrote:

>
>
> Hey Team
>
> Jerry and I have been discussing (on my recent PR) where we should keep the
> Livy Documentation files as we move them out of the README. My PR
> originally moved the two Doc files (on the REST and Programmatic APIs) into
> the livy website repos so we can display them via the website. Jerry
> proposed that we keep them in the main livy repo like how Spark does it.
> Personally given we have just two files I think leaving them in the website
> repo is simplier for us, but I also understand that as the Documentation
> grows and changes from version to version it would be better to store it in
> the more regularly updated repo.
>
> What are everyone's opinions on where the Docs should be stored? If we
> decide to store them in the main repo does anyone have pointers on how to
> simply and cleanly sync them to the website for viewing. You can see my
> Documenation and README update PRs here:
> https://github.com/apache/incubator-livy/pull/21
> https://github.com/apache/incubator-livy-website/pull/5
>
>
>  Alex Bozarth
>  Software Engineer
>  Spark Technology Center
>
>
>
>
>  E-mail: ajboz...@us.ibm.com
>  GitHub: github.com/ajbozarth
>505
> Howard Street
>  San
> Francisco, CA 94105
>
>  United States
>
>
>
>
>
>

Re: When Livy JIRA system will be ready?

[Saisai Shao]

Oh, thanks Alex!

On Mon, Jul 24, 2017 at 2:56 PM, Alex Bozarth <ajboz...@us.ibm.com> wrote:

> You can follow the progress on the INFRA JIRA: https://issues.apache.org/
> jira/browse/INFRA-14469
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Saisai Shao ---07/23/2017 11:32:03
> PM---Hi team, May I ask when Livy JIRA system in Apache will be re]Saisai
> Shao ---07/23/2017 11:32:03 PM---Hi team, May I ask when Livy JIRA system
> in Apache will be ready? Currently most of
>
> From: Saisai Shao <sai.sai.s...@gmail.com>
> To: dev@livy.incubator.apache.org
> Date: 07/23/2017 11:32 PM
> Subject: When Livy JIRA system will be ready?
> --
>
>
>
> Hi team,
>
> May I ask when Livy JIRA system in Apache will be ready? Currently most of
> the transition works are done except JIRA, contributors still need to
> create JIRAs on cloudera.org (which has very limited permission) and
> submit
> code to Apache.
>
> Thanks
> Saisai
>
>
>
>

Re: Commit style best practices, was Re: incubator-livy-website git commit: Fix bug in merge_livy_pr.py because incubator-livy-website repo has no branch it's name started with "branch-"

[Saisai Shao]

Sorry about it. Basically because I don't know whether JIRA is necessary
for incubator-livy-website repo, also where to create JIRA.

On Thu, Jul 20, 2017 at 1:20 PM, Alex Bozarth  wrote:

> +1. I think we've been using close to this format, but with "LIVY-XXX."
> instead of "[LIVY-XXX]". I can include a copy of this Contributing section
> in my next update to the livy website if we want, I almost added something
> similar in my first update, but decided not to since it hadn't been
> discussed yet.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* 
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Marcelo Vanzin ---07/20/2017 01:12:09
> PM---+1. Bad commit messages are one of my pet peeves. Although]Marcelo
> Vanzin ---07/20/2017 01:12:09 PM---+1. Bad commit messages are one of my
> pet peeves. Although I like periods at the end of sentences.
>
> From: Marcelo Vanzin 
> To: dev@livy.incubator.apache.org
> Date: 07/20/2017 01:12 PM
> Subject: Re: Commit style best practices, was Re: incubator-livy-website
> git commit: Fix bug in merge_livy_pr.py because incubator-livy-website repo
> has no branch it's name started with "branch-"
> --
>
>
>
> +1. Bad commit messages are one of my pet peeves. Although I like
> periods at the end of sentences.
>
> On Thu, Jul 20, 2017 at 1:09 PM, Luciano Resende 
> wrote:
> > Could we try to follow some best practices on PR title, commit message,
> etc?
> >
> > Some info from: http://bahir.apache.org/contributing/#Creating+a+Pull+
> > Request
> >
> > - Open a pull request against the master branch
> >
> >- The PR title should be of the form [LIVY-] Title, where
> LIVY-
> >is the relevant JIRA number and Title may be the JIRA’s title or a
> more
> >specific title describing the PR itself.
> >- If the pull request is still a work in progress, and so is not ready
> >to be merged, but needs to be pushed to Github to facilitate review,
> then
> >add [WIP] after the component.
> >- For website work, a JIRA is not required
> >
> > - Follow The 7 rules for a great commit message
> > 
> >
> >- Separate subject from body with a blank line
> >- Limit the subject line to 50 characters
> >- Capitalize the subject line
> >- Do not end the subject line with a period
> >- Use the imperative mood in the subject line
> >- Wrap the body at 72 characters
> >- Use the body to explain what and why vs. how
> >
> > Below is an example of a good commit message
> >
> > [LIVY-001] Performance enhancements for decision tree
> >
> > Generate Matrix with random values through local memory
> > if there is sufficient memory.
> >
> >
> >
> > Thoughts ?
> >
> > On Thu, Jul 20, 2017 at 1:03 PM,  wrote:
> >
> >> Repository: incubator-livy-website
> >> Updated Branches:
> >>   refs/heads/master 27348bab6 -> 572b37b1e
> >>
> >>
> >> Fix bug in merge_livy_pr.py because incubator-livy-website repo has no
> >> branch it's name started with "branch-"
> >>
> >>
> >> Project: http://git-wip-us.apache.org/repos/asf/incubator-livy-websit
> >> e/repo
> >> Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy-websit
> >> e/commit/572b37b1
> >> Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy-websit
> >> e/tree/572b37b1
> >> Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy-websit
> >> e/diff/572b37b1
> >>
> >> Branch: refs/heads/master
> >> Commit: 572b37b1efc2a2947272790261b6ba023ec53b74
> >> Parents: 27348ba
> >> Author: jerryshao 
> >> Authored: Thu Jul 20 13:02:23 2017 -0700
> >> Committer: jerryshao 
> >> Committed: Thu Jul 20 13:03:22 2017 -0700
> >>
> >> --
> >>  merge_livy_pr.py | 2 +-
> >>  1 file changed, 1 insertion(+), 1 deletion(-)
> >> --
> >>
> >>
> >> http://git-wip-us.apache.org/repos/asf/incubator-livy-websit
> >> e/blob/572b37b1/merge_livy_pr.py
> >> --
> >> diff --git a/merge_livy_pr.py b/merge_livy_pr.py
> >> index b527a29..7296aef 100755
> >> --- a/merge_livy_pr.py
> >> +++ b/merge_livy_pr.py
> >> @@ -359,7 +359,7 @@ def main():
> >>  original_head = get_current_ref()
> >>
> >>  branches = get_json("%s/branches" % GITHUB_API_BASE)
> >> -branch_names = filter(lambda x: x.startswith("branch-"), [x['name']
> >> for x in branches])
> >> +branch_names = [x['name'] for x in branches]
> >>  # Assumes branch names can be sorted lexicographically
> >>  latest_branch =