Re: [ANNOUNCE] Apache Livy 0.6.0-incubating released

[Marcelo Vanzin]

(Apologies to all, this is the same announcement as previously, but
including the missing disclaimer.)

The Apache Livy team is proud to announce the release of Apache Livy
0.6.0-incubating.

Livy is web service that exposes a REST interface for managing long
running Apache Spark contexts in your cluster. Livy enables
programmatic, fault-tolerant, multi-tenant submission of Spark jobs
from web/mobile apps (no Spark client needed). So, multiple users can
interact with your Spark cluster concurrently and reliably.

Download Apache Livy 0.6.0-incubating:
http://livy.incubator.apache.org/download/

Release Notes:
http://livy.incubator.apache.org/history/

For more about Livy check our website:
http://livy.incubator.apache.org/

We would like to thank the contributors that made the release possible!

=
*Disclaimer*

Apache Livy (incubating) is an effort undergoing incubation at The
Apache Software Foundation (ASF), sponsored by the name of Apache
Incubator PMC. Incubation is required of all newly accepted projects
until a further review indicates that the infrastructure,
communications, and decision making process have
stabilized in a manner consistent with other successful ASF projects.
While incubation status is not necessarily a reflection of the
completeness or stability of the code, it does indicate that the
project has yet to be fully endorsed by the ASF.

-- 
Marcelo

Re: [ANNOUNCE] Apache Livy 0.6.0-incubating released

[Marcelo Vanzin]

Oops, my bad. Must have missed it when I copy & pasted a previous
announcement. Will send an amended e-mail.

On Tue, Apr 2, 2019 at 12:03 PM sebb  wrote:
>
> On Tue, 2 Apr 2019 at 19:28, Marcelo Vanzin  
> wrote:
> >
> > The Apache Livy team is proud to announce the release of Apache Livy
> > 0.6.0-incubating.
> >
> > Livy is web service that exposes a REST interface for managing long
> > running Apache Spark contexts in your cluster. Livy enables
> > programmatic, fault-tolerant, multi-tenant submission of Spark jobs
> > from web/mobile apps (no Spark client needed). So, multiple users can
> > interact with your Spark cluster concurrently and reliably.
> >
> > Download Apache Livy 0.6.0-incubating:
> > http://livy.incubator.apache.org/download/
> >
> > Release Notes:
> > http://livy.incubator.apache.org/history/
> >
> > For more about Livy check our website:
> > http://livy.incubator.apache.org/
> >
> > We would like to thank the contributors that made the release possible!
>
> This is an incubator release, and should include the standard
> incubator disclaimer.
>
> See for example:
>
> https://lists.apache.org/thread.html/d94b89e6c4e83f2364607b47020ca715aaa7fae1a3b97b374b93ca7f@%3Cgeneral.incubator.apache.org%3E
> https://lists.apache.org/thread.html/ec3d6b90a2b0054eb2f0a2e4ba2e09a594463720e2b28e75b315c693@%3Cgeneral.incubator.apache.org%3E
> https://lists.apache.org/thread.html/8993ecb33e0f73e69faabba8c1f07d3a9965a178df1698c42fc9f4b2@%3Cgeneral.incubator.apache.org%3E
>
> Thanks.



-- 
Marcelo

Re: [VOTE] Release Livy 0.6.0 based on RC2

[Marcelo Vanzin]

Ok, we have 3 binding +1, no 0s or -1s. Vote passes, I'll follow up on
the general@ list.

On Tue, Mar 19, 2019 at 2:58 PM Marcelo Vanzin  wrote:
>
> Err. Should have read "based on RC2". Too fast on the send button!
>
> Anyway, here's my +1.
>
> Note this build does includes the pre-compiled jars for the thrift server.
>
> On Tue, Mar 19, 2019 at 2:56 PM Marcelo Vanzin  wrote:
> >
> > This vote is for releasing Livy 0.6.0 based on RC2.
> >
> > The vote will be open at least 72 hours until Friday March 21st, 22:00 UTC 
> > and
> > will pass with a minimum of 3 +1 binding votes and a majority of positive 
> > votes.
> >
> > [+1] This release is ready to send to the Incubator PMC for approval
> >
> > [-1] This release is not ready because...
> >
> > This vote is held according to Apache Incubator release policy:
> > https://incubator.apache.org/policy/incubation.html#releases
> >
> > The RC is based on tag v0.6.0-incubating-rc2:
> > https://github.com/apache/incubator-livy/commit/28be98cabc
> >
> > The release files can be found here:
> > https://dist.apache.org/repos/dist/dev/incubator/livy/0.6.0-incubating-rc2/
> >
> > The staged maven artifacts can be found here:
> > https://repository.apache.org/content/repositories/orgapachelivy-1008
> >
> > The list of resolved JIRAs in this release can be found here:
> > https://issues.apache.org/jira/projects/LIVY/versions/12342736
> >
> >
> > --
> > Marcelo
>
>
>
> --
> Marcelo



-- 
Marcelo

[VOTE RESULT] Release Livy 0.6.0 based on RC2

[Marcelo Vanzin]

Vote passes. Thanks for all who helped with the release!

+1 votes (* = binding):
Jeff Zhang (*)
Marco Gaido (*)
Marcelo Vanzin (*)

No 0 or -1 votes were cast.

-- 
Marcelo

Re: [VOTE] Release Livy 0.6.0 based on RC2

[Marcelo Vanzin]

Hi,

Sorry, but as I mentioned in the other thread, I haven't used Livy in
a while and I don't really have time to investigate things past the
existing tests. Anyone is free to do it, though.

I'd rather not fail a release because of a problem in the thrift
server; it's new and "experimental", so I'd expect issues.

On Wed, Mar 20, 2019 at 2:07 AM Marco Gaido  wrote:
>
> Hi Marcelo,
>
> thanks for starting the vote. I tried downloading the zip and enabling the
> thriftserver, but connecting with beeline and running a query throws an
> exception to me:
>
> 0: jdbc:hive2://localhost:10090/> create table foo as select a.* from
> (select 1, "2") a;
>
> Error: java.lang.RuntimeException: java.util.NoSuchElementException:
> Statement ce95f6bb-bf51-4fca-b61d-0d40272b3a33 not found in session
> 774fed7a-8388-43fd-978f-8d2bd31c4a12.
>
> org.apache.livy.thriftserver.session.ThriftSessionState.statementNotFound(ThriftSessionState.java:118)
>
> org.apache.livy.thriftserver.session.ThriftSessionState.cleanupStatement(ThriftSessionState.java:107)
>
>
> This may also be caused by some problems in my local environment though,
> since the tests on Travis are passing. May someone else try if they face
> the same issue anyway? In the config I just enabled the thriftserver with
> the proper config.
> Thanks,
> Marco
>
> Il giorno mar 19 mar 2019 alle ore 22:58 Marcelo Vanzin
>  ha scritto:
>
> > Err. Should have read "based on RC2". Too fast on the send button!
> >
> > Anyway, here's my +1.
> >
> > Note this build does includes the pre-compiled jars for the thrift server.
> >
> > On Tue, Mar 19, 2019 at 2:56 PM Marcelo Vanzin 
> > wrote:
> > >
> > > This vote is for releasing Livy 0.6.0 based on RC2.
> > >
> > > The vote will be open at least 72 hours until Friday March 21st, 22:00
> > UTC and
> > > will pass with a minimum of 3 +1 binding votes and a majority of
> > positive votes.
> > >
> > > [+1] This release is ready to send to the Incubator PMC for approval
> > >
> > > [-1] This release is not ready because...
> > >
> > > This vote is held according to Apache Incubator release policy:
> > > https://incubator.apache.org/policy/incubation.html#releases
> > >
> > > The RC is based on tag v0.6.0-incubating-rc2:
> > > https://github.com/apache/incubator-livy/commit/28be98cabc
> > >
> > > The release files can be found here:
> > >
> > https://dist.apache.org/repos/dist/dev/incubator/livy/0.6.0-incubating-rc2/
> > >
> > > The staged maven artifacts can be found here:
> > > https://repository.apache.org/content/repositories/orgapachelivy-1008
> > >
> > > The list of resolved JIRAs in this release can be found here:
> > > https://issues.apache.org/jira/projects/LIVY/versions/12342736
> > >
> > >
> > > --
> > > Marcelo
> >
> >
> >
> > --
> > Marcelo
> >



-- 
Marcelo

Re: [VOTE] Release Livy 0.6.0 based on RC2

[Marcelo Vanzin]

Err. Should have read "based on RC2". Too fast on the send button!

Anyway, here's my +1.

Note this build does includes the pre-compiled jars for the thrift server.

On Tue, Mar 19, 2019 at 2:56 PM Marcelo Vanzin  wrote:
>
> This vote is for releasing Livy 0.6.0 based on RC2.
>
> The vote will be open at least 72 hours until Friday March 21st, 22:00 UTC and
> will pass with a minimum of 3 +1 binding votes and a majority of positive 
> votes.
>
> [+1] This release is ready to send to the Incubator PMC for approval
>
> [-1] This release is not ready because...
>
> This vote is held according to Apache Incubator release policy:
> https://incubator.apache.org/policy/incubation.html#releases
>
> The RC is based on tag v0.6.0-incubating-rc2:
> https://github.com/apache/incubator-livy/commit/28be98cabc
>
> The release files can be found here:
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.6.0-incubating-rc2/
>
> The staged maven artifacts can be found here:
> https://repository.apache.org/content/repositories/orgapachelivy-1008
>
> The list of resolved JIRAs in this release can be found here:
> https://issues.apache.org/jira/projects/LIVY/versions/12342736
>
>
> --
> Marcelo



-- 
Marcelo

[VOTE] Release Livy 0.6.0 based on RC1

[Marcelo Vanzin]

This vote is for releasing Livy 0.6.0 based on RC2.

The vote will be open at least 72 hours until Friday March 21st, 22:00 UTC and
will pass with a minimum of 3 +1 binding votes and a majority of positive votes.

[+1] This release is ready to send to the Incubator PMC for approval

[-1] This release is not ready because...

This vote is held according to Apache Incubator release policy:
https://incubator.apache.org/policy/incubation.html#releases

The RC is based on tag v0.6.0-incubating-rc2:
https://github.com/apache/incubator-livy/commit/28be98cabc

The release files can be found here:
https://dist.apache.org/repos/dist/dev/incubator/livy/0.6.0-incubating-rc2/

The staged maven artifacts can be found here:
https://repository.apache.org/content/repositories/orgapachelivy-1008

The list of resolved JIRAs in this release can be found here:
https://issues.apache.org/jira/projects/LIVY/versions/12342736


-- 
Marcelo

Re: [VOTE] Release Livy 0.6.0 based on RC1

[Marcelo Vanzin]

FYI I'm cancelling this vote in favor of rc2.

On Mon, Mar 18, 2019 at 2:01 PM Marcelo Vanzin  wrote:
>
> This vote is for releasing Livy 0.6.0 based on RC1.
>
> The vote will be open at least 72 hours until Thursday March 21st, 21:00 UTC 
> and
> will pass with a minimum of 3 +1 binding votes and a majority of positive 
> votes.
>
> [+1] This release is ready to send to the Incubator PMC for approval
>
> [-1] This release is not ready because...
>
> This vote is held according to Apache Incubator release policy:
> https://incubator.apache.org/policy/incubation.html#releases
>
> The RC is based on tag v0.6.0-incubating-rc1:
> https://github.com/apache/incubator-livy/commit/b8861bbc3ad9aa0b7069fe1d7ae4a390bd422cb3
>
> The release files can be found here:
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.6.0-incubating-rc1/
>
> The staged maven artifacts can be found here:
> https://repository.apache.org/content/repositories/orgapachelivy-1006
>
> The list of resolved JIRAs in this release can be found here:
> https://issues.apache.org/jira/projects/LIVY/versions/12342736
>
> --
> Marcelo



-- 
Marcelo

Re: [VOTE] Release Livy 0.6.0 based on RC1

[Marcelo Vanzin]

I played with this a bit; found out it's generated by a maven plugin
enabled during the packaging.

The plugin seems to list everything; if a library is multi-licensed,
it will list the library multiple times. It doesn't seem there's a way
to choose which license to pick for a library, and that's probably
intentional.

The plugin also looks through all transitive dependencies, which is
probably where weirdness like jackson 1.8 comes from. But disabling
transitive dependencies makes the list incomplete (e.g. it won't list
jackson 1.9, which is actually packaged in Livy because Hadoop
libraries use it).

So I'm not sure there's much we can do unless someone wants to start
manually maintaining this stuff. I'm not signing up for that.

On Mon, Mar 18, 2019 at 6:27 PM Marcelo Vanzin  wrote:
>
> I'll double check tomorrow, but I checked 3 or 4 of the ones you
> listed and they're all dual licensed, one of them being the *GPL
> license and the other being an ASF-friendly one.
>
> On Mon, Mar 18, 2019 at 5:45 PM Luciano Resende  wrote:
> >
> > On Mon, Mar 18, 2019 at 4:03 PM Marcelo Vanzin
> >  wrote:
> > >
> > > On Mon, Mar 18, 2019 at 2:23 PM Luciano Resende  
> > > wrote:
> > > > Also, there are lots of version mismatching between the included jars
> > > > and the third-party license file
> > >
> > > Could you be a little more explicit about this one? I don't even see a
> > > 3rd party license file. What exactly is the expectation here?
> > >
> > > --
> > > Marcelo
> >
> > When you extract the binary distro, there is a THIRD-PARTY file which
> > lists the external dependencies and their licenses.
> >
> > I tried to validate some and noticed a few mismatches on the version:
> >
> > jars/jackson-jaxrs-1.9.13.jar
> > * JAX-RS provider for JSON content type
> > (org.codehaus.jackson:jackson-jaxrs:1.8.3 -
> > http://jackson.codehaus.org)
> >
> > Also, I just noticed the following (L)GPL dependencies. Are these new
> > ? Were it introduced since the previous release? Or are these
> > dual-licensed?
> >
> >
> >   GNU General Public Library:
> >
> > * Streaming API for XML (javax.xml.stream:stax-api:1.0-2 - no url 
> > defined)
> >
> >   GNU Lesser General Public License (LGPL), Version 2.1:
> >
> > * JAX-RS provider for JSON content type
> > (org.codehaus.jackson:jackson-jaxrs:1.8.3 -
> > http://jackson.codehaus.org)
> > * Xml Compatibility extensions for Jackson
> > (org.codehaus.jackson:jackson-xc:1.8.3 - http://jackson.codehaus.org)
> >
> >   GPL:
> >
> > * JTransforms (com.github.rwl:jtransforms:2.4.0 -
> > http://sourceforge.net/projects/jtransforms/)
> >
> >   GPL2 w/ CPE:
> >
> > * JAXB API bundle for GlassFish V3 (javax.xml.bind:jaxb-api:2.2.2
> > - https://jaxb.dev.java.net/)
> > * JAXB RI (com.sun.xml.bind:jaxb-impl:2.2.3-1 - http://jaxb.java.net/)
> > * javax.ws.rs-api (javax.ws.rs:javax.ws.rs-api:2.0.1 -
> > http://jax-rs-spec.java.net)
> > * jersey-client (com.sun.jersey:jersey-client:1.9 -
> > https://jersey.java.net/jersey-client/)
> > * jersey-core (com.sun.jersey:jersey-core:1.9 -
> > https://jersey.java.net/jersey-core/)
> > * jersey-guice (com.sun.jersey.contribs:jersey-guice:1.9 -
> > https://jersey.java.net/jersey-contribs/jersey-guice/)
> > * jersey-json (com.sun.jersey:jersey-json:1.9 -
> > https://jersey.java.net/jersey-json/)
> > * jersey-server (com.sun.jersey:jersey-server:1.9 -
> > https://jersey.java.net/jersey-server/)
> >
> >   GPLv2+CE:
> >
> > * JavaMail API (compat) (javax.mail:mail:1.4.7 -
> > http://kenai.com/projects/javamail/mail)
> >
> >   LGPL:
> >
> > * JTransforms (com.github.rwl:jtransforms:2.4.0 -
> > http://sourceforge.net/projects/jtransforms/)
> >
> >   LGPL 2.1:
> >
> > * Javassist (org.javassist:javassist:3.18.1-GA - 
> > http://www.javassist.org/)
> >
> >
> >
> >
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
>
>
>
> --
> Marcelo



-- 
Marcelo

Re: [VOTE] Release Livy 0.6.0 based on RC1

[Marcelo Vanzin]

On Mon, Mar 18, 2019 at 2:23 PM Luciano Resende  wrote:
> Also, there are lots of version mismatching between the included jars
> and the third-party license file

Could you be a little more explicit about this one? I don't even see a
3rd party license file. What exactly is the expectation here?

-- 
Marcelo

Re: Making travis logs less verbose

[Marcelo Vanzin]

If you know how to silence messages from the setup phase (apt / pip /
git), go for it. Those seem kinda hidden by Travis, but maybe there's
a setting I'm not familiar with.

Maven also has a -B option that makes things a little less verbose in
non-interactive terminals. I think -quiet might be a little overkill.

On Thu, Feb 7, 2019 at 2:40 PM Meisam Fathi  wrote:
>
> Each build on travis generates 10K+ lines of log. Should we make build
> commands less verbose by passing --quiet to them?
>
> As an example, apt-get installs and pip installs generate 3K+ lines on
> their own. Maven generates another 6K+ lines of log, but I am not sure if
> scilencing maven is a good idea. Passing --quiet to Maven silences scalac
> warnging.
>
> Having said all of that, should we make travis logs less verbose? If yes, I
> can send a PR.
>
> Thanks,
> Meisam



-- 
Marcelo

[VOTE] Move source repo to gitbox

[Marcelo Vanzin]

Creating a formal vote for this. I don't think we have a choice but
they seem to request a vote anyway. This will be lazy consensus, I
plan to create an infra ticket for the migration on Monday if the vote
passes.

Starting with my +1.


-- 
Marcelo

Re: GitHub emails on dev

[Marcelo Vanzin]

Maybe infra did something to re-enable it. We asked them to disable it
way back (INFRA-14973). Maybe reopen that ticket or file a new one.
On Mon, Oct 8, 2018 at 4:19 PM Alex Bozarth  wrote:
>
>
>
> Is there a reason we've begun getting GitHub PR emails on the dev list? The
> regular glut of emails for every comment on every PR probably shouldn't be
> CC'd to dev@
>
>
>  Alex Bozarth
>  Software Engineer
>  Center for Open-Source Data & AI Technologies
>
>
>
>
>  E-mail: ajboz...@us.ibm.com
>  GitHub: github.com/ajbozarth
>505 Howard 
> Street
>  San Francisco, 
> CA 94105
>United 
> States
>
>
>
>
>


-- 
Marcelo

Re: Missing board report

[Marcelo Vanzin]

Hi Justin,

Sorry, it wasn't intentional, it was probably just not in the original
report I based my update on.

I guess we just haven't been asking anything from the mentors, so we
should just mention that in the report?

On Tue, Oct 2, 2018 at 3:40 PM Justin Mclean  wrote:
>
> HI,
>
> I notice you didn't answer this question:
>
> Have your mentors been helpful and responsive or are things falling
> through the cracks? In the latter case, please list any open issues
> that need to be addressed.
>
> And removed it from your report. Any reason why?
>
> Thanks,
> Justin



-- 
Marcelo

Re: Missing board report

[Marcelo Vanzin]

I'm pretty sure I don't have access to the wiki; but I took the last report
and added a small blurb about what's been going on. Feel free to update the
wiki on my behalf.

===

Livy is web service that exposes a REST interface for managing long running
Apache Spark contexts in your cluster. With Livy, new applications can be
built on top of Apache Spark that require fine grained interaction with many
Spark contexts.

Livy has been incubating since 2017-06-05.

Three most important issues to address in the move towards graduation:

  1. Grow the community to have more activities.
  2. Grow more contributors and committers.
  3. Regular release.


Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

None


How has the community developed since the last report?

Activity on the dev and user mailing lists as well as the rate of reported
JIRA issues has stayed steady. There has been an uptick of development
activity around adding a new feature to the project (a Hive-compatible
Thrift
server implementation).


How has the project developed since the last report?

A new feature (Hive-compatible Thrift server) is being implemented, and some
code to support legacy releases of Java, Scala and Spark has been removed.


How would you assess the podling's maturity?
Please feel free to add your own commentary.

  [ ] Initial setup
  [ ] Working towards first release
  [X] Community building
  [ ] Nearing graduation
  [ ] Other:

Date of last release:

  2018-02-06

When were the last committers or PPMC members elected?

  2017-09-18

Signed-off-by:

  [ ](livy) Bikas Saha
 Comments:
  [ ](livy) Brock Noland
 Comments:
  [ ](livy) Luciano Resende
 Comments:
  [ ](livy) Jean-Baptiste Onofre
 Comments:


On Tue, Oct 2, 2018 at 2:13 PM Alex Bozarth  wrote:

> @marcelo and @jerry you've been reviewing and merging the most recent code
> changes as I've been a bit too busy. I know we've added a thrift server in
> the last quarter, could one of you fill out the report this time? If I
> don't hear back by end of work day then I'll just fill it out to the best
> of my understanding.
>
>
> *Alex Bozarth*
> Software Engineer
> Center for Open-Source Data & AI Technologies
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* 
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Justin Mclean ---10/02/2018 01:54:50
> PM---HI, Just a reminder that the report is due today and I don']Justin
> Mclean ---10/02/2018 01:54:50 PM---HI, Just a reminder that the report is
> due today and I don't see any discussion on it. It helps me f
>
> From: Justin Mclean 
> To: 
> Date: 10/02/2018 01:54 PM
> Subject: Re: Missing board report
> --
>
>
>
> HI,
>
> Just a reminder that the report is due today and I don't see any
> discussion on it. It helps me finish the IPMC report on time if you get
> your report in on time.
>
> Thanks,
> Justin
>
>
>
>
>

-- 
Marcelo

Re: [DISCUSS] Getting rid of old stuff

[Marcelo Vanzin]

Older versions of Spark are supported on J8, so we could still support them.

But it doesn't seem like there's a desire to support older Spark
anyway, so great! I'll file a few bugs tomorrow and clean that up.
That'll make my patch for LIVY-503 simpler, too.
On Thu, Sep 13, 2018 at 6:44 PM Saisai Shao  wrote:
>
> +1，
>
> To support J8, Spark 2.2+ might be the only option. I'm not sure if those
> vendors will continue to support Spark 2.2-, but IMHO for the community
> release, I think moving forward would be a better choice.
>
> Thanks
> Saisai
>
>
> Jeff Zhang  于2018年9月14日周五 上午9:39写道：
>
> > +1
> >
> >
> > Alex Bozarth 于2018年9月14日周五 上午6:26写道：
> >
> > > I agree with all of Marcelo's points. The last time we discussed this was
> > > when Spark 2.2 was new and it was decided that it was probably too soon,
> > > but that was awhile ago now. I've been in support of deprecating and
> > > removing support for older versions of Java/Scala/Spark for a while and I
> > > believe it will allow us to clean up and unify large portions of our
> > code.
> > >
> > >
> > > *Alex Bozarth*
> > > Software Engineer
> > > Center for Open-Source Data & AI Technologies
> > > --
> > > *E-mail:* *ajboz...@us.ibm.com* 
> > > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
> > >
> > >
> > > 505 Howard Street
> > > <
> > https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g
> > >
> > > San Francisco, CA 94105
> > > <
> > https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g
> > >
> > > United States
> > > <
> > https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g
> > >
> > >
> > >
> > >
> > > [image: Inactive hide details for Marcelo Vanzin ---09/13/2018 03:10:28
> > > PM---Hey all, I'd like to gauge people's reaction to some propo]Marcelo
> > > Vanzin ---09/13/2018 03:10:28 PM---Hey all, I'd like to gauge people's
> > > reaction to some proposals regarding what
> > >
> > > From: Marcelo Vanzin 
> > > To: dev@livy.incubator.apache.org
> > > Date: 09/13/2018 03:10 PM
> > > Subject: [DISCUSS] Getting rid of old stuff
> > > --
> > >
> > >
> > >
> > >
> > > Hey all,
> > >
> > > I'd like to gauge people's reaction to some proposals regarding what
> > > is supported in Livy.
> > >
> > > #1: Java versions
> > >
> > > I propose dropping support for Java 7. Even J8 is already EOL,
> > > although it's pretty obvious nobody is getting rid of it anytime soon.
> > > But I don't see a good reason to support J7. Even testing it is a
> > > nuisance, since most people only have jdk8 around...
> > >
> > > #2: Spark versions
> > >
> > > I think we should drop 1.6. At least. This would clean up some code
> > > that currently uses reflection, and fix some parts of the API (like
> > > the JobContext method to retrieve a "SparkSession" instance).
> > >
> > > Changing the API is well, not backwards compatible, but I think it's
> > > better to do that sort of cleanup earlier than later when the project
> > > is more mature.
> > >
> > > Separately, we could consider dropping 2.0 and 2.1 also. There was
> > > talk in the Spark list about making 2.1 EOL - I don't remember a final
> > > verdict, but I don't imagine there will be a lot of new deployments of
> > > 2.1 going forward, since the only reason to use it is if you're stuck
> > > with J7.
> > >
> > > #3: Scala versions
> > >
> > > If we decide to only support Spark 2.2+, then the decision is easy.
> > > But if we drop 1.6, we should consider dropping Scala 2.10 support.
> > > Spark does not ship official builds with 2.10 support in the 2.x line,
> > > and it was dropped altogether in 2.2.
> > >
> > > We shouldn't remove support for multiple versions of Scala, though,
> > > since 2.12 will be beta in Spark 2.4, and there will be a 2.13 at some
> > > point.
> > >
> > > Thoughts?
> > >
> > >
> > > --
> > > Marcelo
> > >
> > >
> > >
> > >
> > >
> >



-- 
Marcelo

[DISCUSS] Getting rid of old stuff

[Marcelo Vanzin]

Hey all,

I'd like to gauge people's reaction to some proposals regarding what
is supported in Livy.

#1: Java versions

I propose dropping support for Java 7. Even J8 is already EOL,
although it's pretty obvious nobody is getting rid of it anytime soon.
But I don't see a good reason to support J7. Even testing it is a
nuisance, since most people only have jdk8 around...

#2: Spark versions

I think we should drop 1.6. At least. This would clean up some code
that currently uses reflection, and fix some parts of the API (like
the JobContext method to retrieve a "SparkSession" instance).

Changing the API is well, not backwards compatible, but I think it's
better to do that sort of cleanup earlier than later when the project
is more mature.

Separately, we could consider dropping 2.0 and 2.1 also. There was
talk in the Spark list about making 2.1 EOL - I don't remember a final
verdict, but I don't imagine there will be a lot of new deployments of
2.1 going forward, since the only reason to use it is if you're stuck
with J7.

#3: Scala versions

If we decide to only support Spark 2.2+, then the decision is easy.
But if we drop 1.6, we should consider dropping Scala 2.10 support.
Spark does not ship official builds with 2.10 support in the 2.x line,
and it was dropped altogether in 2.2.

We shouldn't remove support for multiple versions of Scala, though,
since 2.12 will be beta in Spark 2.4, and there will be a 2.13 at some
point.

Thoughts?


-- 
Marcelo

Re: A new link request to my project and one question

[Marcelo Vanzin]

Superusers are a little more than "allowed to impersonate others". I
don't remember exactly what are the things that it allows, but it
would be better to add finer grained permissions.

On Mon, Jun 25, 2018 at 6:30 PM, Saisai Shao  wrote:
> Yes, has a configuration "livy.superusers". Here in this case, the sql
> server user should be added as a superuser, who can impersonate other
> different users.
>
> Marcelo Vanzin  于2018年6月26日周二 上午9:12写道：
>
>> You're talking about another service between the user and the application.
>>
>> In that case a parameter probably makes sense. But then you'd need to
>> add those config options, because this is a dangerous feature, and
>> Livy should know who is allowed to impersonate who. In this case the
>> service needs to authenticate to Livy as a privileged user, and Livy's
>> configuration would say that the service's user is allowed to
>> impersonate certain users or groups (same as the other services that
>> allow impersonation like YARN).
>>
>>
>> On Mon, Jun 25, 2018 at 5:41 PM, Takeshi Yamamuro 
>> wrote:
>> > Yea, I know the Livy supports impersonation.
>> > I assume a case blow
>> > [different users] ---Some protocols---> [the server applications managing
>> > multiple sessions for users] ---REST---> [Livy server]
>> > In this case, Livy already has a way to pass proxyUser from the
>> application
>> > to Livy?
>> > Sorry, but I'm not familiar with Livy internal logic.
>> >
>> >
>> > On Tue, Jun 26, 2018 at 9:14 AM Marcelo Vanzin
>> 
>> > wrote:
>> >
>> >> On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro <
>> linguin@gmail.com>
>> >> wrote:
>> >> > In that case, I think Livy is useful; the application can pass
>> proxyUser
>> >> to
>> >> > build LivyClient for each user
>> >> > and run spark queries as each user authorization.
>> >>
>> >> But Livy already supports impersonation. It can impersonate the
>> >> authenticated user.
>> >>
>> >> You're suggesting adding a parameter so the user can request
>> >> impersonation of some specific user, which is a different thing. What
>> >> is the use case for that?
>> >>
>> >> --
>> >> Marcelo
>> >>
>> >
>> >
>> > --
>> > ---
>> > Takeshi Yamamuro
>>
>>
>>
>> --
>> Marcelo
>>



-- 
Marcelo

Re: A new link request to my project and one question

[Marcelo Vanzin]

You're talking about another service between the user and the application.

In that case a parameter probably makes sense. But then you'd need to
add those config options, because this is a dangerous feature, and
Livy should know who is allowed to impersonate who. In this case the
service needs to authenticate to Livy as a privileged user, and Livy's
configuration would say that the service's user is allowed to
impersonate certain users or groups (same as the other services that
allow impersonation like YARN).


On Mon, Jun 25, 2018 at 5:41 PM, Takeshi Yamamuro  wrote:
> Yea, I know the Livy supports impersonation.
> I assume a case blow
> [different users] ---Some protocols---> [the server applications managing
> multiple sessions for users] ---REST---> [Livy server]
> In this case, Livy already has a way to pass proxyUser from the application
> to Livy?
> Sorry, but I'm not familiar with Livy internal logic.
>
>
> On Tue, Jun 26, 2018 at 9:14 AM Marcelo Vanzin 
> wrote:
>
>> On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro 
>> wrote:
>> > In that case, I think Livy is useful; the application can pass proxyUser
>> to
>> > build LivyClient for each user
>> > and run spark queries as each user authorization.
>>
>> But Livy already supports impersonation. It can impersonate the
>> authenticated user.
>>
>> You're suggesting adding a parameter so the user can request
>> impersonation of some specific user, which is a different thing. What
>> is the use case for that?
>>
>> --
>> Marcelo
>>
>
>
> --
> ---
> Takeshi Yamamuro



-- 
Marcelo

Re: A new link request to my project and one question

[Marcelo Vanzin]

On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro  wrote:
> In that case, I think Livy is useful; the application can pass proxyUser to
> build LivyClient for each user
> and run spark queries as each user authorization.

But Livy already supports impersonation. It can impersonate the
authenticated user.

You're suggesting adding a parameter so the user can request
impersonation of some specific user, which is a different thing. What
is the use case for that?

-- 
Marcelo

Re: A new link request to my project and one question

[Marcelo Vanzin]

re: proxy user, you have to be extremely careful with that.

Livy currently supports proxy user, but for the server only. It allows
the server to impersonate anyone, so that sessions can run as the
requesting user.

If you let the user decide who the session will be run as, you'll need
to add configuration, just as those available in HDFS, YARN, etc, to
tell Livy which users can impersonate which other users. Otherwise
you're basically making authentication meaningless.


On Thu, Jun 14, 2018 at 7:36 PM, Saisai Shao  wrote:
> Sure, I will merge the website code, thanks!
>
> For proxyUser thing, I think there's no particular reason not adding it,
> maybe we just forgot to add the proxyUser support.
>
> It would be better if you could create a JIRA to track this issue. If
> you're familiar with Livy code, you can also submit a PR about it.
>
> Thanks
> Jerry
>
> Takeshi Yamamuro  于2018年6月15日周五 上午7:33写道：
>
>> Hi, Livy dev,
>>
>> I opened a new pr in incubator-livy-website to add a new link in
>> third-party-projects.md. It'd be great if you could check this;
>> https://github.com/apache/incubator-livy-website/pull/23
>>
>> Btw, I have one question; currently, we cannot pass proxyUser
>> in LivyClientBuilder. Any reason not to add code for that?
>> I know we can handle this in an application side by adding a bit code like
>>
>> https://github.com/maropu/spark-sql-server/blob/master/sql/sql-server/src/main/java/org/apache/livyclient/common/CreateClientRequestWithProxyUser.java
>> But, If Livy itself supported this, it'd be nice to me.
>>
>> Best,
>> takeshi
>>
>> --
>> ---
>> Takeshi Yamamuro
>>



-- 
Marcelo

Re: spark-submit command

[Marcelo Vanzin]

On Fri, Mar 9, 2018 at 4:08 PM, Meisam Fathi  wrote:
> If Livy is compiled against a particular major version
> of Spark (say 2.1.0), it cannot run interactive sessions on a different
> Spark version (say 1.6). I am interested to know how we can get around this
> restriction.

It would require a lot of changes to the build, but the main thing to
do is to compile the rsc module against different versions of scala /
spark and packaging all of those in the same livy distribution.

The server itself doesn't really care what version of Spark is being run.

-- 
Marcelo

Re: spark-submit command

[Marcelo Vanzin]

On Fri, Mar 9, 2018 at 1:36 AM, Matteo Durighetto
 wrote:
>   I think it's correct that the Livy Admin manages the multiple
> version of spark, but the user need to choose what version to use
> to submit the job.
...
> So a Livy Admin could manage the configuration and a Datascience or a Dev
> could submit the job calling the "alias" ( i.e. spak_1.6 or spark_2.1 or
> spark 2.2 ) to a different spark / java
> and test different env for they applications or machine learning project.

That sounds closer to what I had in mind originally. User asks for a
specific version of Spark using a name defined by the admin, instead
of providing an explicit SPARK_HOME env variable or something like
that.


-- 
Marcelo

Re: how to assign issue to myself?

[Marcelo Vanzin]

If I remember how the tracker was set up, only Livy committers can
assign bugs. I believe we follow a model similar to Spark's where
issues are only assigned after the PR associated with them is merged.

On Mon, Feb 5, 2018 at 5:55 AM, Albert Cheng  wrote:
> According to the document in `https://livy.incubator.apache.org/community/`.I 
> have subscribed dev@livy.incubator.apache.org, but I cannot assign issue to 
> myself still.



-- 
Marcelo

Re: [VOTE] Livy 0.5.0-incubating (RC2)

[Marcelo Vanzin]

I'm a -0 on this.

The signatures check and things seem to compile, but I can't get unit
tests to pass.

The default profile (spark 1.6) breaks with jacoco errors for me.

The spark 2.2 profile is a bit better; but I still run into what look
like flaky tests (TestSparkClient fails sometimes, also
SessionHeartbeatSpec).

I don't want to block the release because of those, since maybe it's
something in my environment (haven't really kept up with the
requirements), but it sounds a little odd for unit tests to fail in
the default profile.


On Thu, Jan 25, 2018 at 3:57 PM, Alex Bozarth  wrote:
>
>
> Hey team,
>
> Now that the security vulnerability that was found has been addressed, RC2
> of Livy 0.5.0-incubating is ready for testing and approval.
>
>
> The vote will be open 72 hours until Monday January 29th at 12:00AM UTC and
> will pass with a minimum of 3 +1 votes and a majority of positive votes.
>
> [+1] This release is ready to send to the Incubator PMC for approval
>
> [-1] This release is not ready because...
>
> This vote is held according to Apache Incubator release policy:
> https://incubator.apache.org/policy/incubation.html#releases
>
> This release files can be found here:
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.5.0-incubating-rc2/
>
> The list of resolved JIRAs in this release can be found here:
> https://issues.apache.org/jira/projects/LIVY/versions/12341543
>
>
> I will also include my own +1 here
>
>
>
>  Alex Bozarth
>  Software Engineer
>  Spark Technology Center
>
>
>
>
>  E-mail: ajboz...@us.ibm.com
>  GitHub: github.com/ajbozarth
>505 Howard 
> Street
>  San Francisco, 
> CA 94105
>United 
> States
>
>
>
>
>



-- 
Marcelo

Re: 0.5-incubating release timing

[Marcelo Vanzin]

Just added both of you as admins. Admin away.

On Tue, Jan 16, 2018 at 2:17 PM, Bikas Saha  wrote:
> Looks like I am also not an admin on the Project. Let me see if I can fix the 
> released label.
>
> Bikas
> 
> From: Alex Bozarth 
> Sent: Tuesday, January 16, 2018 12:32 PM
> To: dev@livy.incubator.apache.org
> Subject: 0.5-incubating release timing
>
>
>
> Hey team,
>
> So with LIVY-104 wrapping up today and LIVY-141/LIVY-175 wrapping up later
> this week I don't see any other open PRs that are aimed for 0.5. With that
> in mind, do we want to plan on cutting RC1 at the end of the week?
>
> On a related note, 0.4 is still not marked as "released" in the Apache
> JIRA, could someone with Admin privileges can update that?
>
>
>  Alex Bozarth
>  Software Engineer
>  Spark Technology Center
>
>
>
>
>  E-mail: ajboz...@us.ibm.com
>  GitHub: github.com/ajbozarth
>505 Howard 
> Street
>  San Francisco, 
> CA 94105
>United 
> States
>
>
>
>
>



-- 
Marcelo

Re: [DISCUSS] Add new "channels" to interact with the managed cluster and extend support cluster mode

[Marcelo Vanzin]

Seems like one of the proposals (channels) isn't really tied to the
other (supporting more than Spark). I don't really have an opinion on
the latter (other than nobody has shown enough interest to even try to
prototype anything).

For the former, why do you need these separate "channels"? What
different functionality would they support that you could not achieve
via REST? Any new transport brings a whole lot of baggage with it -
you need to make sure things like encryption and authentication work
properly over it, at the very least. Then there's issue of clients.
Then you have to support all apps over all transport types or things
get confusing. So I'd like to understand exactly what's the advantage
here before opening up that Pandora's box.

On Mon, Nov 13, 2017 at 6:17 AM, Jean-Baptiste Onofré  wrote:
> Hi guys,
>
> Currently, Livy provides a great "façade" as REST service to interact and
> deploy with a Spark cluster.
>
> I wonder if we could not extend this to other "channels" than REST. I'm
> thinking about leveraging Apache Camel and others for that (why not
> interacting with MQTT/AMQP, ...).
> In terms of architecture, the Livy server could "expose" the cluster via
> different channels (in term of wording).
>
> On the other hand, we are mostly focus on Spark cluster today. It could be
> interesting to extend this to other execution engines (Flink, Gearpump,
> Apex, ...) and backend (Yarn, Kubernetes, ...).
>
> The reason for those ideas is that basically I'm thinking about a Apache
> Beam pipelines dispenser powered partly by Livy ;)
>
> Thoughts ?
>
> Regards
> JB
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com



-- 
Marcelo

Re: Can anyone get email about livy jira change

[Marcelo Vanzin]

Actually I think I have permission to fix this. Let me just copy what
Spark does.

On Wed, Sep 13, 2017 at 11:10 AM, Marcelo Vanzin <van...@cloudera.com> wrote:
> On Wed, Sep 13, 2017 at 11:08 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:
>> I think he was talking about getting email notifications from the Apache 
>> JIRA when someone comments on or changes a Livy JIRA.
>
> Yes, that's what the issues@ list is supposed to be. It's how I get
> updates for Spark and other ASF projects I monitor.
>
> If that's not working, file an INFRA ticket. Might be good to update
> the web site too.
>
>
> --
> Marcelo



-- 
Marcelo

Re: Can anyone get email about livy jira change

[Marcelo Vanzin]

On Wed, Sep 13, 2017 at 11:08 AM, Alex Bozarth  wrote:
> I think he was talking about getting email notifications from the Apache JIRA 
> when someone comments on or changes a Livy JIRA.

Yes, that's what the issues@ list is supposed to be. It's how I get
updates for Spark and other ASF projects I monitor.

If that's not working, file an INFRA ticket. Might be good to update
the web site too.


-- 
Marcelo

Re: Can anyone get email about livy jira change

[Marcelo Vanzin]

Are you subscribed to iss...@livy.incubator.apache.org?

I just found out I wasn't, so I'm not sure it's actually working. It's
not listed on the web site, so easy to miss.

On Wed, Sep 13, 2017 at 2:26 AM, Jeff Zhang  wrote:
> Hi all,
>
> I can get email notification about jira changes of other apache projects,
> but never receive email about livy ticket.  Can anyone get email about livy
> jira change ? Or does it need some configuration on the jira system ?



-- 
Marcelo

Re: user defined sessionId / URI for Livy sessions

[Marcelo Vanzin]

BTW take my comment there with a grain of salt; at the time Livy was being
targeted at mostly being hidden from users, making such a feature not make
much sense in Livy itself. But things may have changed since then,
especially as people started using it more.

On Mon, Sep 11, 2017 at 1:51 PM, Alex Bozarth  wrote:

> I would agree with Marcelo's comment the JIRA that this isn't a good
> feature for livy, but I'll take a look at your impl if you open a PR and
> see if it changes my mind.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* 
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Meisam Fathi ---09/11/2017 10:23:49
> AM---+ dev Is there any interest in adding this feature to Livy?]Meisam
> Fathi ---09/11/2017 10:23:49 AM---+ dev Is there any interest in adding
> this feature to Livy? I can send a PR
>
> From: Meisam Fathi 
> To: "u...@livy.incubator.apache.org" , "
> dev@livy.incubator.apache.org" 
> Date: 09/11/2017 10:23 AM
> Subject: Re: user defined sessionId / URI for Livy sessions
> --
>
>
>
> + dev
> Is there any interest in adding this feature to Livy? I can send a PR
>
> Ideally, it would be helpful if we could mint a session ID with a PUT
> > request, something like PUT /sessions/foobar, where "foobar" is the newly
> > created sessionId.
> >
> > I suggest we make session names unique and nonnumeric values (to
> guarantee
> a session name does not clash with another session name or session ID).
>
> Design doc:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_meisam_incubator-2Dlivy_wiki_Design-2Ddoc-2Dfor-
> 2DLivy-2D41-3A-2DAccessing-2Dsessions-2Dby-2Dname=
> DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-
> cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=
> xTiY52FMWMdTRgCmiNRWe6yEoCchxKNxQrYPEkPupbw=
> JIRA ticket: https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_LIVY-2D41=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_
> S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-
> juQkf5Q2FMYtA5kv5sqiMM=lFed2hYlDA_wUo94RWUAw7N01lSN368P-ABmP_npWrM=
>
>
> Thanks,
> Meisam
>
>
>
>


-- 
Marcelo

Re: Commit permission

[Marcelo Vanzin]

I don't think Alex has an account; which means he'd probably have to
sign an ICLA before that happens. Not sure whether he's still around.

On Tue, Sep 5, 2017 at 10:44 AM, Luciano Resende <luckbr1...@gmail.com> wrote:
> Based on the Livy project proposal, I just added the following as PPMC
> members (couldn't find Alex Man):
>
>
>-
>
>Marcelo Vanzin (van...@cloudera.com)
>-
>
>Jeff Zhang (zjf...@gmail.com)
>-
>
>Saisai Shao (ss...@hortonworks.com)
>-
>
>Kostas Sakellis (kos...@cloudera.com)
>
>
> On Tue, Sep 5, 2017 at 9:59 AM, Marcelo Vanzin <van...@cloudera.com> wrote:
>
>> I looked at Whimsy but couldn't find the list of committers...
>> https://whimsy4.apache.org/roster/ppmc/livy
>>
>> I this is just a matter of opening an INFRA ticket to add you, I can
>> do it. But I'm not really super familiar with the process.
>>
>> On Mon, Sep 4, 2017 at 8:15 PM, Jeff Zhang <zjf...@gmail.com> wrote:
>> > Can anyone help to check whether I have commit permission, I am trying to
>> > commit PR 43, but fails due to permission issue.   My apache mail is :
>> > zjf...@apache.org
>> >
>> >
>> > Writing objects: 100% (12/12), 870 bytes | 0 bytes/s, done.
>> >
>> > Total 12 (delta 4), reused 0 (delta 0)
>> >
>> > remote: You are not authorized to edit this repository.
>> >
>> > remote:
>> >
>> > To https://git-wip-us.apache.org/repos/asf/incubator-livy.git
>> >
>> >  ! [remote rejected] PR_TOOL_MERGE_PR_43_MASTER -> master (pre-receive
>> hook
>> > declined)
>> >
>> > error: failed to push some refs to '
>> > https://git-wip-us.apache.org/repos/asf/incubator-livy.git'
>> >
>> > Restoring head pointer to master
>> >
>> > git checkout master
>>
>>
>>
>> --
>> Marcelo
>>
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/



-- 
Marcelo

Re: Commit permission

[Marcelo Vanzin]

I looked at Whimsy but couldn't find the list of committers...
https://whimsy4.apache.org/roster/ppmc/livy

I this is just a matter of opening an INFRA ticket to add you, I can
do it. But I'm not really super familiar with the process.

On Mon, Sep 4, 2017 at 8:15 PM, Jeff Zhang  wrote:
> Can anyone help to check whether I have commit permission, I am trying to
> commit PR 43, but fails due to permission issue.   My apache mail is :
> zjf...@apache.org
>
>
> Writing objects: 100% (12/12), 870 bytes | 0 bytes/s, done.
>
> Total 12 (delta 4), reused 0 (delta 0)
>
> remote: You are not authorized to edit this repository.
>
> remote:
>
> To https://git-wip-us.apache.org/repos/asf/incubator-livy.git
>
>  ! [remote rejected] PR_TOOL_MERGE_PR_43_MASTER -> master (pre-receive hook
> declined)
>
> error: failed to push some refs to '
> https://git-wip-us.apache.org/repos/asf/incubator-livy.git'
>
> Restoring head pointer to master
>
> git checkout master



-- 
Marcelo

Re: The process of Livy 0.4.0-incubating release

[Marcelo Vanzin]

On Sun, Aug 27, 2017 at 11:53 PM, Saisai Shao  wrote:
> Also regarding push artifacts to repo, are we inclining to change to Maven
> central repo, or we still use Cloudera repo, any suggestion?

As others have said, things have to be published to maven central
(although I don't know how to do that myself). Even if it made sense,
you wouldn't be able to publish to Cloudera's repos.

-- 
Marcelo

Re: [VOTE] Release Livy 0.4.0-incubating based on Livy 0.4.0 RC2

[Marcelo Vanzin]

Just echoing my +1 from the Livy list.

On Tue, Aug 22, 2017 at 12:33 AM, Jerry Shao  wrote:
> Hello Incubator PMC’ers,
>
> The Apache Livy community has decided to release Apache Livy
> 0.4.0-incubating based on 0.4.0-incubating Release Candidate 2. We now
> kindly request the Incubator PMC members to review and vote on this incubator
> release.
>
> Livy is web service that exposes a REST interface for managing long running
> Apache Spark contexts in your cluster. With Livy, new applications can be
> built on top of Apache Spark that require fine grained interaction with
> many Spark contexts.
>
> Artifacts are available at
> https://dist.apache.org/repos/dist/dev/incubator/livy/, public keys are
> available at https://dist.apache.org/repos/dist/dev/incubator/livy/KEYS.
>
> livy-0.4.0-incubating-src.zip <
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.4.0-incubating/livy-0.4.0-incubating-src-RC2.zip
>> is a source release. Along with it, for convenience, please find the
> binary release as livy-0.4.0-incubating-bin-RC2.zip <
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.4.0-incubating/livy-0.4.0-incubating-bin-RC2.zip
>>.
>
>
> Git tag:
> *https://github.com/apache/incubator-livy/releases/tag/v0.4.0-incubating-rc2
> *
>
> The vote will be open for at least 72 hours or until necessary number of
> votes are reached.
>
> Members please be sure to indicate "(Binding)" with your vote which will
> help in tallying the vote(s).
>
> * Here is my +1 (non-binding) *
>
> Cheers,
> Jerry



-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin]

I like that approach on paper, although I currently don't have much
time to actually be able to review the PR and provide decent feedback.

I think that regardless of the approach, one goal should be to
probably separate what is being monitored from how it's being
monitored; that way you can later change the monitoring code to be
smarter without having to change the rest of the code that calls into
it. I remember reviewing this code when it first was submitted and it
could definitely use some refactoring.

> we have Livy Multi-Node HA i.e livy running on 6 servers for each cluster,

I'm not really familiar with how multi-node HA was implemented (I
stopped at session recovery), but why isn't a single server doing the
update and storing the results in ZK? Unless it's actually doing
load-balancing, it seems like that would avoid multiple servers having
to hit YARN.



On Wed, Aug 16, 2017 at 4:18 PM, Prabhu Kasinathan
 wrote:
> As Meisam highlighted, in our case, we have Livy Multi-Node HA i.e livy
> running on 6 servers for each cluster, load-balanced, sharing livy metadata
> on zookeeper and running thousands of applications. With below changes, we
> are seeing good improvements due to batching the requests (one per livy
> node) instead of each livy node making multiple requests. Please review the
> changes and let us know if improvements needed or we are open to explore
> other alternative option if works.
>
>> We are making one big request to get ApplicationReports, Then we make an
>> individual + thread pool request to get the tracking URL, Spark UI URL,
>> YARN diagnostics, etc for each application separately. For our cluster
>> settings and our workloads, one big request turned out to be a better
>> solution. But we were limited to the API provided in YarnClient. With the
>> home-made REST client a separate request is not needed and that can change
>> the whole equation.
>
>
>
> On Wed, Aug 16, 2017 at 3:33 PM, Meisam Fathi 
> wrote:
>
>>
>> On Wed, Aug 16, 2017 at 2:09 PM Nan Zhu  wrote:
>>
>>> With time goes, the reply from YARN can only be larger and larger. Given
>>> the consistent workload pattern, the cost of a large query can be
>>> eventually larger than individual request
>>>
>>
>> I am under the impression that there is a limit to the number of reports
>> that YARN retains, which is set by 
>> yarn.resourcemanager.max-completed-applications
>> in yarn.xml and defaults to 10,000. But I could be wrong about the
>> semantics of yarn.resourcemanager.max-completed-applications.
>>
>> I would say go with individual request + thread pool  or large batch for
>>> all first, if any performance issue is observed, add the optimization on
>>> top of it
>>>
>>
>> We are making one big request to get ApplicationReports, Then we make an
>> individual + thread pool request to get the tracking URL, Spark UI URL,
>> YARN diagnostics, etc for each application separately. For our cluster
>> settings and our workloads, one big request turned out to be a better
>> solution. But we were limited to the API provided in YarnClient. With the
>> home-made REST client a separate request is not needed and that can change
>> the whole equation.
>>
>> @Prabhu, can you chime in?
>>
>>
>>> However, even with rest API, there are some corner cases, e.g. a
>>> long running app lasting for days (training some models), and some short
>>> ones which last only for minutes
>>>
>>
>> We are running Spark streaming jobs on Livy that virtually run for ever.
>>
>> Thanks,
>> Meisam
>>



-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin]

On Wed, Aug 16, 2017 at 12:57 PM, Nan Zhu  wrote:
> yes, we finally converge on the idea
>
> how large the reply can be? if I have only one running applications and I
> still need to fetch 1000
>
> on the other side
>
> I have 1000 running apps, what's the cost of sending 1000 requests even the
> thread pool and yarn client are shared?

I don't know the answers, but I'm asking you, since you are proposing
the design, to consider that as an option, since it does not seem like
you considered that tradeoff when suggesting your current approach.

My comments about filtering are targeted at making things better in
your first case; if there's really only one app being monitored, and
you can figure out a filter that returns let's say 50 apps instead of
1000 that may be monitored by YARN, then you can do that.

Or maybe you can go with a hybrid approach, where you use individual
requests but past a certain threshold you fall back to bulk requests
to avoid overloading YARN.

Again, I'm asking you to consider alternatives that are not mentioned
in your design document, because I identified potential performance
issues in the current approach.


-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin]

On Wed, Aug 16, 2017 at 12:27 PM, Nan Zhu  wrote:
> I am using your words *current*. What's the definition of "current" in
> livy? I think that's all application which still keep some records in the
> livy's process's memory space

There are two views of what is current: Livy's and YARN's. They may
not be the same.

>From your reply below, you seem to want to query YARN for the state of
applications that are current to Livy. There's no API for that, as you
said. But that is not what I'm talking about.

I'm saying that Livy should query YARN for YARN's current view of what
applications exist, and then match those against its own view.

Again, it's all a question about what is cheaper: a single request to
YARN that results in a large reply, parts of which Livy will ignore
because it's not interested in the data, or hundreds of small requests
to YARN polling specific applications?

> 1. How you express this "current" in a query to YARN? I think you have to
> use ApplicationID (maybe there are some other ways) in a query
>
> 2. The problem is that I didn't see such an API to make such a "big call"
> by passing in all applications's IDs


-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin]

On Wed, Aug 16, 2017 at 11:34 AM, Nan Zhu  wrote:
> Yes, I know there is such an API, what I don't understand is what I should
> pass in the filtering API you mentioned, say we query YARN for every 5
> tickets
>
> 0: Query and get App A is running
>
> 4: App A is done
>
> 5: Query...so what I should fill as filtering parameters at 5 get capture
> the changes of App A's state?

You don't query for app state *changes*. You query for the current app
state, and compare against what you have, and then you can detect
changes that way. The trick is how to filter to get the information
you want, so you limit how much data you request from YARN.

I'm not aware of any YARN API to query for state changes like that. So
even in the individual request case, you'd have to get app A's state,
and update the Livy handle if the state has changed from what was
previously know.

That's most probably why Meisam's PR only filters by app type. If
there are further filters than can be applied, then great, but you
still need logic in Livy to detect the state changes you want.

> If you look at Meisam's PR, they can only filter based on appType
> https://github.com/apache/incubator-livy/pull/36/files#diff-a3f879755cfe10a678cc08ddbe60a4d3R75


-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin]

On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi  wrote:
> I do agree that actor based design is cleaner and more maintainable. But we
> had to discard it because it adds more dependencies to Livy.

I've been reading "actor system" as a design pattern, not as
introducing a new dependency to Livy.

If the document is actually proposing using Akka (instead of just
using Akka as an example of an actor system implementation), then I'm
a -1 on that.

-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin]

On Wed, Aug 16, 2017 at 9:06 AM, Nan Zhu  wrote:
>> I'm not really sure what you're talking about here, since I did not
> suggest a "shared data structure", and I'm not really sure what that
> means in this context.
>
> What you claimed is just monitoring/updating the state with a single thread
> *given* all applications have been there.

What I proposed is having a single request to YARN to get all
applications' statuses, if that's possible. You'd still have multiple
application handles that are independent of each other. They'd all be
updated separately from that one thread talking to YARN.

This has nothing to do with a "shared data structure". There's no
shared data structure here to track application status.

>> Yes. While there are applications that need monitoring, you poll YARN
> at a constant frequency. Basically what would be done by multiple
> threads, but there's a single one.
>
> Did you find the bulk API?

No, but I suggested that you look whether that exists since I think
that's a better solution both from YARN and Livy's perspectives, since
it requires less resources. It should at least be mentioned as an
alternative in your mini-spec and, if it doesn't work for whatever
reason, deserves an explanation.

>> Why not. The expensive part is not parsing results, I'll bet, but
> having a whole bunch of different tasks opening and closing YARN
> connections.
>
> First, YARNClient is thread safe and can be shared by multiple threads

Irrelevant.

> Second, If I have 1000 applications, what's your expectation to the
> following cases
>
> 1. YARN processed request for 999 and failed on the last one for some reason
>
> 2. Livy received 999 well-formatted response but get 1 malformed response

What if YARN goes down? What if your datacenter has a massive power failure?

You have to handle errors in any scenario.


-- 
Marcelo

Re: To release a first Apache version Livy

[Marcelo Vanzin]

Spark has a "create_release.sh" script, I wonder if that can be reused
/ adapted for Livy to make this easier in the future.

I tracked all the dependencies' licenses for the incubation proposal,
if that helps; although I didn't have the actual text of the licenses
there.

On Mon, Aug 7, 2017 at 5:29 PM, Luciano Resende  wrote:
> Just took a quick look at it, and here are some comments:
>
> Apache release requires source releases as described below
> http://www.apache.org/legal/release-policy.html#source-packages
>
> Binary releases can also be provided as a convenience for users:
> http://www.apache.org/legal/release-policy.html#compiled-packages
>
>
> Also, for the binary artifact, I would list each included jar and it's
> license type in the license file (see below as an example)
> http://svn.apache.org/repos/asf/tuscany/sca-java-2.x/trunk/distribution/all/src/main/release/bin/LICENSE
>
> But note that, the source release artifact, which should not include any
> dependency jars, should have the LICENSE file similar to what is in the
> root of git today.
>
>
>
>
>
> On Mon, Aug 7, 2017 at 2:16 AM, Saisai Shao  wrote:
>
>> Hi Team,
>>
>> Today I cut a RC1 release based on branch-0.4, here is the link (
>> https://github.com/apache/incubator-livy/releases/tag/
>> v0.4.0-incubating-rc1
>> and https://dist.apache.org/repos/dist/dev/incubator/livy/), would you
>> please help to test and verify. Thanks a lot and appreciate your help.
>>
>> Best regards,
>> Saisai
>>
>> On Sat, Aug 5, 2017 at 10:44 AM, Alex Bozarth  wrote:
>>
>> > Last comment on the INFRA JIRA seemed to indicate that they hit a snag
>> > with the import over a week ago and he never got back to us after. He
>> told
>> > us to keep using the Cloudera JIRA until he successfully completed a test
>> > import then we could re-export for him.
>> >
>> >
>> > *Alex Bozarth*
>> > Software Engineer
>> > Spark Technology Center
>> > --
>> > *E-mail:* *ajboz...@us.ibm.com* 
>> > *GitHub: **github.com/ajbozarth* 
>> >
>> >
>> > 505 Howard Street
>> > San Francisco, CA 94105
>> > United States
>> >
>> >
>> >
>> > [image: Inactive hide details for Bikas Saha ---08/04/2017 07:22:50
>> > PM---Hi, Most of those jiras looked like bug fixes to me. Hence I t]Bikas
>> > Saha ---08/04/2017 07:22:50 PM---Hi, Most of those jiras looked like bug
>> > fixes to me. Hence I thought 0.4 could be a bug fix release.
>> >
>> > From: Bikas Saha 
>> > To: "dev@livy.incubator.apache.org" 
>> > Date: 08/04/2017 07:22 PM
>> > Subject: Re: To release a first Apache version Livy
>> > --
>> >
>> >
>> >
>> > Hi,
>> >
>> >
>> > Most of those jiras looked like bug fixes to me. Hence I thought 0.4
>> could
>> > be a bug fix release. But I am ok releasing the current state so users
>> can
>> > gets an Apache release to transition to.
>> >
>> >
>> > Given that its still a new project, a shorter cadence would help make bug
>> > fix releases available.
>> >
>> >
>> > Btw, does anyone know whats holding up the Apache jira process? If not, I
>> > can follow up on that.
>> >
>> >
>> > Bikas
>> >
>> > 
>> > From: Saisai Shao 
>> > Sent: Thursday, August 3, 2017 7:31:10 PM
>> > To: dev@livy.incubator.apache.org
>> > Subject: Re: To release a first Apache version Livy
>> >
>> > From my side 0.4 might be a feasible choice to release as for Apache.
>> > Reverting all the features and release 0.3.1 is too time-consuming and
>> not
>> > so necessary.
>> >
>> > On Fri, Aug 4, 2017 at 4:16 AM, Alex Bozarth 
>> wrote:
>> >
>> > > @Bikas The list of JIRAs fixed in 0.4 is 50 long (
>> > > https://issues.cloudera.org/issues/?jql=project%20%3D%
>> > > 20LIVY%20AND%20fixVersion%20%3D%200.4) so I'm wondering what you mean
>> by
>> > > not including feature work. Are you suggesting we revert some of the
>> work
>> > > for this release and the re-merge it later, or just that you would've
>> > > preferred it that was released without new features, but are okay with
>> > how
>> > > it is anyway? If you're worried about adding features in the first
>> Apache
>> > > release should we also look at re-releasing 0.3.0 as 0.3.1-incubating
>> or
>> > > back support, personally I think it's not worth it.
>> > >
>> > > As for release cadence, so far we've seemed to shoot for a 6 month
>> > cadence
>> > > (was longer this time with the move to Apache) but I'd be fine moving
>> to
>> > a
>> > > 4 month cadence. I also prefer a time based release.
>> > >
>> > >
>> > > *Alex Bozarth*
>> > > Software Engineer
>> > > Spark Technology Center
>> > > --
>> > > *E-mail:* *ajboz...@us.ibm.com* 
>> > > *GitHub: **github.com/ajbozarth* 
>> > >
>> > >
>> > > 505 

Re: New Livy Logo Discussion and Input

[Marcelo Vanzin]

I don't really have opinions about the current logo, but you'd probably get
more feedback if you suggested a new logo instead of asking whether it
should be changed.

On Mon, Aug 7, 2017 at 12:34 PM, Alex Bozarth  wrote:

> No one has given feedback on the idea of a new Livy logo. Is everyone
> busy, ambivalent, have no ideas, or don't want to change? I can move
> forward on my own idea, but would love feedback first.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* 
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for "Alex Bozarth" ---07/25/2017 02:48:07
> PM---Hey Team, With our move to Apache I think we should take t]"Alex
> Bozarth" ---07/25/2017 02:48:07 PM---Hey Team, With our move to Apache I
> think we should take time to create a new logo
>
> From: "Alex Bozarth" 
> To: dev@livy.incubator.apache.org
> Date: 07/25/2017 02:48 PM
> Subject: New Livy Logo Discussion and Input
> --
>
>
>
>
>
> Hey Team,
>
> With our move to Apache I think we should take time to create a new logo
> for the project. Our current logo can be seen in the header of the website
> currently and personally I think we can do better. I have a designer
> available to help with this but need opinions and feedback from more people
> than just myself. Any ideas for a logo, from a theme or icon to use, color
> scheme, name to icon placement is appreciated.
>
> Some of the vague concepts I've thought of so far include the idea of a
> gateway or bridge to symbolize what Livy does. The idea I ran the most with
> though take the word word Livy itself as the base, Livy is a Latin
> derivative of Olive so I was looking at olives/olive leaves as an icon and
> the color olive as a color scheme, which can already been seen on the
> website. I've also considered leaving in the Spark orange is some capacity.
>
> Any amount of input is helpful, I don't want to end up limiting ideas to
> just what I've already worked on.
>
>
> Alex Bozarth
>
> Software Engineer
>
> Spark Technology Center
>
>
>
>
>
>
>
>
>
> E-mail: ajboz...@us.ibm.com
>
> GitHub: github.com/ajbozarth
>
>   505
> Howard Street
> San Francisco,
> CA 94105
>
> United States
>
>
>
>
>
>
>
>
>
>


-- 
Marcelo

Re: Livy website is now live

[Marcelo Vanzin]

Sweet!

Shouldn't the page say "incubating" though?

e.g. http://impala.incubator.apache.org/


On Thu, Jul 20, 2017 at 1:33 PM, Luciano Resende  wrote:
> We are live now:
>
> http://livy.incubator.apache.org/
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/



-- 
Marcelo

Re: Commit style best practices, was Re: incubator-livy-website git commit: Fix bug in merge_livy_pr.py because incubator-livy-website repo has no branch it's name started with "branch-"

[Marcelo Vanzin]

+1. Bad commit messages are one of my pet peeves. Although I like
periods at the end of sentences.

On Thu, Jul 20, 2017 at 1:09 PM, Luciano Resende  wrote:
> Could we try to follow some best practices on PR title, commit message, etc?
>
> Some info from: http://bahir.apache.org/contributing/#Creating+a+Pull+
> Request
>
> - Open a pull request against the master branch
>
>- The PR title should be of the form [LIVY-] Title, where LIVY-
>is the relevant JIRA number and Title may be the JIRA’s title or a more
>specific title describing the PR itself.
>- If the pull request is still a work in progress, and so is not ready
>to be merged, but needs to be pushed to Github to facilitate review, then
>add [WIP] after the component.
>- For website work, a JIRA is not required
>
> - Follow The 7 rules for a great commit message
> 
>
>- Separate subject from body with a blank line
>- Limit the subject line to 50 characters
>- Capitalize the subject line
>- Do not end the subject line with a period
>- Use the imperative mood in the subject line
>- Wrap the body at 72 characters
>- Use the body to explain what and why vs. how
>
> Below is an example of a good commit message
>
> [LIVY-001] Performance enhancements for decision tree
>
> Generate Matrix with random values through local memory
> if there is sufficient memory.
>
>
>
> Thoughts ?
>
> On Thu, Jul 20, 2017 at 1:03 PM,  wrote:
>
>> Repository: incubator-livy-website
>> Updated Branches:
>>   refs/heads/master 27348bab6 -> 572b37b1e
>>
>>
>> Fix bug in merge_livy_pr.py because incubator-livy-website repo has no
>> branch it's name started with "branch-"
>>
>>
>> Project: http://git-wip-us.apache.org/repos/asf/incubator-livy-websit
>> e/repo
>> Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy-websit
>> e/commit/572b37b1
>> Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy-websit
>> e/tree/572b37b1
>> Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy-websit
>> e/diff/572b37b1
>>
>> Branch: refs/heads/master
>> Commit: 572b37b1efc2a2947272790261b6ba023ec53b74
>> Parents: 27348ba
>> Author: jerryshao 
>> Authored: Thu Jul 20 13:02:23 2017 -0700
>> Committer: jerryshao 
>> Committed: Thu Jul 20 13:03:22 2017 -0700
>>
>> --
>>  merge_livy_pr.py | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>> --
>>
>>
>> http://git-wip-us.apache.org/repos/asf/incubator-livy-websit
>> e/blob/572b37b1/merge_livy_pr.py
>> --
>> diff --git a/merge_livy_pr.py b/merge_livy_pr.py
>> index b527a29..7296aef 100755
>> --- a/merge_livy_pr.py
>> +++ b/merge_livy_pr.py
>> @@ -359,7 +359,7 @@ def main():
>>  original_head = get_current_ref()
>>
>>  branches = get_json("%s/branches" % GITHUB_API_BASE)
>> -branch_names = filter(lambda x: x.startswith("branch-"), [x['name']
>> for x in branches])
>> +branch_names = [x['name'] for x in branches]
>>  # Assumes branch names can be sorted lexicographically
>>  latest_branch = sorted(branch_names, reverse=True)[0]
>>
>>
>>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/



-- 
Marcelo

Re: Another approach for Livy Website

[Marcelo Vanzin]

I can see internally to figure out who can change the redirect.

On Wed, Jul 19, 2017 at 8:15 PM, Alex Bozarth  wrote:

> Perfect, on a related note,
>
> @everyone once the website is up, should we update livy.io to redirect to
> livy.incubator.apache.com or shut it down?
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* 
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Luciano Resende ---07/19/2017 08:11:12
> PM---One pub-sub is configured, every time we commit to a "con]Luciano
> Resende ---07/19/2017 08:11:12 PM---One pub-sub is configured, every time
> we commit to a "content" branch, the website gets updated.
>
> From: Luciano Resende 
> To: dev@livy.incubator.apache.org
> Date: 07/19/2017 08:11 PM
> Subject: Re: Another approach for Livy Website
> --
>
>
>
> One pub-sub is configured, every time we commit to a "content" branch, the
> website gets updated.
>
> On Wed, Jul 19, 2017 at 8:08 PM, Alex Bozarth  wrote:
>
> > Thanks, will there be a straight forward process to it for future updates
> > or will we have to open a INFRA JIRA each time?
> >
> >
> > *Alex Bozarth*
> > Software Engineer
> > Spark Technology Center
> > --
> > *E-mail:* *ajboz...@us.ibm.com* 
> > *GitHub: **github.com/ajbozarth* 
> >
> >
> > 505 Howard Street
> > San Francisco, CA 94105
> > United States
> >
> >
> >
> > [image: Inactive hide details for Luciano Resende ---07/19/2017 08:02:27
> > PM---Yes, let me create a INFRA JIRA to publish it. On Wed, Ju]Luciano
> > Resende ---07/19/2017 08:02:27 PM---Yes, let me create a INFRA JIRA to
> > publish it. On Wed, Jul 19, 2017 at 7:59 PM, Alex Bozarth  >
> > From: Luciano Resende 
> > To: dev@livy.incubator.apache.org
> > Date: 07/19/2017 08:02 PM
> > Subject: Re: Another approach for Livy Website
> > --
> >
> >
> >
> > Yes, let me create a INFRA JIRA to publish it.
> >
> > On Wed, Jul 19, 2017 at 7:59 PM, Alex Bozarth 
> wrote:
> >
> > > Thanks Luciano, are there any steps we still need to take for the
> website
> > > to actually start showing up at livy.incubating.apache.org? It
> currently
> > > throws a 404 and livy.apache.org, which should also work, doesn't even
> > > exist.
> > >
> > >
> > > *Alex Bozarth*
> > > Software Engineer
> > > Spark Technology Center
> > > --
> > > *E-mail:* *ajboz...@us.ibm.com* 
> > > *GitHub: **github.com/ajbozarth* 
> > >
> > >
> > > 505 Howard Street
> > > San Francisco, CA 94105
> > > United States
> > >
> > >
> > >
> > > [image: Inactive hide details for Luciano Resende ---07/19/2017
> 06:56:52
> > > PM---After my little mistake, a pushing from different local r]Luciano
> > > Resende ---07/19/2017 06:56:52 PM---After my little mistake, a pushing
> > from
> > > different local repositry, things are fixed and it should be
> > >
> > > From: Luciano Resende 
> > > To: dev@livy.incubator.apache.org
> > > Date: 07/19/2017 06:56 PM
> > > Subject: Re: Another approach for Livy Website
> > > --
> >
> > >
> > >
> > >
> > > After my little mistake, a pushing from different local repositry,
> things
> > > are fixed and it should be ready now:
> > >
> > >
> > https://github.com/apache/incubator-livy-website
> >
> > >
> > > Thanks, Alex and others for the patience getting this ready.
> > >
> > >
> > > On Wed, Jul 19, 2017 at 5:20 PM, Alex Bozarth 
> > wrote:
> > >
> > > > Thanks Jerry,
> > > >
> > > > @Luciano if the code looks good to you can you do copy? I'll follow
> up
> > by
> > > > opening a few PRs to move the Docs out of the README into the website
> > > once
> > > > you've finished the copy.
> > > >
> > > >
> > > > *Alex Bozarth*
> > > > Software Engineer
> > > > Spark Technology Center
> > > > --
> > > > *E-mail:* *ajboz...@us.ibm.com* 
> > > > *GitHub: **github.com/ajbozarth* 
> > > >
> > > >
> > > > 505 Howard Street
> > > > San Francisco, CA 94105
> > > > United States
> > > >
> > > >
> > > >
> > > > [image: Inactive hide details for Saisai Shao ---07/19/2017 05:17:12
> > > > PM---Hi Alex, if think the changes are pretty ready, you can
> > creat]Saisai
> > > > Shao ---07/19/2017 05:17:12 PM---Hi Alex, if think the changes are
> > pretty
> > > > ready, you can create a PR to incubator-livy-website.
> > > >
> > > > From: Saisai Shao 
> > > > To: dev@livy.incubator.apache.org
> > > > Date: 07/19/2017 05:17 PM
> > > > Subject: Re: Another