Re: Livy on Kubernetes support

[Marco Gaido]

Hi Aliaksandr,

thanks for your email and you work on this feature. As I mentioned to you
in the PR, I agree with you on the usefulness of this feature and you have
a big +1 from me having it in Livy. Unfortunately, it is not my area of
expertise, so I don't feel confident merging it without other reviewers
taking a careful look at it.

For the future, I think a better approach would be first to discuss and
define the architecture with the community, so that it is shared and
accepted by the whole community before the PR is out. This helps also
getting people involved and makes easier having them being able to review
the PR. Anyway, after you have split the PRs, I think they are reasonable
and we can discuss on them.

Looking forward to have your contribution in Livy.

Thanks,
Marco

Il giorno mar 14 gen 2020 alle ore 12:48 Aliaksandr Sasnouskikh <
jahstreetl...@gmail.com> ha scritto:

> Hi community,
>
> About a year ago I've started to work on the patch to Apache Livy for Spark
> on Kubernetes support in the scope of the project I've been working on.
> Since that time I've created a PR
> https://github.com/apache/incubator-livy/pull/167 which have already been
> discussed and reviewed a lot. After finalizing the work in the result of
> the PR discussions I've started to split the work introduced in the base PR
> into smaller pieces to make it easier to separate the core and aux
> functionality, and as a result - easier to review and merge. The first core
> PR is https://github.com/apache/incubator-livy/pull/249.
>
> Also I've created the repos with Docker images (
> https://github.com/jahstreet/spark-on-kubernetes-docker) and Helm charts (
> https://github.com/jahstreet/spark-on-kubernetes-helm) with the possible
> stack the users may want to use Livy on Kubernetes with, which potentially
> in the future can be partially moved to Livy repo to keep the artifacts
> required to run Livy on Kubernetes in a single place.
>
> Until now I've received the positive feedback from more than 10 projects
> about the usage of the patch. Several of them could be found in the
> discussions of the base PR. Also my repos supporting this feature have
> around 35 stars and 15 forks in total and were referenced in Spark related
> Stackoverflow and Kubernetes slack channel discussions. So the users use it
> already.
>
> You may think "What this guy wants from us then!?"... Well, I would like to
> ask for your time and expertise to help push it forward and ideally make it
> merged.
>
> Probably before I started coding I should have checked with the
> contributors if this feature may have value for the project and how is
> better to implement it, but I hope it is never too late;) So I'm here to
> share with you the the thoughts behind it.
>
> The idea of Livy on Kubernetes is simply to replicate the logic it has for
> Yarn API to Kubernetes API, which can be easily done since the interfaces
> for the Yarn API are really similar to the ones of the Kubernetes.
> Nevertheless this easy-to-do patch opens Livy the doors to Kubernetes which
> seems to be really useful for the community taking into account the
> popularity of Kubernetes itself and the latest releases of Spark supporting
> Kubernetes as well.
>
> Proposed Livy job submission flow:
> - Generate appTag and add
> `spark.kubernetes.[driver/executor].label.SPARK_APP_TAG_LABEL` to Spark
> config
> - Run spark-submit in cluster-mode with Kubernetes master
> - Start monitoring thread which resolves Spark Driver and Executor Pods
> using the `SPARK_APP_TAG_LABEL`s assigned during the job submission
> - Create additional Kubernetes resource if necessary: Spark UI service,
> Ingress, CRDs, etc.
> - Fetch Spark Pods statuses, Driver logs and other diagnostics information
> while Spark Pods are running
> - Remove Spark job resources (completed/failed Driver Pod, Service,
> ConfigMap, etc.) from the cluster after the job completion/failure after
> the configured timeout
>
> The core functionality (covered by
> https://github.com/apache/incubator-livy/pull/249):
> - Submission of Batch jobs and Interactive sessions
> - Caching Driver logs and Kubernetes Pods diagnostics
>
> Aux features (introduced in
> https://github.com/apache/incubator-livy/pull/167 and planned to be pushed
> after the core PR is merged):
> - Proxying Spark UI with Kubernetes Ingress'es and providing the access to
> it with the link on the Livy UI while the job is running (Nginx ingress as
> an example)
> - Proxying Spark History Server with Kubernetes Ingress'es and providing
> the access to it with the link on the Livy UI after the job
> completion/failure (Nginx ingress as an example)
>
> Misc:
> - Add documentation to the Livy web-site and the guide on how to run Livy
> on Kubernetes with the possible config options
> - Add Docker manifests and Helm charts
>
> Having Livy running on Kubernetes we do get not only the possibility to run
> Spark on Kubernetes with the REST API but also get out of the box
> 

Re: Livy metrics

[Marco Gaido]

I think the best approach would be first preparing a design doc with your
proposal so we can discuss on it and decide how to go about it.
You can go ahead with this if you want to contribute.

Thanks,
Marco
.

Il giorno ven 10 gen 2020 alle ore 03:00 安辉  ha scritto:

> In my company, We are maintaining a metrc version of livy, I am willing to
> submit this patch, if you do not mind.
>
> -邮件原件-
> 发件人: Saisai Shao [mailto:sai.sai.s...@gmail.com]
> 发送时间: 2020年1月10日 9:39
> 收件人: dev@livy.incubator.apache.org
> 主题: Re: Livy metrics
>
> Currently we don't have concrete plan on this, it would be great if you
> could contribute on this.
>
> Thanks
> Saisai
>
> Shanyu Zhao  于2020年1月10日周五 上午8:44写道：
>
> > Hi,
> >
> > Are there ongoing effort to add Livy metrics? I saw that dropwizard is
> > added as dependency in pom.xml but didn't see it get used anywhere.
> >
> > Thanks,
> > Shanyu
> >
> 
> OPPO
>
>
> 本电子邮件及其附件含有OPPO公司的保密信息，仅限于邮件指明的收件人使用（包含个人及群组）。禁止任何人在未经授权的情况下以任何形式使用。如果您错收了本邮件，请立即以电子邮件通知发件人并删除本邮件及其附件。
>
> This e-mail and its attachments contain confidential information from
> OPPO, which is intended only for the person or entity whose address is
> listed above. Any use of the information contained herein in any way
> (including, but not limited to, total or partial disclosure, reproduction,
> or dissemination) by persons other than the intended recipient(s) is
> prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!
>

Re: [RESULT][Vote][LIVY-718] Support multi-active high availability in Livy

[Marco Gaido]

Thank you for this proposal and your work. Looking forward to it.
Thanks,
Marco

On Thu, 19 Dec 2019, 07:27 Yiheng Wang,  wrote:

> Hi All
>
> Thanks for participating in the vote. Here's the result:
> +1 (binding)
> ajbozarth
> zjffdu
> mgaido91
> jerryshao
>
> +1 (no-binding)
> 5
>
> The vote passes. I will create subtasks for it.
>
> Thanks
> Yiheng
>
> On Thu, Dec 12, 2019 at 11:30 AM Yiheng Wang  wrote:
>
> > Dear Community
> >
> > I'd like to call a vote on LIVY-718
> >  "Support
> > multi-active high availability in Livy".
> >
> > Currently, Livy only supports single node recovery. This is not
> sufficient
> > in some production environments. In our scenario, the Livy server serves
> > many notebook and JDBC services. We want to make Livy service more
> > fault-tolerant and scalable.
> >
> > There're already some proposals in the community for high availability.
> > But they're not so complete or just for active-standby high availability.
> > So we propose a multi-active high availability design to achieve the
> > following goals:
> >
> >- One or more servers will serve the client requests at the same time.
> >- Sessions are allocated among different servers.
> >- When one node crashes, the affected sessions will be moved to other
> >active services.
> >
> > Please find the design doc here
> > <
> https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing
> >,
> > which has been reviewed for several weeks.
> >
> > This vote is open until next Wednesday (Dec. 18).
> >
> > [] +1: Accept the proposal
> > [] +0
> > [] -1: I don't think this is a good idea because ...
> >
> > Thank you
> >
> > Yiheng
> >
>

Re: Unable to retrieve livy statement status

[Marco Gaido]

I think this question is better to be submitted to the user list: you're
more likely to get answers.

Thanks,
Marco

Il giorno gio 12 dic 2019 alle ore 07:53 venkatesh sami <
venkateshsa...@gmail.com> ha scritto:

> Team,
>
> I have a problem in getting statement status in another livy statement,
> detailed description is posted here
>
> https://stackoverflow.com/questions/59285113/not-able-to-retrieve-livy-statement-statusget-in-another-livy-statement
> <
> https://slack-redir.net/link?url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F59285113%2Fnot-able-to-retrieve-livy-statement-statusget-in-another-livy-statement
> >
>
> can someone please help me.
>
> *Thanks,*
> *VenkateshSami*
>

ASF Diversity & Inclusion Survey

[Marco Gaido]

Hello everyone,

If you have an apache.org email, you should have received an email with an
invitation to take the 2020 ASF Community Survey. Please take 15 minutes to
complete it.

If you do not have an apache.org email address or you didn’t receive a
link, please follow this link to the survey:
https://communitysurvey.limequery.org/454363

This survey is important because it will provide us with scientific
information about our community, and shed some light on how we can
collaborate better and become more diverse. Our last survey of this kind
was implemented in 2016, which means that our existing data about Apache
communities is outdated. The deadline to complete the survey is January
4th, 2020. You can find information about privacy on the survey’s
Confluence page [1].

Your participation is paramount to the success of this project! Please
consider filling out the survey, and share this news with your fellow
Apache contributors. As individuals form the Apache community, your opinion
matters: we want to hear your voice.

If you have any questions about the survey or otherwise, please reach out
to us!

Kindly,
ASF Diversity & Inclusion
https://diversity.apache.org/


[1]
https://cwiki.apache.org/confluence/display/EDI/Launch+Plan+-+The+2020+ASF+Community+Survey

Re: Plan to cut new branch and prepare to release 0.7

[Marco Gaido]

I think it is harder to do that for the thriftserver case. By its nature,
the REST API allows multiple single requests, while in the thrifserver case
the connection is always the same. I think without automatic failover, that
feature is not so critical.

Il giorno gio 10 ott 2019 alle ore 11:12 yihengw  ha
scritto:

> Regarding the thrift server GA, should we consider supporting recovery in
> thrift server? I open a JIRA for it
> https://issues.apache.org/jira/browse/LIVY-696
>
>
> welcome for discussion.
>
>
>  原始邮件
> 发件人: Saisai Shao
> 收件人: dev
> 发送时间: 2019年10月10日(周四) 16:24
> 主题: Re: Plan to cut new branch and prepare to release 0.7
>
>
> I'm hesitant to include service discovery into 0.7.0. Because all of the
> JIRAs only addressed a small piece of high availability, but lack a whole
> picture of HA design, I would suggest to have a whole HA design, then
> splitting into small pieces, rather than opposite. So currently I'm not
> confident if the JIRAs are good enough to merge. Thanks Saisai Marco Gaido <
> marcogaid...@gmail.com> 于2019年10月10日周四 下午4:19写道： > Hi Saisai, > > thanks
> for starting this discussion. I agree starting talking about a 0.7.0 >
> release. Just a couple of notes: > > [LIVY-692][THRIFT] Use configured
> principal for Kerberos authentication: as > I mentioned in the PR, I don't
> think this is an actual problem. > > What about including in the scope of
> this release also service discovery? > I've seen a couple of PRs and it may
> be worth to have it in this release. > Any thoughts on this? > > Thanks, >
> Marco > > Il giorno gio 10 ott 2019 alle ore 06:08 Jeff Zhang <
> zjf...@gmail.com> ha > scritto: > > > Make sense to me > > > > Saisai
> Shao  于2019年10月10日周四 下午12:04写道： > > > > > Hi all,
> > > > > > > I'm planning to cut a new branch and prepare for the 0.7
> release, which > > did > > > lots of improvements to thrift module, I think
> we could make thrift > > module > > > GA as for this release. > > > > > >
> Currently I'm waiting for this JIRAs to be merged before cutting the > new
> > > > branch. > > > > > > [LIVY-356][SERVER]Add LDAP authentication for
> livy-server. > > > [LIVY-678] Thrift ldap authentication, based on ldapurl,
> basedn, domain > > > [LIVY-692][THRIFT] Use configured principal for
> Kerberos authentication > > > [LIVY-689] Deliver stage process message to
> the end user using > > thriftserver > > > > > > Any thought? > > > > > >
> Thanks > > > Saisai > > > > > > > > > -- > > Best Regards > > > > Jeff
> Zhang > > >

Re: Plan to cut new branch and prepare to release 0.7

[Marco Gaido]

Hi Saisai,

thanks for starting this discussion. I agree starting talking about a 0.7.0
release. Just a couple of notes:

[LIVY-692][THRIFT] Use configured principal for Kerberos authentication: as
I mentioned in the PR, I don't think this is an actual problem.

What about including in the scope of this release also service discovery?
I've seen a couple of PRs and it may be worth to have it in this release.
Any thoughts on this?

Thanks,
Marco

Il giorno gio 10 ott 2019 alle ore 06:08 Jeff Zhang  ha
scritto:

> Make sense to me
>
> Saisai Shao  于2019年10月10日周四 下午12:04写道：
>
> > Hi all,
> >
> > I'm planning to cut a new branch and prepare for the 0.7 release, which
> did
> > lots of improvements to thrift module, I think we could make thrift
> module
> > GA as for this release.
> >
> > Currently I'm waiting for this JIRAs to be merged before cutting the new
> > branch.
> >
> > [LIVY-356][SERVER]Add LDAP authentication for livy-server.
> > [LIVY-678] Thrift ldap authentication, based on ldapurl, basedn, domain
> > [LIVY-692][THRIFT] Use configured principal for Kerberos authentication
> > [LIVY-689] Deliver stage process message to the end user using
> thriftserver
> >
> > Any thought?
> >
> > Thanks
> > Saisai
> >
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Podling Report Reminder - October 2019

[Marco Gaido]

Thanks Bikas,

I checked and everything sounds fine to me. I just edited the wiki in order
to fix some typos and other small edits.

Thanks,
Marco

Il giorno mar 1 ott 2019 alle ore 05:18 Bikas Saha  ha
scritto:

> Hi,
>
> The issue seems to have resolved itself. Coming to why I started the edit.
>
> Please see my note under the Livy section so that we can discuss if it
> makes sense or not.
>
> https://cwiki.apache.org/confluence/display/INCUBATOR/October2019#three-most-important-unfinished-issues-to-address-before-graduating-7
>
> Essentially, IMO, I see the Apache Livy community and code practices
> following the tenets of the Apache way. The community also has multiple
> public releases following due process. There has been addition of new
> contributors and committers based on their participation. Based on that I
> feel we could start a conversation on whether the project is ready to
> graduate.
>
> There is no doubt that the code and community activity is not as
> voluminous as some of the larger projects like Apache Hadoop or Spark. But
> those projects also have a significantly larger scope in terms of
> features/functionality compared to Livy. So it may be that, as of now,
> Apache Livy is a stable project that meets its intended scope. Thus its ok
> for it to have moderate code and community activity until such time as new
> scope/features are identified which would spurt the next wave of high
> activity.
>
> With the above context, does it make sense to discuss whether the project
> is ready to graduate with its current (relative small) community.
>
> Thoughts?
> Bikas
>
> 
> From: Jean-Baptiste Onofré 
> Sent: Sunday, September 22, 2019 10:31 PM
> To: dev@livy.incubator.apache.org 
> Subject: Re: Podling Report Reminder - October 2019
>
> Hi,
>
> it works fine for me.
>
> Regards
> JB
>
> On 22/09/2019 18:47, Bikas Saha wrote:
> > Hi,
> >
> > I noticed the wiki page said this wiki has been enabled with LDAP and
> login with Apache credentials. I did and it logged me in but reported an
> error on every link saying I don't have permissions. Anyone else hitting
> this issue?
> >
> > Bikas
> >
> > 
> > From: jmcl...@apache.org 
> > Sent: Sunday, September 22, 2019 3:46 AM
> > To: dev@livy.incubator.apache.org 
> > Subject: Podling Report Reminder - October 2019
> >
> > Dear podling,
> >
> > This email was sent by an automated system on behalf of the Apache
> > Incubator PMC. It is an initial reminder to give you plenty of time to
> > prepare your quarterly board report.
> >
> > The board meeting is scheduled for Wed, 16 October 2019, 10:30 am PDT.
> > The report for your podling will form a part of the Incubator PMC
> > report. The Incubator PMC requires your report to be submitted 2 weeks
> > before the board meeting, to allow sufficient time for review and
> > submission (Wed, October 02).
> >
> > Please submit your report with sufficient time to allow the Incubator
> > PMC, and subsequently board members to review and digest. Again, the
> > very latest you should submit your report is 2 weeks prior to the board
> > meeting.
> >
> > Candidate names should not be made public before people are actually
> > elected, so please do not include the names of potential committers or
> > PPMC members in your report.
> >
> > Thanks,
> >
> > The Apache Incubator PMC
> >
> > Submitting your Report
> >
> > --
> >
> > Your report should contain the following:
> >
> > *   Your project name
> > *   A brief description of your project, which assumes no knowledge of
> > the project or necessarily of its field
> > *   A list of the three most important issues to address in the move
> > towards graduation.
> > *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> > aware of
> > *   How has the community developed since the last report
> > *   How has the project developed since the last report.
> > *   How does the podling rate their own maturity.
> >
> > This should be appended to the Incubator Wiki page at:
> >
> > https://cwiki.apache.org/confluence/display/INCUBATOR/October2019
> >
> > Note: This is manually populated. You may need to wait a little before
> > this page is created from a template.
> >
> > Note: The format of the report has changed to use markdown.
> >
> > Mentors
> > ---
> >
> > Mentors should review reports for their project(s) and sign them off on
> > the Incubator wiki page. Signing off reports shows that you are
> > following the project - projects that are not signed may raise alarms
> > for the Incubator PMC.
> >
> > Incubator PMC
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Podling Report Reminder - October 2019

[Marco Gaido]

Hi,

I don't have access either. May it be I wasn't granted access?

Thanks,
Marco

Il giorno lun 23 set 2019 alle ore 07:31 Jean-Baptiste Onofré <
j...@nanthrax.net> ha scritto:

> Hi,
>
> it works fine for me.
>
> Regards
> JB
>
> On 22/09/2019 18:47, Bikas Saha wrote:
> > Hi,
> >
> > I noticed the wiki page said this wiki has been enabled with LDAP and
> login with Apache credentials. I did and it logged me in but reported an
> error on every link saying I don't have permissions. Anyone else hitting
> this issue?
> >
> > Bikas
> >
> > 
> > From: jmcl...@apache.org 
> > Sent: Sunday, September 22, 2019 3:46 AM
> > To: dev@livy.incubator.apache.org 
> > Subject: Podling Report Reminder - October 2019
> >
> > Dear podling,
> >
> > This email was sent by an automated system on behalf of the Apache
> > Incubator PMC. It is an initial reminder to give you plenty of time to
> > prepare your quarterly board report.
> >
> > The board meeting is scheduled for Wed, 16 October 2019, 10:30 am PDT.
> > The report for your podling will form a part of the Incubator PMC
> > report. The Incubator PMC requires your report to be submitted 2 weeks
> > before the board meeting, to allow sufficient time for review and
> > submission (Wed, October 02).
> >
> > Please submit your report with sufficient time to allow the Incubator
> > PMC, and subsequently board members to review and digest. Again, the
> > very latest you should submit your report is 2 weeks prior to the board
> > meeting.
> >
> > Candidate names should not be made public before people are actually
> > elected, so please do not include the names of potential committers or
> > PPMC members in your report.
> >
> > Thanks,
> >
> > The Apache Incubator PMC
> >
> > Submitting your Report
> >
> > --
> >
> > Your report should contain the following:
> >
> > *   Your project name
> > *   A brief description of your project, which assumes no knowledge of
> > the project or necessarily of its field
> > *   A list of the three most important issues to address in the move
> > towards graduation.
> > *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> > aware of
> > *   How has the community developed since the last report
> > *   How has the project developed since the last report.
> > *   How does the podling rate their own maturity.
> >
> > This should be appended to the Incubator Wiki page at:
> >
> > https://cwiki.apache.org/confluence/display/INCUBATOR/October2019
> >
> > Note: This is manually populated. You may need to wait a little before
> > this page is created from a template.
> >
> > Note: The format of the report has changed to use markdown.
> >
> > Mentors
> > ---
> >
> > Mentors should review reports for their project(s) and sign them off on
> > the Incubator wiki page. Signing off reports shows that you are
> > following the project - projects that are not signed may raise alarms
> > for the Incubator PMC.
> >
> > Incubator PMC
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Possible missing mentor(s)

[Marco Gaido]

Sid, answering to the PRs you pointed out, I think open meisam's PRs are:
 - 1 WIP;
 - 1 changes were requested and he has not done them;
 - 1 I commented on it and it was stated that the feature could have been
merged if the PR is brought up to date, which hasn't so far by the
contributor.

Moreover, I have never seen on the PRs committers getting pinged. IMHO,
those PRs are stuck due to the inactivity of the contributor.
Surely we have to do more in order to be able to review more PRs, but, as
Saisai said, we are stil a small community with few active committers, so
we try and do our best.
More importantly, I think there is really no evidence at all of the alleged
bias.

In general, I think we all know that you need to be very active in the
community to avoid PRs getting stale. This is true also for bigger
communities like Spark.
So what I'd encourage is also more proactivity by the contributors. I think
the message should not be: "I open a PR and committer must immediately
merge it"; but "I open a PR, I ask for committers' help on reviews and try
and push in order to have it in if the PR is important".

Thanks,
Marco

Il giorno dom 1 set 2019 alle ore 20:52 Pat Ferrel 
ha scritto:

> Seems like some action should be taken before 2 years, even if it is to
> close the PR because it is not appropriate. Isn’t this the responsibility
> of the chair to guard against committer changes where the contributor is
> still willing? Or if a mentor is guiding the PR they should help it get
> unstalled if the contributor is still willing to make changes.
>
> The point (2 year old PRs) seems well taken. The question should be; what
> can be done about this?
>
> For what it’s worth, we are just starting to use Livy and wish it was part
> of Spark. We would like to treat Spark as a “microservice” as a compute
> engine. The Spark team seems to want to make Spark integral to the
> architecture of ALL applications that use it. Very odd from our point of
> view.
>
> So to integrate Livy we deeply hope it doesn’t fall into disrepair and are
> willing to help when we run into something.
>
>
> From: Sid Anand  
> Reply: dev@livy.incubator.apache.org 
> 
> Date: September 1, 2019 at 11:19:00 AM
> To: dev@livy.incubator.apache.org 
> 
> Subject:  Re: Possible missing mentor(s)
>
> "Second, if someone has a *good* and *large* contribution history, and
> actively participates in community, we will add him without doubt. Third,
> 2-year old open PRs doesn't stand anything, some reviewers left the
> community and PRs get staled, it is quite common, especially in large
> community."
>
> Meisam has 7 closed and 3 open PRs - of the 4 oldest open PRs in Livy (I
> see 4 in 2017), 2 are his. He's ranked #10 in the contributor list --
> It's not a large contribution history mostly because it takes so long to
> merge and he has been consistently active for 2 years. The size of the
> community doesn't seem a factor here with <200 closed PRs and <50
> contributors.
>
> How are you prioritizing PR merges if you think having a 2 year old open PR
> is okay and you don't a ton of open PRs?
> -s
>
> On Sun, Sep 1, 2019 at 2:25 AM Saisai Shao  wrote:
>
> > First, we're scaling the PR review, but we only have few active
> committers,
> > so the merge may not be fast.
> > Second, if someone has a *good* and *large* contribution history, and
> > actively participates in community, we will add him without doubt.
> > Third, 2-year old open PRs doesn't stand anything, some reviewers left
> the
> > community and PRs get staled, it is quite common, especially in large
> > community.
> >
> > Sid Anand  于2019年9月1日周日 下午4:46写道：
> >
> > > Apache projects promote contributors to committers based on
> contributions
> > > made, not on an expectation of future activity. That's the Apache way
> per
> > > my understanding. Over time, folks become inactive and busy -- life
> > > happens, I get it. May I ask what are you folks doing to scale PR
> review
> > > and merging? Are you adding new committers? Do you feel that 2-year old
> > > open PRs is where you wish to be and is the right way to grow a
> > community?
> > >
> > > On Sun, Sep 1, 2019 at 1:46 AM Sid Anand  wrote:
> > >
> > > > Apache projects promote contributors to committers based on
> > contributions
> > > > made, not on an expectation of future activity. That's the Apache way
> > per
> > > > my understanding. Over time, folks become inactive and busy -- life
> > > > happens, I get it. May I ask what are you folks doing to scale PR
> > review
> > > > and merging? Are you adding new committers? Do you feel that 2-year
> > old
> > > > open PRs is where you wish to be and is the right way to grow a
> > > community?
> > > >
> > > > -s
> > > >
> > > > On Sun, Sep 1, 2019 at 12:59 AM Saisai Shao 
> > > > wrote:
> > > >
> > > >> It's unfair to say there's underlying bias. Livy project is a small
> > > >> project, the contributor diversity may not be as rich as popular
> > project
> > > >> like Spark, it is not 

Re: Possible missing mentor(s)

[Marco Gaido]

Hi Sid,

what you are stating is pretty serious. I am not sure whether you have
specific examples you're referring to.
Since I've been asked to join the PMC, I regularly check open PRs and I try
and review/check all those on the part I have more confidence with, ie. the
thriftserver. Honestly, I don't recall to miss to review any PR I was
pinged on.

Recently, I remember only the proposal of the Livy clients, for which,
AFAIK, there were checks on the legal implications and proper ways to have
that contributed to Livy. And I think I have been active part of that
discussion too. So, I cannot see the bias you're referring to.

Thanks,
Marco

Il giorno sab 31 ago 2019 alle ore 22:20 Sid Anand  ha
scritto:

> Folks!
> We've (several devs, myself included) contacted the livy dev list and the
> owners DL several times. Our PRs stagnated over a few years. Livy is a
> central component in PayPal's Data Infra (Our data footprint is 80+ PB).
> The project seems pretty unhealthy. After a few years, this dev moved on
> and the state of our PR may be harder to define, with both absentee
> mentors/PMC/committers and PR author.
>
> I see only a narrow band of contributors being merged. I hope there is no
> underlying bias, given that this would not be the Apache way. As mentors, I
> hope the goal is to watch for such bias and eliminate it to promote
> community health, engagement, and integrity.
>
> -s
>
> On Fri, Aug 30, 2019 at 10:22 PM Jean-Baptiste Onofré 
> wrote:
>
> > Hi Justin,
> >
> > Like Luciano, I'm also around. I've proposed couple of new features that
> > I plan to work on and I try to review/verify the releases.
> >
> > Regards
> > JB
> >
> > On 21/10/2018 01:50, Justin Mclean wrote:
> > > Hi,
> > >
> > > We have discussed missing mentors on the incubator general list,
> > identified those who may be missing and over a couple of months tried to
> > contact your mentor in several ways but have got no reply so they have
> been
> > removed from the roster.
> > >
> > > If this is error, and your mentor is actually active, please contact me
> > and I'll add them back to the roster.
> > >
> > > The IPMC will also do what it can to find extra mentors for those
> > podling that need them.
> > >
> > > Thanks,
> > > Justin
> > > (V.P. Incubator)
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>

Re: Livy Sessions

[Marco Gaido]

Hi Carlos,

I meant profiling the Livy server with a JVM profiler.

Thanks,
Bests.
Marco

Il giorno mar 9 lug 2019 alle ore 00:18 Kadu Vido <
carlos.v...@lendico.com.br> ha scritto:

> Hi, Marco,
>
> We're using livy 0.6.0, which I'm afraid ships by default with EMR 5.24,
> so I cannot test a former version.
>
> It's a holiday in Brazil and our network is IP-gated (which is also why I
> don't have logs), so we won't be able to access it tomorrow. I'll run that
> profiling as soon as we're back on Wednesday. I'm assuming you want a *perf
> record*, let me know otherwise.
>
> *Carlos Vido *
>
> Data Engineer @ Lendico Brasil <https://www.lendico.com.br>
>
>
> On Mon, 8 Jul 2019 at 16:55, Marco Gaido  wrote:
>
>> Hi all,
>>
>> Seems like a perf issue in livy server. I assume you are using a recent
>> version of livy.
>>
>> If this is he case, may you profile livy server in order to understand
>> which is the problem?
>>
>> Thanks,
>> Marco
>>
>> On Mon, 8 Jul 2019, 21:03 Kadu Vido,  wrote:
>>
>>> Hi, I'm working with Hugo in the same project.
>>>
>>> Shubham, we're using almost the same setup, only difference is Airflow
>>> 1.10.1. I coded a workaround in our Livy hook, it has a parameter for
>>> retries and whenever the session returns anything different from 'idle', we
>>> try again before failing the task. It's not ideal but at least our
>>> pipelines aren't stuck anymore.
>>>
>>> Zhang, I don't have yarn logs in hand but I can search for them if you'd
>>> like to take a look. However, our latest clues point a different way:
>>>
>>> 1 - running *top* on the master node, we observed that LIvy rapidly
>>> takes all the available CPUs after we send just a few requests (3 or 4
>>> already cause this to happen, if we send upwards of 10, it'll crash the
>>> service).
>>>
>>> 2. We can get around this spacing them out a bit -- that is, if we use a
>>> loop to open the sessions and wait ~10s betwen them, it'll give Livy enough
>>> time to release the CPU resources before trying to open a new one. We've
>>> had help from some AWS engineers that tried on several instance sizes and
>>> found out that on larger instances they can try to open 10 or 12
>>> simultaneously, but:
>>>
>>> 3. Regardless of the size of the cluster, we cannot hold more than 9
>>> simultaneous sessions open. It doesn't matter if our cluster has enough
>>> vCPUs or RAM to handle more, and the size of the master node doesn't matter
>>> either: from the 10th session onwards, each one seems to either die or drop.
>>>
>>> *Carlos Vido *
>>>
>>> Data Engineer @ Lendico Brasil <https://www.lendico.com.br>
>>>
>>>
>>> On Sat, 6 Jul 2019 at 13:30, Shubham Gupta 
>>> wrote:
>>>
>>>> I'm facing precisely same issue.
>>>> .
>>>> I've written a LivySessionHook that's just a wrapper over PyLivy
>>>> Session <https://pylivy.readthedocs.io/en/latest/api/session.html>.
>>>>
>>>>- I'm able to use this hook to send code-snippets to remote EMR via
>>>>Python shell a few times, after which it starts throwing "caught
>>>>exception 500 Server Error: Internal Server Error for url" (and
>>>>continues to do so for next hour or so).
>>>>- However when the same hook is triggered via Airflow operator, I
>>>>get absolutely no success (always results in 500 error).
>>>>
>>>> .
>>>> I'm using
>>>>
>>>>- Airflow 1.10.3
>>>>- Python 3.7.3
>>>>- EMR 5.24.1
>>>>- Livy 0.6.0
>>>>- Spark 2.4.2
>>>>
>>>>
>>>> *Shubham Gupta*
>>>> Software Engineer
>>>>  zomato
>>>>
>>>>
>>>> On Sat, Jul 6, 2019 at 6:56 PM Jeff Zhang  wrote:
>>>>
>>>>> For the dead/killed session, could you check the yarn app logs ?
>>>>>
>>>>> Hugo Herlanin  于2019年7月4日周四 下午9:41写道：
>>>>>
>>>>>>
>>>>>> Hey, user mail is not working out!
>>>>>>
>>>>>> I am having some problems with livy setup. My use case is as follows:
>>>>>> I use a DAG in airflow (1.10) to create a cluster in EMR (5.24.1, one
>>>>>> master is m4.large and two nodes in m5a.xlarge), and when it is ready,
>>>

Re: Livy Sessions

[Marco Gaido]

Hi all,

Seems like a perf issue in livy server. I assume you are using a recent
version of livy.

If this is he case, may you profile livy server in order to understand
which is the problem?

Thanks,
Marco

On Mon, 8 Jul 2019, 21:03 Kadu Vido,  wrote:

> Hi, I'm working with Hugo in the same project.
>
> Shubham, we're using almost the same setup, only difference is Airflow
> 1.10.1. I coded a workaround in our Livy hook, it has a parameter for
> retries and whenever the session returns anything different from 'idle', we
> try again before failing the task. It's not ideal but at least our
> pipelines aren't stuck anymore.
>
> Zhang, I don't have yarn logs in hand but I can search for them if you'd
> like to take a look. However, our latest clues point a different way:
>
> 1 - running *top* on the master node, we observed that LIvy rapidly takes
> all the available CPUs after we send just a few requests (3 or 4 already
> cause this to happen, if we send upwards of 10, it'll crash the service).
>
> 2. We can get around this spacing them out a bit -- that is, if we use a
> loop to open the sessions and wait ~10s betwen them, it'll give Livy enough
> time to release the CPU resources before trying to open a new one. We've
> had help from some AWS engineers that tried on several instance sizes and
> found out that on larger instances they can try to open 10 or 12
> simultaneously, but:
>
> 3. Regardless of the size of the cluster, we cannot hold more than 9
> simultaneous sessions open. It doesn't matter if our cluster has enough
> vCPUs or RAM to handle more, and the size of the master node doesn't matter
> either: from the 10th session onwards, each one seems to either die or drop.
>
> *Carlos Vido *
>
> Data Engineer @ Lendico Brasil 
>
>
> On Sat, 6 Jul 2019 at 13:30, Shubham Gupta 
> wrote:
>
>> I'm facing precisely same issue.
>> .
>> I've written a LivySessionHook that's just a wrapper over PyLivy Session
>> .
>>
>>- I'm able to use this hook to send code-snippets to remote EMR via
>>Python shell a few times, after which it starts throwing "caught
>>exception 500 Server Error: Internal Server Error for url" (and
>>continues to do so for next hour or so).
>>- However when the same hook is triggered via Airflow operator, I get
>>absolutely no success (always results in 500 error).
>>
>> .
>> I'm using
>>
>>- Airflow 1.10.3
>>- Python 3.7.3
>>- EMR 5.24.1
>>- Livy 0.6.0
>>- Spark 2.4.2
>>
>>
>> *Shubham Gupta*
>> Software Engineer
>>  zomato
>>
>>
>> On Sat, Jul 6, 2019 at 6:56 PM Jeff Zhang  wrote:
>>
>>> For the dead/killed session, could you check the yarn app logs ?
>>>
>>> Hugo Herlanin  于2019年7月4日周四 下午9:41写道：
>>>

 Hey, user mail is not working out!

 I am having some problems with livy setup. My use case is as follows: I
 use a DAG in airflow (1.10) to create a cluster in EMR (5.24.1, one master
 is m4.large and two nodes in m5a.xlarge), and when it is ready,  this dag
 sends 5 to 7 simultaneous requests to Livy. I think I'm not messing with
 the Livy settings, I  just set livy.spark.deploy-mode = client and
 livy.repl.enable-hive-context = true.

 The problem is that from these ~ 5 to 7 sessions, just one or two opens
 (goes to 'idle') and all others go straight to 'dead' or 'killed', in logs
 Yarn returns that the sessions were killed by 'livy' user. I tried to
 tinker with all possible timeout settings, but this is still happening. If
 I send more than ~10 simultaneous requests, livy responds with 500, and if
 I continue sending requests, the server freezes. This happens even if EMR
 has enough resources available.

 I know the cluster is able to handle that many questions because it
 works when I open them via a loop with an interval of 15 seconds or more,
 but it feels like livy should be able to deal with that many requests
 simultaneously. It seems strange that I should need to manage the queue in
 such a way for an API of a distributed system.

 Do you have any clue about where I might be doing wrong? Is there any
 known limitation that I'm unaware of?

 Best,

 Hugo Herlanin


>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>

Re: Accessing Detailed Livy Session Information (session name?)

[Marco Gaido]

I think in general our website documentation is not complete and I think it
would be great if we could improve it significantly. For instance, there is
no place where all configurations are described properly (other than
reading the code). I think that PRs and help in improving the doc would be
highly appreciated.

Thanks,
Marco

On Tue, 16 Apr 2019, 05:13 Meisam Fathi,  wrote:

> Hi Peter,
>
> Are you using ZooKeeper for recovery store?
> If yes, in conf/livy.conf, is
> livy.server.recovery.zk-state-store.key-prefix
> set to different values in different Livy instances? If not, all of Livy
> instances will read/write the recovery data from/to the same path, which is
> default is /livy/v1 by default.
>
> @dev mailing list:
> This behavior is not documented in livy.conf nor on the website. It might
> be a good idea to document it somewhere.
>
> Thanks,
> Meisam
>
> On Fri, Apr 12, 2019 at 3:20 PM Meisam Fathi 
> wrote:
>
> > Hi Peter,
> >
> > Livy 0.6 has a new feature to give each session a name:
> > https://github.com/apache/incubator-livy/pull/48
> >
> > Would this feature be useful in your usecase?
> >
> > Thanks,
> > Meisam
> >
> > On Fri, Apr 12, 2019, 8:51 AM Peter Wicks (pwicks) 
> > wrote:
> >
> >> Greetings,
> >>
> >>
> >>
> >> I have a custom service that connects to Livy, v0.4 soon to be v0.5 once
> >> we go to HDP3. If sessions already exist it logs the session ID’s and
> >> starts using them, if sessions don’t exist it creates new ones. The
> problem
> >> is the account used to launch the Livy sessions is not unique to this
> >> service, nor is the kind of session. So sometimes it grabs other
> people’s
> >> sessions and absconds off with them. Also, there are multiple instances
> of
> >> the service, running under the same account, and they are not supposed
> to
> >> use each other’s sessions… that’s not working out so well.
> >>
> >>
> >>
> >> The service names the sessions, but I can’t find any way to retrieve
> >> detailed session data so that I can update the service to check if the
> Livy
> >> Session belongs to the service or not.
> >>
> >>
> >>
> >> I found some older comments 2016/2017 about retrieving Livy sessions by
> >> name. I don’t really need that, I just want to be able to read the name
> >> through the regular sessions REST call.
> >>
> >>
> >>
> >> Any REST calls I missed, or undocumented calls… that can help?
> >>
> >>
> >>
> >> Thanks,
> >>
> >>   Peter
> >>
> >>
> >>
> >> Ref:
> >>
> https://github.com/meisam/livy/wiki/Design-doc-for-Livy-41:-Accessing-sessions-by-name
> ,
> >> https://issues.cloudera.org/browse/LIVY-41
> >>
> >>
> >>
> >>
> >>
> >
>

Re: [VOTE] Release Livy 0.6.0 based on RC2

[Marco Gaido]

I am seeing several issues on the thriftserver, anyway , since it is kind
of a preview of the thriftserver, probably this is not a big deal. The
major one is https://issues.apache.org/jira/browse/LIVY-572, which can be
workarounded with a config, though.
The SHA is ok, and the files in the distributions are too.
Since we already said that thriftserver issues are not a reason good enough
for failing this RC vote, I am +1 on this.
Thanks Marcelo!


Il giorno mer 20 mar 2019 alle ore 18:25 Marco Gaido 
ha scritto:

> Agree on not failing for Thriftserver issues, just wondering if we are
> doing something wrong when generating the JARs.
>
> Il giorno mer 20 mar 2019 alle ore 18:10 Marcelo Vanzin
>  ha scritto:
>
>> Hi,
>>
>> Sorry, but as I mentioned in the other thread, I haven't used Livy in
>> a while and I don't really have time to investigate things past the
>> existing tests. Anyone is free to do it, though.
>>
>> I'd rather not fail a release because of a problem in the thrift
>> server; it's new and "experimental", so I'd expect issues.
>>
>> On Wed, Mar 20, 2019 at 2:07 AM Marco Gaido 
>> wrote:
>> >
>> > Hi Marcelo,
>> >
>> > thanks for starting the vote. I tried downloading the zip and enabling
>> the
>> > thriftserver, but connecting with beeline and running a query throws an
>> > exception to me:
>> >
>> > 0: jdbc:hive2://localhost:10090/> create table foo as select a.* from
>> > (select 1, "2") a;
>> >
>> > Error: java.lang.RuntimeException: java.util.NoSuchElementException:
>> > Statement ce95f6bb-bf51-4fca-b61d-0d40272b3a33 not found in session
>> > 774fed7a-8388-43fd-978f-8d2bd31c4a12.
>> >
>> >
>> org.apache.livy.thriftserver.session.ThriftSessionState.statementNotFound(ThriftSessionState.java:118)
>> >
>> >
>> org.apache.livy.thriftserver.session.ThriftSessionState.cleanupStatement(ThriftSessionState.java:107)
>> >
>> >
>> > This may also be caused by some problems in my local environment though,
>> > since the tests on Travis are passing. May someone else try if they face
>> > the same issue anyway? In the config I just enabled the thriftserver
>> with
>> > the proper config.
>> > Thanks,
>> > Marco
>> >
>> > Il giorno mar 19 mar 2019 alle ore 22:58 Marcelo Vanzin
>> >  ha scritto:
>> >
>> > > Err. Should have read "based on RC2". Too fast on the send button!
>> > >
>> > > Anyway, here's my +1.
>> > >
>> > > Note this build does includes the pre-compiled jars for the thrift
>> server.
>> > >
>> > > On Tue, Mar 19, 2019 at 2:56 PM Marcelo Vanzin 
>> > > wrote:
>> > > >
>> > > > This vote is for releasing Livy 0.6.0 based on RC2.
>> > > >
>> > > > The vote will be open at least 72 hours until Friday March 21st,
>> 22:00
>> > > UTC and
>> > > > will pass with a minimum of 3 +1 binding votes and a majority of
>> > > positive votes.
>> > > >
>> > > > [+1] This release is ready to send to the Incubator PMC for approval
>> > > >
>> > > > [-1] This release is not ready because...
>> > > >
>> > > > This vote is held according to Apache Incubator release policy:
>> > > > https://incubator.apache.org/policy/incubation.html#releases
>> > > >
>> > > > The RC is based on tag v0.6.0-incubating-rc2:
>> > > > https://github.com/apache/incubator-livy/commit/28be98cabc
>> > > >
>> > > > The release files can be found here:
>> > > >
>> > >
>> https://dist.apache.org/repos/dist/dev/incubator/livy/0.6.0-incubating-rc2/
>> > > >
>> > > > The staged maven artifacts can be found here:
>> > > >
>> https://repository.apache.org/content/repositories/orgapachelivy-1008
>> > > >
>> > > > The list of resolved JIRAs in this release can be found here:
>> > > > https://issues.apache.org/jira/projects/LIVY/versions/12342736
>> > > >
>> > > >
>> > > > --
>> > > > Marcelo
>> > >
>> > >
>> > >
>> > > --
>> > > Marcelo
>> > >
>>
>>
>>
>> --
>> Marcelo
>>
>

Re: [VOTE] Move source repo to gitbox

[Marco Gaido]

Thanks for starting this Marcelo, +1.

Il giorno gio 3 gen 2019 alle ore 18:57 Marcelo Vanzin
 ha scritto:

> Creating a formal vote for this. I don't think we have a choice but
> they seem to request a vote anyway. This will be lazy consensus, I
> plan to create an infra ticket for the migration on Monday if the vote
> passes.
>
> Starting with my +1.
>
>
> --
> Marcelo
>

Re: [DISCUSS] Getting rid of old stuff

[Marco Gaido]

+1 too, thanks for bringing this up Marcelo.

Thanks,
Marco

Il giorno ven 14 set 2018 alle ore 06:14 Jean-Baptiste Onofré <
j...@nanthrax.net> ha scritto:

> +1
>
> Regards
> JB
>
> On 14/09/2018 00:10, Marcelo Vanzin wrote:
> > Hey all,
> >
> > I'd like to gauge people's reaction to some proposals regarding what
> > is supported in Livy.
> >
> > #1: Java versions
> >
> > I propose dropping support for Java 7. Even J8 is already EOL,
> > although it's pretty obvious nobody is getting rid of it anytime soon.
> > But I don't see a good reason to support J7. Even testing it is a
> > nuisance, since most people only have jdk8 around...
> >
> > #2: Spark versions
> >
> > I think we should drop 1.6. At least. This would clean up some code
> > that currently uses reflection, and fix some parts of the API (like
> > the JobContext method to retrieve a "SparkSession" instance).
> >
> > Changing the API is well, not backwards compatible, but I think it's
> > better to do that sort of cleanup earlier than later when the project
> > is more mature.
> >
> > Separately, we could consider dropping 2.0 and 2.1 also. There was
> > talk in the Spark list about making 2.1 EOL - I don't remember a final
> > verdict, but I don't imagine there will be a lot of new deployments of
> > 2.1 going forward, since the only reason to use it is if you're stuck
> > with J7.
> >
> > #3: Scala versions
> >
> > If we decide to only support Spark 2.2+, then the decision is easy.
> > But if we drop 1.6, we should consider dropping Scala 2.10 support.
> > Spark does not ship official builds with 2.10 support in the 2.x line,
> > and it was dropped altogether in 2.2.
> >
> > We shouldn't remove support for multiple versions of Scala, though,
> > since 2.12 will be beta in Spark 2.4, and there will be a 2.13 at some
> > point.
> >
> > Thoughts?
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>