Revised Proposal with opt-out for committers and JB as a mentor which adds
more diversity to the company based mentors.
These were items suggested on this thread. Thanks again, Justin!
Abstract
Livy is a web service that exposes a REST interface for managing long
running Apache Spark contexts in your cluster. With Livy, new applications
can be built on top of Apache Spark that require fine grained interaction
with many Spark contexts [1].
While this project has been well regarded and used in many contexts as the
defacto standard API to Spark environments, it has been incubating for over
5 years without graduation to a TLP and it has become difficult to
impossible for fixes and improvements to be contributed as the current
community seems to have moved on.
There has been discussion regarding retirement of this podling where there
seems to be some increasing interest in joining and reviving the community
[2].
The intent of this proposal is to avoid retiring a well regarded, actively
used and rather mature project by reviving the PPMC and community with new
folks that have a vested interest in the project and health of the
community.
Proposal
We propose to revive the PPMC with a set of contributors and maintainers as
mentors, PPMC members and committers.
The retirement DISCUSS thread [2] has shown a growing interest in providing
new committers and bringing improvements and fixes from organization’s
internally maintained forks back to a revived community.
General Approach to Revival:
-
Add new Mentors
-
Larry McCay, lmc...@apache.org , Cloudera
-
Sunil Govindan, sun...@apache.org, Cloudera
-
Imran Rashid - iras...@apache.org, Cloudera
-
Jean-Baptiste Onofré, jbono...@apache.org, Talend
-
Add new Committers/PPMC
-
Larry McCay, lmc...@apache.org, Cloudera
-
Vinod Kumar Vavilapalli, vino...@cloudera.com, Cloudera
-
Gyorgy Gal, ggal ,gal.gyo...@gmail.com, Cloudera
-
Wing Yew Poon, wyp...@cloudera.com, Cloudera
-
Xilang Yan, xilang....@gmail.com, Shopee
-
Jianzhen Wu, myjianz...@gmail.com, Shopee
-
Nagella Jagadeewara Rao, jnage...@visa.com, Visa
-
Pralab Kumar, pralk...@visa.com, Visa
-
Prasad Shrikant, shrikant....@gmail.com, Visa
-
Brahma Reddy Battula, bra...@apache.org, Visa
-
Invite existing PPMC members to opt-in or otherwise go emeritus
-
Jean-Baptiste Onofré, jbono...@apache.org, Talend (opted-in via
Retirement DISCUSS thread [2])
-
Invite existing Committers to opt-out or otherwise continue
-
Establish Roadmap via follow up DISCUSS thread
-
Known Improvements from Forks which will need proposals and
discussion:
-
Adding HA for Livy
-
Updating security capabilities (eg. kerberos for jdbc, fixing bugs
in encryption)
-
Expanding the support for kubernetes
-
Responding to CVEs in dependencies (eg. log4j, thrift)
-
Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
-
Support multi Spark versions
-
Implemented a metrics system for Livy
-
Support customize batch/interactive session lifecycle event
handler, default log event with log4j, very helpful for troubleshooting
-
Optimize log to track which session id the log message came from,
also very helpful for troubleshooting
-
Support customize Spark config optimization rules, can be used to
optimize config for users’ job
-
A set of command line tool which can be used to replace Spark’s
spark-submit, pyspark, spark-sql but actually submit
application in Livy
-
We are planning to implement a JDBC state store, and allow multi
Livy Thrift sessions to share one backend Spark application
in the next few
months.
-
These items and others that are brought to community may need
consolidation or multiple configurable options and will need to
be part of
the discussion
-
One-pager Livy Improvement Proposals (LIP) may make sense to drive
these discussions and convergence
-
Feature Branch Strategy for large changes
-
Large features are hard to review we will need to define a
strategy
-
CTR for feature branches and a we will define a walkthrough of
implementation details to aid in review for merge
-
Determine the Improvements to be delivered across first 3 Releases
with Target Release Dates
-
Ensure CVE and Dependency management hygiene is in place
The above approach will usher the community back to an active status with a
Roadmap of 3 or more release plans and security hygiene in place.
Development Practices
The Livy project follows a review before commit philosophy. Every commit
automatically runs through the unit tests and generates coverage reports
presented as a pull request comment. Our experience with this process leads
us to believe that it helps ease new contributors into the project. They
get feedback quickly on common mistakes, lowering the burden on reviewers.
Those same reviewers get to lead by example, showing the new contributors
that we value feedback within our community even when changes are done by
more experienced folks. Taken from the original Apache Livy Proposal [1],
this should continue to be true. As mentioned, Livy is a mature project and
as such RTC is the most appropriate for continued quality and awareness.
1.
Original Apache Livy Proposal
https://cwiki.apache.org/confluence/display/incubator/LivyProposal
2.
Retirement DISCUSS thread
https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
On Sun, Oct 16, 2022 at 5:14 AM Jean-Baptiste Onofré <j...@nanthrax.net>
wrote:
> Hi Larry,
>
> thanks for this !
>
> I'm still interested to work on Livy. Can you please add me as mentor ?
>
> Thanks !
> Regards
> JB
>
> On Fri, Oct 14, 2022 at 8:38 PM larry mccay <larry.mc...@gmail.com> wrote:
> >
> > Abstract
> >
> > Livy is a web service that exposes a REST interface for managing long
> > running Apache Spark contexts in your cluster. With Livy, new
> applications
> > can be built on top of Apache Spark that require fine grained interaction
> > with many Spark contexts [1].
> >
> > While this project has been well regarded and used in many contexts as
> the
> > defacto standard API to Spark environments, it has been incubating for
> over
> > 5 years without graduation to a TLP and it has become difficult to
> > impossible for fixes and improvements to be contributed as the current
> > community seems to have moved on.
> >
> > There has been discussion regarding retirement of this podling where
> there
> > seems to be some increasing interest in joining and reviving the
> community
> > [2].
> >
> > The intent of this proposal is to avoid retiring a well regarded,
> actively
> > used and rather mature project by reviving the PPMC and community with
> new
> > folks that have a vested interest in the project and health of the
> > community.
> > Proposal
> >
> > We propose to revive the PPMC with a set of contributors and maintainers
> as
> > mentors, PPMC members and committers.
> >
> > The retirement DISCUSS thread [2] has shown a growing interest in
> providing
> > new committers and bringing improvements and fixes from organization’s
> > internally maintained forks back to a revived community.
> >
> > General Approach to Revival:
> >
> > -
> >
> > Add new Mentors
> > -
> >
> > Larry McCay, lmc...@apache.org , Cloudera
> > -
> >
> > Sunil Govindan, sun...@apache.org, Cloudera
> > -
> >
> > Imran Rashid - iras...@apache.org, Cloudera
> >
> >
> >
> > -
> >
> > Add new Committers/PPMC
> > -
> >
> > Larry McCay, lmc...@apache.org, Cloudera
> > -
> >
> > Vinod Kumar Vavilapalli, vino...@cloudera.com, Cloudera
> > -
> >
> > Gyorgy Gal, ggal ,gal.gyo...@gmail.com, Cloudera
> > -
> >
> > Wing Yew Poon, wyp...@cloudera.com, Cloudera
> > -
> >
> > Xilang Yan, xilang....@gmail.com, Shopee
> > -
> >
> > Jianzhen Wu, myjianz...@gmail.com, Shopee
> > -
> >
> > Nagella Jagadeewara Rao, jnage...@visa.com, Visa
> > -
> >
> > Pralab Kumar, pralk...@visa.com, Visa
> > -
> >
> > Prasad Shrikant, shrikant....@gmail.com, Visa
> > -
> >
> > Brahma Reddy Battula, bra...@apache.org, Visa
> >
> >
> >
> > -
> >
> > Invite existing PPMC members to opt-in or otherwise go emeritus
> > -
> >
> > Jean-Baptiste Onofré, jbono...@apache.org, Talend (opted-in via
> > Retirement DISCUSS thread [2])
> >
> >
> >
> > -
> >
> > Invite existing Committers/PPMC members to opt-in or otherwise go
> > emeritus
> >
> >
> >
> > -
> >
> > Establish Roadmap via follow up DISCUSS thread
> > -
> >
> > Known Improvements from Forks which will need proposals and
> > discussion:
> > -
> >
> > Adding HA for Livy
> > -
> >
> > Updating security capabilities (eg. kerberos for jdbc, fixing
> bugs
> > in encryption)
> > -
> >
> > Expanding the support for kubernetes
> > -
> >
> > Responding to CVEs in dependencies (eg. log4j, thrift)
> > -
> >
> > Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
> > -
> >
> > Support multi Spark versions
> > -
> >
> > Implemented a metrics system for Livy
> > -
> >
> > Support customize batch/interactive session lifecycle event
> > handler, default log event with log4j, very helpful for
> > trouble shooting
> > -
> >
> > Optimize log to track which session id the log message came
> from,
> > also very helpful for trouble shooting
> > -
> >
> > Support customize Spark config optimization rules, can be used
> to
> > optimize config for users’ job
> > -
> >
> > A set of command line tool which can be used to replace Spark’s
> > spark-submit, pyspark, spark-sql but actually submit
> > application in Livy
> > -
> >
> > We are planning to implement a JDBC state store, and allow multi
> > Livy Thrift sessions to share one backend Spark application
> > in the next few
> > months.
> > -
> >
> > These items and others that are brought to community may need
> > consolidation or multiple configurable options and will need to
> > be part of
> > the discussion
> > -
> >
> > One-pager Livy Improvement Proposals (LIP) may make sense to
> drive
> > these discussions and convergence
> > -
> >
> > Feature Branch Strategy for large changes
> > -
> >
> > Large features are hard to review we will need to define a
> > strategy
> > -
> >
> > Determine the Improvements to be delivered across first 3 Releases
> > with Target Release Dates
> >
> >
> >
> > -
> >
> > Ensure CVE and Dependency management hygiene is in place
> >
> >
> > The above approach will usher the community back to an active status
> with a
> > Roadmap of 3 or more release plans and security hygiene in place.
> > Development Practices
> >
> > The Livy project follows a review before commit philosophy. Every commit
> > automatically runs through the unit tests and generates coverage reports
> > presented as a pull request comment. Our experience with this process
> leads
> > us to believe that it helps ease new contributors into the project. They
> > get feedback quickly on common mistakes, lowering the burden on
> reviewers.
> > Those same reviewers get to lead by example, showing the new contributors
> > that we value feedback within our community even when changes are done by
> > more experienced folks. Taken from the original Apache Livy Proposal [1],
> > this should continue to be true. As mentioned, Livy is a mature project
> and
> > as such RTC is the most appropriate for continued quality and awareness.
> >
> >
> >
> > 1.
> >
> > Original Apache Livy Proposal
> > https://cwiki.apache.org/confluence/display/incubator/LivyProposal
> > 2.
> >
> > Retirement DISCUSS thread
> > https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>
