Hi all,
Thanks for the review and questions. I'm one of the Kyuubi committers, and have participated in the recent 6 months, include code changes[1], PR reviews, documents improvements, discussions, etc. I'd like to say, both Kent and other committers are nice and patient persons, and most of the discussions and code reviews happen in GitHub, I got many benefits from the contribution of Kyuubi. Kyuubi community is still young, but growing. From GitHub statistics[2], in the last 6 months, there are over 20 people participated in Kyuubi code development, include 6 people over hundred lines change, 4 people over thousand lines change. As a new committer, besides on code contribution, I'm also trying my best to extend the community. I recommend those engineers who are finding the way to migrate ETL infrastructure from Hive to Spark SQL to have a try on Kyuubi, and collect the bug reports[3][4] and some feature requests from them. I'm also looking for some potential contributors from the open-source community. For example, there is a project[5] build on top of Kyuubi to simplify to the deployment of Spark SQL on Kubernetes. And there is a blog[6](in Chinese) written in recent days that analyzed the architecture design and code implementation of Kyuubi. And from the mail lists in Apache Incubator discussions, I also see some people are interested in this project. I know there is still a long way to go to building the community, and hope to get helped by Apache Incubator. [1] https://github.com/NetEase/kyuubi/commits?author=pan3793 [2] https://github.com/NetEase/kyuubi/graphs/contributors?from=2020-12-01&to=2021-06-02&type=c [3] https://github.com/NetEase/kyuubi/issues/280 [4] https://github.com/NetEase/kyuubi/issues/647 [5] https://github.com/yilong2001/spark-sql-on-k8s [6] https://blog.51cto.com/xpleaf/2780248 Best, Cheng Pan ------------------ ???????? ------------------ ??????: "general" <a...@apache.org>; ????????: 2021??6??2??(??????) ????11:26 ??????: "general"<general@incubator.apache.org>; ????: Re: [DISCUSS] Incubating Proposal of Kyuubi I am genuinely worried about one contributor being the dominant of all. This puts the stability of the project in jeopardy and unless this is remedied, I feel the project should focus on developing a more diverse community first. On Wed, 2 Jun 2021, 20:23 Sheng Wu, <wu.sheng.841...@gmail.com> wrote: > Hi Kent > > Thanks for replying, but to be honest, your mail format is chaos. > Take a look at bellowing, which is what I saw in the gmail. > > This may be an issue of your mail client or some else. > > > The current #.2 - 5 top > > contributors will stay as committers, and they will join the PPMC during > > incubation. > > PPMC usually is the default for all initial committers, considering your > community is still small and not very active. > You should invite all people you trusted to join, rather than pushing them > away. > This is not just a too-high bar issue. Excluding 4 out of 5 is strange. > I can't guess what is the root cause, why the other 4 PPMC members don't > touch codes but are trusted completely. Meanwhile, people with easily > measurable code contributions are just as committers? > We(ASF) always remind the community to recognize no-code contribution, but > in this case, it seems the opposite? Don't recognize code contributors > enough? > Please reach your mentors, this seems not a typical ASF style. If you have > other people already involved in the ASF, asking them too. > > And from your description of the release process, I think you should talk > more with your mentors. > Such as what is a qualified ASF release, what should be checked. > This is not a block, the point I mentioned this is, you will need more > people. > > > Sheng Wu ???? > Twitter, wusheng1108 > > > Kent Yao <y...@apache.org> ??2021??6??2?????? ????10:37?????? > > > Hi Sheng, > > > > Thanks for your detailed check and valuable comments. They are > > accommodating. I have addressed some of them directly in the proposal. > > Please also check the table below for some Q&As. > > QA > > You listed only 5 core maintainers, but clearly only one has dominated > > contributions. The distortion and skewness of the contributions are > mainly > > caused by the way how we merge PRs changed(mentioned in another email on > > the list). > > I agree with Atri, orphaned is measurable. It is better to adjust this > > statement a little. Address the comment to notify the risk in the Kyuubi > > proposal. > > Also, with this fact, the core maintainers should notice whether this > small > > group of people to make sure your features iteration, bug fix, and > release > > process working by following ASF's requirements.This would not be an easy > > job. Currently, we learn the release process from some Apache projects, > > using major.minor.patch for versioning. As we can see, since Kyuubi 1.0, > it > > takes us 2 ~ 3 weeks for a patch release that contains 5-10 bug fixes > until > > EOL. And it takes us about two months to finish a minor release with 3 ~ > 5 > > big improvements and various small features/bug fixes. We also will start > > with some release candidates for preview and test before the final. This > is > > how we develop and release Kyuubi for now, but we do have a long way to > go > > and learn to match the ASF requirement. > > And your committer list is very strange... 1. Why some of them are PPMC > and > > others are committers? Where this hierarchy comes from? The current PPMC > > candidates help us a lot build the Kyuubi project and community, not > > directly to the codebase but as mentors(before Apache). Most of them are > > from Apache mature TLPs, which have close relationships with Kyuubi. They > > have sufficient Apache Way experience to guide us to become a more > > Apache-like project before submitting this proposal in many ways. We will > > continue to promote more contributors to the PPMC/committer based on > their > > code or non-code contributions during incubation. > > 2. Your core maintainers have 5, which only 4 PPMC? The current #.2 - 5 > top > > contributors will stay as committers, and they will join the PPMC during > > incubation. > > 3. I think only the top 4 have over 400 LOC, why you declared you have 5 > > core developers? I didn't notice that 400 LOC is a bar for core > developer, > > will change 5 to 4 core developers accordingly > > I think you don't have enough no-salaried developers, am I missing > > anything? Yes. > > Changed in the proposal, most of the contributors are salaried, and some > of > > them are willing to help in the spare time. Thanks > > This project belongs to NetEase Inc. clearly, why NO SGA? Added SGA in > > Proposal. All work here is approved by NetEase and can be forward. > > At last, you are asking about github.com/apache/incubator-kyuubi-shaded > > repo, > > what does this represent? I can only see you have only one source repo > and > > a doc repo. Removed this requirement in the proposal, and we will apply > it > > from IPMC if we do need this in the future. > > ------------------------------ > > > > Bests > > > > Kent Yao > > > > Sheng Wu <wu.sheng.841...@gmail.com> ??2021??6??2?????? ????5:45?????? > > > > > Hi > > > > > > I read the contributor list and the proposal > > > > The risk of the Kyuubi project being abandoned is minimal. Many > > > organizations are using Kyuubi to build critical big data pipelines and > > > willing to help develop Kyuubi's community if it becomes an ASF > project. > > > > > > I think the team should evaluate the status again. You listed only 5 > core > > > maintainers, but clearly only one has dominated contributions. I agree > > with > > > Atri, orphaned is measurable. > > > It is better to adjust this statement a little. Also, with this fact, > the > > > core maintainers should notice whether this small group of people to > make > > > sure your features iteration, bug fix, and release process working by > > > following ASF's requirements. > > > This would not be an easy job. > > > > > > And your committer list is very strange > > > ``` > > > Kent Yao (y...@apache.org) (PPMC) > > > Wenchen Fan (wenc...@apache.org) (PPMC) > > > Wanqiang Ji (j...@apache.org) (PPMC) > > > Xun Liu (liu...@apache.org) (PPMC) > > > Fei Wang (cn.feiw...@gmail.com) > > > Ulysses You (ulyssesyo...@gmail.com) > > > Cheng Pan (pan3...@gmail.com) > > > Hongxiang Jiang (jhx1...@gmail.com) > > > Zen Jin(xinjingziranc...@gmail.com) > > > ``` > > > 1. Why some of them are PPMC and others are committers? Where this > > > hierarchy comes from? > > > 2. Your core maintainers have 5, which only 4 PPMC? (Even hierarchy > makes > > > sense, most likely not) > > > 3. I think only the top 4 have over 400 LOC, why you declared you have > 5 > > > core developers? > > > > > > Another re-evaluation request is about `Reliance on Salaried > Developers:` > > > > They are all passionate about the project, and we are confident that > > the > > > project will continue even if no salaried developers contribute to the > > > project. > > > > > > I think you don't have enough no-salaried developers, am I missing > > > anything? > > > > > > About IP clearance, `Initial Source and Intellectual Property > Submission > > > Plan` > > > > As soon as Kyuubi is approved to join Apache Incubator, our initial > > > committers will submit iCLA(s) and CCLA(s). The codebase is already > > > licensed under the Apache License 2.0. > > > > > > This project belongs to NetEase Inc. clearly, why NO SGA? > > > > > > At last, you are asking about ` > > github.com/apache/incubator-kyuubi-shaded` > <http://github.com/apache/incubator-kyuubi-shaded> > > <http://github.com/apache/incubator-kyuubi-shaded> > > > <http://github.com/apache/incubator-kyuubi-shaded> > > > repo, what does this represent? I can only see you have only one source > > > repo and a doc repo. > > > > > > Sheng Wu ???? > > > Twitter, wusheng1108 > > > > > > > > > Atri Sharma <a...@apache.org> ??2021??6??2?????? ????4:49?????? > > > > > > > Thanks. > > > > > > > > I looked at the link you advised on and if I am reading it correctly, > > top > > > > committer has 630ish commits and the next committer has 73 commits? > > > > > > > > IMO this is an indication of the community not being active enough > and > > > > hence the risk of project getting orphaned is measurable. > > > > > > > > Atri > > > > > > > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <willem.ji...@gmail.com> > > wrote: > > > > > > > > > Hi Atri Sharma, > > > > > > > > > > It's easier to check out the recent contribution of kyuubi from > this > > > > > link[1]. > > > > > We already identified the one person contribution issue of kyuubi > > when > > > > > Kent came to us and we kept reminding him to put more effort into > > > > > building the community. > > > > > Last week I had a long F2F conversation with Kent, I think he > already > > > > > knows about it and there have been some improvements in recent > months > > > > > but there is still a long way to go if Kyuubi is accepted as a > > podling > > > > > in the Apache Incubator. > > > > > > > > > > [1]https://github.com/NetEase/kyuubi/graphs/contributors > > > > > > > > > > Willem Jiang > > > > > > > > > > Twitter: willemjiang > > > > > Weibo: ????willem > > > > > > > > > > On Tue, Jun 1, 2021 at 10:52 PM Atri Sharma <a...@apache.org> > wrote: > > > > > > > > > > > > Hi, > > > > > > > > > > > > I took a look at commit log and seems like all the commits for > > last 3 > > > > > > months have been done by a single person. > > > > > > > > > > > > How active are the remaining committers? > > > > > > > > > > > > On Mon, 31 May 2021, 12:59 Kent Yao, <y...@apache.org> wrote: > > > > > > > > > > > > > Hi Alexander Alten, > > > > > > > > > > > > > > Thanks for your interest and +1 for Kyuubi. > > > > > > > > > > > > > > Not being very familiar with Dremel, I just took a quick look > at > > > the > > > > > > > profile of a very similar and exciting project, Apache Drill. > If > > I > > > > > > > understand correctly, Kyuubi is built on top of the compute > layer > > > > > > > (Apache Spark), while Apache Drill is built on top of the > storage > > > > > > > layer. > > > > > > > > > > > > > > Apache Spark has a large user base and usage scenarios all over > > the > > > > > > > world. When users have some basic Spark programming and tuning > > > > skills, > > > > > > > things are easy for them. But many times, most users do not > have > > > all > > > > > > > of these abilities when they use Spark directly or indirectly. > We > > > > want > > > > > > > to build a more easy-to-use platform on top of it to help these > > > > users. > > > > > > > On the one hand, we will work on the interface layer to make it > > > easy > > > > > > > to access (of course, this effort will make our project look > very > > > > > > > similar to other projects, such as HiveServer2). On the other > > hand, > > > > we > > > > > > > will work under Kyuubi's multi-tenant architecture to extend > our > > > > > > > on-prem Spark SQL engine(a.k.a Kyuubi engine) to make it more > > > stable > > > > > > > and efficient in our scenario. > > > > > > > > > > > > > > What's more, we are going to add more on-prem Kyuubi engine > > types, > > > > e.g. > > > > > > > Flink. > > > > > > > > > > > > > > Alexander Alten <a...@scalytics.io> ??2021??5??31?????? ????2:35?????? > > > > > > > > > > > > > > > > > > > > > > > Hey, > > > > > > > > > > > > > > > > Looks like an interesting project, competition to Starburst. > > How > > > > > does it > > > > > > > > compare to Apache Dremel? > > > > > > > > > > > > > > > > +1 (non-binding) for incubation. > > > > > > > > > > > > > > > > Thanks and stay safe, > > > > > > > > --alex > > > > > > > > > > > > > > > > On Mon, May 31, 2021, 08:21 Kent Yao <y...@apache.org> wrote: > > > > > > > > > > > > > > > > > Dear all, > > > > > > > > > > > > > > > > > > We want to propose Kyuubi as a new Apache Incubator > project. > > > > > > > > > > > > > > > > > > Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC > server > > > for > > > > > > > > > large-scale data management, processing, and analytics, > built > > > on > > > > > top > > > > > > > > > of Apache Spark and designed to support more engines (i.e., > > > > Apache > > > > > > > > > Flink). We are aiming to make Kyuubi an "out-of-the-box" > tool > > > for > > > > > data > > > > > > > > > warehouses and data lakes. Kyuubi concentrates on improving > > the > > > > > > > > > experience for non-Spark users, making it easy for them to > > use > > > > > Spark > > > > > > > > > as a service and focus on business data of their own, not > > where > > > > it > > > > > > > > > stores, how it computes. NetEase has open-sourced Kyuubi > > since > > > > > 2018, > > > > > > > > > https://github.com/NetEase/kyuubi. > > > > > > > > > > > > > > > > > > Many thanks to Willem Jiang (Champion), Jeff Zhang, Duo > > Zhang, > > > > > Akira > > > > > > > > > Ajisaka for being the mentors of the Kyuubi project and > > helping > > > > us > > > > > set > > > > > > > > > up this proposal. The proposal can be found at > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal > > . > > > > > > > > > > > > > > > > > > Looking forward to your feedback and thank you very much. > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > Kent Yao > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > To unsubscribe, e-mail: > > > general-unsubscr...@incubator.apache.org > > > > > > > > > For additional commands, e-mail: > > > > general-h...@incubator.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Kent Yao > > > > > > > @ Data Science Center, Hangzhou Research Institute, NetEase > Corp. > > > > > > > a spark enthusiast > > > > > > > kyuubiis a unified multi-tenant JDBC interface for large-scale > > data > > > > > > > processing and analytics, built on top of Apache Spark. > > > > > > > > > > > > > > spark-authorizerA Spark SQL extension which provides SQL > Standard > > > > > > > Authorization for Apache Spark. > > > > > > > spark-postgres A library for reading data from and transferring > > > data > > > > > > > to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x > > > > faster. > > > > > > > itatchiA library that brings useful functions from various > modern > > > > > > > database management systems to Apache Spark. > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: > general-unsubscr...@incubator.apache.org > > > > > > > For additional commands, e-mail: > > general-h...@incubator.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > > > > > > > > > > > > > > > > > > > -- > > > > *Kent Yao * > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp. > > a spark enthusiast > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant > JDBC > > interface for large-scale data processing and analytics, built on top of > > Apache > > Spark <http://spark.apache.org/>. > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark > SQL > > extension which provides SQL Standard Authorization for Apache Spark > > <http://spark.apache.org/>. > > spark-postgres <https://github.com/yaooqinn/spark-postgres> A library > for > > reading data from and transferring data to Postgres / Greenplum with > Spark > > SQL and DataFrames, 10~100x faster. > > itatchi <https://github.com/yaooqinn/spark-func-extras>A library that > > brings useful functions from various modern database management > > systems to Apache > > Spark <http://spark.apache.org/>. > > >
