+1 (non-binding) Kazu
> On Mar 1, 2024, at 5:44 PM, L. C. Hsieh <vii...@gmail.com> wrote: > > +1 (binding) > > On Fri, Mar 1, 2024 at 1:25 PM Joris Van den Bossche > <jorisvandenboss...@gmail.com> wrote: >> >> +1 (binding) >> >> On Fri, 1 Mar 2024 at 22:18, Sutou Kouhei <k...@clear-code.com> wrote: >>> >>> +1 >>> >>> In <CAFhtnRy2J9GCU6e2K56-KPVc=gawemuipeyhmnwcd+htkfa...@mail.gmail.com> >>> "[VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project" >>> on Fri, 1 Mar 2024 06:33:08 -0500, >>> Andrew Lamb <al...@influxdata.com> wrote: >>> >>>> Hello, >>>> >>>> As we have discussed[1][2] I would like to vote on the proposal to >>>> create a new Apache Top Level Project for DataFusion. The text of the >>>> proposed resolution and background document is copy/pasted below >>>> >>>> If the community is in favor of this, we plan to submit the resolution >>>> to the ASF board for approval with the next Arrow report (for the >>>> April 2024 board meeting). >>>> >>>> The vote will be open for at least 7 days. >>>> >>>> [ ] +1 Accept this Proposal >>>> [ ] +0 >>>> [ ] -1 Do not accept this proposal because... >>>> >>>> Andrew >>>> >>>> [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341 >>>> [2] https://github.com/apache/arrow-datafusion/discussions/6475 >>>> >>>> ---------- Proposed Resolution --------- >>>> >>>> Resolution to Create the Apache DataFusion Project from the Apache >>>> Arrow DataFusion Sub Project >>>> >>>> ============================================================= >>>> >>>> X. Establish the Apache DataFusion Project >>>> >>>> WHEREAS, the Board of Directors deems it to be in the best >>>> interests of the Foundation and consistent with the >>>> Foundation's purpose to establish a Project Management >>>> Committee charged with the creation and maintenance of >>>> open-source software related to an extensible query engine >>>> for distribution at no charge to the public. >>>> >>>> NOW, THEREFORE, BE IT RESOLVED, that a Project Management >>>> Committee (PMC), to be known as the "Apache DataFusion Project", >>>> be and hereby is established pursuant to Bylaws of the >>>> Foundation; and be it further >>>> >>>> RESOLVED, that the Apache DataFusion Project be and hereby is >>>> responsible for the creation and maintenance of software >>>> related to an extensible query engine; and be it further >>>> >>>> RESOLVED, that the office of "Vice President, Apache DataFusion" be >>>> and hereby is created, the person holding such office to >>>> serve at the direction of the Board of Directors as the chair >>>> of the Apache DataFusion Project, and to have primary responsibility >>>> for management of the projects within the scope of >>>> responsibility of the Apache DataFusion Project; and be it further >>>> >>>> RESOLVED, that the persons listed immediately below be and >>>> hereby are appointed to serve as the initial members of the >>>> Apache DataFusion Project: >>>> >>>> * Andy Grove (agr...@apache.org) >>>> * Andrew Lamb (al...@apache.org) >>>> * Daniël Heres (dhe...@apache.org) >>>> * Jie Wen (jake...@apache.org) >>>> * Kun Liu (liu...@apache.org) >>>> * Liang-Chi Hsieh (vii...@apache.org) >>>> * Qingping Hou: (ho...@apache.org) >>>> * Wes McKinney(w...@apache.org) >>>> * Will Jones (wjones...@apache.org) >>>> >>>> RESOLVED, that the Apache DataFusion Project be and hereby >>>> is tasked with the migration and rationalization of the Apache >>>> Arrow DataFusion sub-project; and be it further >>>> >>>> RESOLVED, that all responsibilities pertaining to the Apache >>>> Arrow DataFusion sub-project encumbered upon the >>>> Apache Arrow Project are hereafter discharged. >>>> >>>> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb >>>> be appointed to the office of Vice President, Apache DataFusion, to >>>> serve in accordance with and subject to the direction of the >>>> Board of Directors and the Bylaws of the Foundation until >>>> death, resignation, retirement, removal or disqualification, >>>> or until a successor is appointed. >>>> ============================================================= >>>> >>>> >>>> ------- >>>> >>>> >>>> Summary: >>>> >>>> We propose creating a new top level project, Apache DataFusion, from >>>> an existing sub project of Apache Arrow to facilitate additional >>>> community and project growth. >>>> >>>> Abstract >>>> >>>> Apache Arrow DataFusion[1] is a very fast, extensible query engine >>>> for building high-quality data-centric systems in Rust, using the >>>> Apache Arrow in-memory format. DataFusion offers SQL and Dataframe >>>> APIs, excellent performance, built-in support for CSV, Parquet, JSON, >>>> and Avro, extensive customization, and a great community. >>>> >>>> [1] https://arrow.apache.org/datafusion/ >>>> >>>> >>>> Proposal >>>> >>>> We propose creating a new top level ASF project, Apache DataFusion, >>>> governed initially by a subset of the Apache Arrow project’s PMC and >>>> committers. The project’s code is in five existing git repositories, >>>> currently governed by Apache Arrow which would transfer to the new top >>>> level project. >>>> >>>> Background >>>> >>>> When DataFusion was initially donated to the Arrow project, it did not >>>> have a strong enough community to stand on its own. It has since grown >>>> significantly, and benefited immensely from being part of Arrow and >>>> nurturing of the Apache Way, and now has a community strong enough to >>>> stand on its own and that would benefit from focused governance >>>> attention. >>>> >>>> The community has discussed this idea publicly for more than 6 months >>>> https://github.com/apache/arrow-datafusion/discussions/6475 and >>>> briefly on the Arrow PMC mailing list >>>> https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As >>>> of the time of this writing both had exclusively positive reactions. >>>> >>>> Several current members of the Arrow PMC are both active contributors >>>> to DataFusion and understand and believe deeply in the Apache Way, and >>>> play active governance roles in the Arrow project as PMC members and >>>> PMC chairs, guiding the community, and releasing software versions. >>>> With this existing governance experience and structure, the new top >>>> level project will be able to function well immediately and >>>> independently. >>>> >>>> Overview of DataFusion >>>> >>>> Current Status >>>> >>>> Meritocracy >>>> >>>> DataFusion has been developed as part of Apache Arrow and thus has >>>> been operating as a meritocracy. Many of the developers of DataFusion >>>> are Arrow PMC members or committers. The DataFusion project plans to >>>> continue adding new PMC and committers as the project matures and >>>> grows. >>>> >>>> Community >>>> >>>> The DataFusion development team seeks to foster the development and >>>> user communities. We hope that becoming a separate project will help >>>> both Arrow and DataFusion communities by being more focused. Focused >>>> governance will make it easier to grow the community of committers and >>>> PMC members and make the organization more clear to others. >>>> >>>> Alignment >>>> >>>> The ASF is a natural host for DataFusion given that it is already the >>>> home of Arrow, Parquet, and other related distributed system, storage >>>> and query execution systems. >>>> >>>> Project Leadership >>>> >>>> Proposed Initial PMC >>>> >>>> We propose the following people as the initial DataFusion PMC members. >>>> This is a subset of the existing Arrow PMC members who contribute to >>>> DataFusion https://people.apache.org/phonebook.html?unix=arrow >>>> >>>> Andy Grove (agrove): Arrow PMC Chair >>>> Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair >>>> Daniël Heres (dheres) Arrow PMC >>>> Jie Wen (jakevin): Arrow PMC, Doris Committer >>>> Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC >>>> Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC >>>> Qingping Hou: (houqp): Arrow PMC >>>> Wes McKinney(wesm): Arrow PMC, ASF Member >>>> Will Jones (wjones127): Arrow PMC >>>> >>>> We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF >>>> VP) for the DataFusion project. >>>> >>>> Affiliations >>>> >>>> Andy Grove (agrove): NVidia >>>> Andrew Lamb (alamb): InfluxData >>>> Daniël Heres (dheres): Coralogix >>>> Jie Wen (jakevin): SelectDB >>>> Kun Liu (liukun): Ebay >>>> Liang-Chi Hsieh (viirya): Apple >>>> Qingping Hou: (houqp): Scribd >>>> Wes McKinney(wesm): Posit >>>> Will Jones (wjones127): LanceDB >>>> >>>> Proposed Initial Committers >>>> >>>> In addition to the PMC, we propose the following people as the initial >>>> DataFusion committers. This is a subset of the existing Arrow >>>> committers who contribute to DataFusion >>>> https://people.apache.org/phonebook.html?unix=arrow >>>> >>>> akurmustafa Mustafa Akur (Synnada) >>>> avantgardner Brent Gardner (Coralogix) >>>> comphead Oleks V. (Unaffiliated) >>>> jayzhan Jay Zhan (Unaffiliated) >>>> jeffreyvo Jeffry Vo (Unaffiliated) >>>> jiayuliu Liu Jiayu (Airbnb) >>>> mete Metehan Yildirim (Synnada) >>>> mingmwang Wang Mingming (Ebay) >>>> mneumann Marco Neumann (InfluxData) >>>> nju_yaho Zhong Yanghong (Ebay) >>>> ozankabak Mehmet Ozan Kabak (Synnada) >>>> paddyhoran Paddy Horan (Assured Allies) >>>> rdettai Rémi Dettai (Cloudfuse) >>>> sunchao Chao Sun (Apple) >>>> thinkharderdev Daniel Harris (Coralogix) >>>> tustvold Raphael Taylor-Davies (InfluxData) >>>> wayne Ruihang Xia (Greptime) >>>> xudong963 Xudong Wang (ByteDance) >>>> yjshen Yijie Shen (Space and Time) >>>> yangjiang Yang Jiang (ebay) >>>> >>>> >>>> Risk Assessments >>>> >>>> Naming / Trademarks >>>> >>>> As a sub-project of Arrow, the DataFusion name has been used for over >>>> 4 years without any known issues. A podling name search did not turn >>>> up any concerns and was approved: >>>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219 >>>> >>>> Legal / IP Clearance >>>> >>>> All DataFusion code has either been donated to the Arrow project with >>>> appropriate IP clearance or has been developed directly under ASF >>>> processes and procedures. Thus creating a new top level project poses >>>> no new Legal or IP risks. >>>> >>>> Code Extraction >>>> >>>> The relevant code is already in 5 separate repositories: >>>> https://github.com/apache/arrow-datafusion/ >>>> https://github.com/apache/arrow-datafusion-python >>>> https://github.com/apache/arrow-ballista >>>> https://github.com/apache/arrow-ballista-python >>>> https://github.com/apache/arrow-datafusion-comet >>>> >>>> We foresee no issues with code extraction and propose these >>>> repositories be renamed to reflect top level projects >>>> >>>> Note: https://github.com/apache/arrow-rs, the Rust implementation of >>>> Arrow, would remain part of the Arrow project. >>>> >>>> Orphaned Products >>>> >>>> DataFusion is known to be used in many open source and commercial >>>> projects >>>> https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users, >>>> has had multiple commits daily for several years, and its adoption and >>>> number of contributors appears to be growing. We do not foresee the >>>> project being orphaned in the next several years. >>>> >>>> Inexperience with Open Source >>>> >>>> The proposed PMC has extensive experience with Apache Arrow and other >>>> Apache projects, and includes PMC members, PMC chairs and an ASF >>>> Member. The DataFusion PMC and more experienced committers will >>>> continue to coach new community members who may be less familiar with >>>> the Apache Way. >>>> >>>> Homogeneous Developers >>>> >>>> The 9 proposed PMC members are from 9 different employers and the >>>> proposed committers are similarly distributed across affiliations. No >>>> specific entity employs more than 3 total proposed developers. >>>> >>>> Reliance on Salaried Developers >>>> >>>> A substantial amount of work on DataFusion has been by salaried >>>> developers, but it also has a long tradition of attracting >>>> contributions from students and hobbyists and we plan no changes in >>>> contribution structure. >>>> >>>> Relationships with Other Apache Products >>>> >>>> DataFusion will obviously have a strong relationship with the Arrow >>>> project given the overlap in people. We don’t foresee close >>>> collaboration with other projects at this time. >>>> >>>> Cryptography >>>> >>>> DataFusion does not directly support encryption and there are no >>>> near-term plans to add support for encryption. Users who need this >>>> functionality can use the extension APIs. >>>> >>>> Required Resources >>>> >>>> Mailing Lists >>>> >>>> - priv...@datafusion.apache.org for private PMC discussions (with >>>> moderated subscriptions) >>>> - d...@datafusion.apache.org >>>> - comm...@datafusion.apache.org >>>> - u...@datafusion.apache.org >>>> >>>> Version Control >>>> >>>> We propose to continue to use git for source control and github for >>>> hosting and testing resources. >>>> >>>> We also need to rename the github repositories to reflect the new top >>>> level names: >>>> >>>> https://github.com/apache/arrow-datafusion/ → apache/datafusion >>>> https://github.com/apache/arrow-datafusion-python → >>>> apache/datafusion-python >>>> https://github.com/apache/arrow-ballista → apache/datafusion-ballista >>>> https://github.com/apache/arrow-ballista-python → >>>> apache/datafusion-ballista-python >>>> https://github.com/apache/arrow-datafusion-comet → apache/datafusion-comet >>>> >>>> >>>> >>>> Issue Tracking >>>> >>>> DataFusion would continue to use github for its issue tracking and >>>> communications >>>> >>>> Other Resources >>>> >>>> The existing repositories already make use of existing Apache >>>> infrastructure, and we expect no change in the initial resource usage. >>>> As the project continues to grow, we expect continued infrastructure >>>> demand growth. >>>> >>>> >>>> FAQ: Has a sub project been promoted to a top level project before? >>>> >>>> Yes, and it appears to happen commonly. The Arrow project itself was >>>> created as a top level project from work that started in Apache Drill, >>>> and there are many sub projects of Hadoop that spun out as their own >>>> top level projects such as Mahout, Avro and HBase: >>>> https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4 >>>> >>>> >>>> >>>> Related material: >>>> Name search request / research for DataFusion: >>>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219 >>>> Discussion about this proposal on the arrow mailing list: >>>> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341 >>>> Discussion about which repositories on the arrow mailing list: >>>> https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q >>>> Discussion about initial PMC on the arrow mailing list: >>>> https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b >>>> Discussion in github about creating a new DataFusion top level >>>> project: https://github.com/apache/arrow-datafusion/discussions/6475 >>>> Discussion about graduating on incubator list: >>>> https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99 >>>> Original Proposal for the Arrow project: >>>> https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
