alamb commented on code in PR #6817: URL: https://github.com/apache/datafusion-comet/pull/6817#discussion_r4231773783
########## docs/source/about/top_level_project_proposal.md: ########## @@ -0,0 +1,320 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Proposal to Establish Apache Comet as a Top-Level Project + +This page is a draft of the proposal and board resolution to promote Comet from a sub-project of +Apache DataFusion to a top-level Apache project, Apache Comet. It is adapted from the +[proposal][datafusion-proposal] that established Apache DataFusion as a top-level project in 2024, +when DataFusion graduated from Apache Arrow. Progress is tracked in [#6793][epic]. + +The DataFusion PMC has not voted on this proposal yet. + +## Board Resolution + +```text +X. Establish the Apache Comet Project + +WHEREAS, the Board of Directors deems it to be in the best interests of the +Foundation and consistent with the Foundation's purpose to establish a Project +Management Committee charged with the creation and maintenance of open-source +software related to accelerating Apache Spark using native, columnar +execution, for distribution at no charge to the public. + +NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee (PMC), to +be known as the "Apache Comet Project", be and hereby is established pursuant +to Bylaws of the Foundation; and be it further + +RESOLVED, that the Apache Comet Project be and hereby is responsible for the +creation and maintenance of software related to accelerating Apache Spark +using native, columnar execution; and be it further + +RESOLVED, that the office of "Vice President, Apache Comet" be and hereby is +created, the person holding such office to serve at the direction of the Board +of Directors as the chair of the Apache Comet Project, and to have primary +responsibility for management of the projects within the scope of +responsibility of the Apache Comet Project; and be it further + +RESOLVED, that the persons listed immediately below be and hereby are +appointed to serve as the initial members of the Apache Comet Project: + + * Andy Grove <[email protected]> + * Andrew Lamb <[email protected]> + * Liang-Chi Hsieh <[email protected]> + * Matt Butrovich <[email protected]> + * Oleks V. <[email protected]> + +NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andy Grove be appointed to the +office of Vice President, Apache Comet, to serve in accordance with and +subject to the direction of the Board of Directors and the Bylaws of the +Foundation until death, resignation, retirement, removal or disqualification, +or until a successor is appointed; and be it further + +RESOLVED, that the Apache Comet Project be and hereby is tasked with the +migration and rationalization of the Apache DataFusion Comet sub-project; and +be it further + +RESOLVED, that all responsibilities pertaining to the Apache DataFusion Comet +sub-project encumbered upon the Apache DataFusion Project are hereafter +discharged. +``` + +The scope statement is under discussion in [#6789][scope-issue]. + +## Summary + +We propose creating a new top-level project, Apache Comet, from an existing sub-project of Apache +DataFusion to facilitate additional community and project growth. + +## Abstract + +[Apache DataFusion Comet][comet-docs] is a high-performance accelerator for Apache Spark. It +replaces Spark operators and expressions with implementations that process Apache Arrow columnar +data, most of them native Rust code built on Apache DataFusion, and it runs existing Spark SQL, +DataFrame, and PySpark workloads without code changes. Comet accelerates Parquet and Apache Iceberg +scans, shuffle, joins, aggregations, and hundreds of Spark expressions, and it is about 1.8 times +faster than Spark 4.2 on TPC-DS at scale factor 1000. + +## Proposal + +We propose creating a new top-level ASF project, Apache Comet, governed initially by existing +Apache DataFusion PMC members and committers. The project's code is in one existing git +repository, currently governed by Apache DataFusion, which would transfer to the new top-level +project. + +## Background + +Apple donated Comet to the Apache Arrow project in early 2024 ([IP clearance][ip-clearance]), and +it became a sub-project of Apache DataFusion when DataFusion became a top-level project in +April 2024. Starting as a DataFusion sub-project made sense because of the overlap in contributors. + +Comet is different from the other DataFusion sub-projects, which extend DataFusion: Python and Java Review Comment: FWIW I think the most compelling difference is that they have largely distinct and strong communities. The technical differences I think are more nuanced and not as clear ot me ########## docs/source/about/top_level_project_proposal.md: ########## @@ -0,0 +1,320 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Proposal to Establish Apache Comet as a Top-Level Project + +This page is a draft of the proposal and board resolution to promote Comet from a sub-project of +Apache DataFusion to a top-level Apache project, Apache Comet. It is adapted from the +[proposal][datafusion-proposal] that established Apache DataFusion as a top-level project in 2024, +when DataFusion graduated from Apache Arrow. Progress is tracked in [#6793][epic]. + +The DataFusion PMC has not voted on this proposal yet. + +## Board Resolution + +```text +X. Establish the Apache Comet Project + +WHEREAS, the Board of Directors deems it to be in the best interests of the +Foundation and consistent with the Foundation's purpose to establish a Project +Management Committee charged with the creation and maintenance of open-source +software related to accelerating Apache Spark using native, columnar +execution, for distribution at no charge to the public. + +NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee (PMC), to +be known as the "Apache Comet Project", be and hereby is established pursuant +to Bylaws of the Foundation; and be it further + +RESOLVED, that the Apache Comet Project be and hereby is responsible for the +creation and maintenance of software related to accelerating Apache Spark +using native, columnar execution; and be it further + +RESOLVED, that the office of "Vice President, Apache Comet" be and hereby is +created, the person holding such office to serve at the direction of the Board +of Directors as the chair of the Apache Comet Project, and to have primary +responsibility for management of the projects within the scope of +responsibility of the Apache Comet Project; and be it further + +RESOLVED, that the persons listed immediately below be and hereby are +appointed to serve as the initial members of the Apache Comet Project: + + * Andy Grove <[email protected]> + * Andrew Lamb <[email protected]> + * Liang-Chi Hsieh <[email protected]> + * Matt Butrovich <[email protected]> + * Oleks V. <[email protected]> + +NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andy Grove be appointed to the +office of Vice President, Apache Comet, to serve in accordance with and +subject to the direction of the Board of Directors and the Bylaws of the +Foundation until death, resignation, retirement, removal or disqualification, +or until a successor is appointed; and be it further + +RESOLVED, that the Apache Comet Project be and hereby is tasked with the +migration and rationalization of the Apache DataFusion Comet sub-project; and +be it further + +RESOLVED, that all responsibilities pertaining to the Apache DataFusion Comet +sub-project encumbered upon the Apache DataFusion Project are hereafter +discharged. +``` + +The scope statement is under discussion in [#6789][scope-issue]. + +## Summary + +We propose creating a new top-level project, Apache Comet, from an existing sub-project of Apache +DataFusion to facilitate additional community and project growth. + +## Abstract + +[Apache DataFusion Comet][comet-docs] is a high-performance accelerator for Apache Spark. It +replaces Spark operators and expressions with implementations that process Apache Arrow columnar +data, most of them native Rust code built on Apache DataFusion, and it runs existing Spark SQL, +DataFrame, and PySpark workloads without code changes. Comet accelerates Parquet and Apache Iceberg +scans, shuffle, joins, aggregations, and hundreds of Spark expressions, and it is about 1.8 times +faster than Spark 4.2 on TPC-DS at scale factor 1000. + +## Proposal + +We propose creating a new top-level ASF project, Apache Comet, governed initially by existing +Apache DataFusion PMC members and committers. The project's code is in one existing git +repository, currently governed by Apache DataFusion, which would transfer to the new top-level +project. + +## Background + +Apple donated Comet to the Apache Arrow project in early 2024 ([IP clearance][ip-clearance]), and +it became a sub-project of Apache DataFusion when DataFusion became a top-level project in +April 2024. Starting as a DataFusion sub-project made sense because of the overlap in contributors. + +Comet is different from the other DataFusion sub-projects, which extend DataFusion: Python and Java +bindings, and distributed execution with Ballista. Comet is an accelerator for Apache Spark that +uses DataFusion as its execution engine. DataFusion and Comet now have largely distinct and strong +communities, though there is overlap, and the Comet community is large and active enough to stand +on its own. Focused governance will make it easier to grow that community and to recognize +contributors whose work is mainly on Comet. + +The community has discussed this idea publicly since July 2026 ([#5184][discussion]). As of the +time of this writing the reactions have been exclusively positive, including from members of the +DataFusion PMC. + +Several current members of the DataFusion PMC are active in Comet and understand and believe deeply Review Comment: I recommend mentioning that the proposed slate of PMC members includes at least two ASF members (although this is mentioned later on as well) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
