Thank you for sharing, Jia. I have the same questions like the previous Weiting's thread.
Do you think you can share the future milestone of Apache Gluten? I'm wondering when the first stable release will come and how we can coordinate across the ASF communities. > This project is still under active development now, and doesn't have a stable release. > https://github.com/apache/incubator-gluten/releases/tag/v1.1.1 In the Apache Spark community, Apache Spark 3.2 and 3.3 is the end of support. And, 3.4 will have 3.4.3 next week and 3.4.4 (another EOL release) is scheduled in October. For the SPIP, I guess it's applicable for Apache Spark 4.0.0 only if there is something we need to do from Spark side. Thanks, Dongjoon. On Tue, Apr 9, 2024 at 12:22 AM Ke Jia <kejia1...@gmail.com> wrote: > Apache Spark currently lacks an official mechanism to support > cross-platform execution of physical plans. The Gluten project offers a > mechanism that utilizes the Substrait standard to convert and optimize > Spark's physical plans. By introducing Gluten's plan conversion, > validation, and fallback mechanisms into Spark, we can significantly > enhance the portability and interoperability of Spark's physical plans, > enabling them to operate across a broader spectrum of execution > environments without requiring users to migrate, while also improving > Spark's execution efficiency through the utilization of Gluten's advanced > optimization techniques. And the integration of Gluten into Spark has > already shown significant performance improvements with ClickHouse and > Velox backends and has been successfully deployed in production by several > customers. > > References: > JIAR Ticket <https://issues.apache.org/jira/browse/SPARK-47773> > SPIP Doc > <https://docs.google.com/document/d/1v7sndtIHIBdzc4YvLPI8InXxhI7SnnAQ5HvmM2DGjVE/edit?usp=sharing> > > Your feedback and comments are welcome and appreciated. Thanks. > > Thanks, > Jia Ke >
