I like the idea of improving flexibility of Sparks physical plans and really anything that might reduce code duplication among the ~4 or so different accelerators.
Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau On Tue, Apr 9, 2024 at 3:14 AM Dongjoon Hyun <dongjoon.h...@gmail.com> wrote: > Thank you for sharing, Jia. > > I have the same questions like the previous Weiting's thread. > > Do you think you can share the future milestone of Apache Gluten? > I'm wondering when the first stable release will come and how we can > coordinate across the ASF communities. > > > This project is still under active development now, and doesn't have a > stable release. > > https://github.com/apache/incubator-gluten/releases/tag/v1.1.1 > > In the Apache Spark community, Apache Spark 3.2 and 3.3 is the end of > support. > And, 3.4 will have 3.4.3 next week and 3.4.4 (another EOL release) is > scheduled in October. > > For the SPIP, I guess it's applicable for Apache Spark 4.0.0 only if there > is something we need to do from Spark side. > +1 I think any changes need to target 4.0 > > Thanks, > Dongjoon. > > > On Tue, Apr 9, 2024 at 12:22 AM Ke Jia <kejia1...@gmail.com> wrote: > >> Apache Spark currently lacks an official mechanism to support >> cross-platform execution of physical plans. The Gluten project offers a >> mechanism that utilizes the Substrait standard to convert and optimize >> Spark's physical plans. By introducing Gluten's plan conversion, >> validation, and fallback mechanisms into Spark, we can significantly >> enhance the portability and interoperability of Spark's physical plans, >> enabling them to operate across a broader spectrum of execution >> environments without requiring users to migrate, while also improving >> Spark's execution efficiency through the utilization of Gluten's advanced >> optimization techniques. And the integration of Gluten into Spark has >> already shown significant performance improvements with ClickHouse and >> Velox backends and has been successfully deployed in production by several >> customers. >> >> References: >> JIAR Ticket <https://issues.apache.org/jira/browse/SPARK-47773> >> SPIP Doc >> <https://docs.google.com/document/d/1v7sndtIHIBdzc4YvLPI8InXxhI7SnnAQ5HvmM2DGjVE/edit?usp=sharing> >> >> Your feedback and comments are welcome and appreciated. Thanks. >> >> Thanks, >> Jia Ke >> >
