Apache Spark currently lacks an official mechanism to support cross-platform execution of physical plans. The Gluten project offers a mechanism that utilizes the Substrait standard to convert and optimize Spark's physical plans. By introducing Gluten's plan conversion, validation, and fallback mechanisms into Spark, we can significantly enhance the portability and interoperability of Spark's physical plans, enabling them to operate across a broader spectrum of execution environments without requiring users to migrate, while also improving Spark's execution efficiency through the utilization of Gluten's advanced optimization techniques. And the integration of Gluten into Spark has already shown significant performance improvements with ClickHouse and Velox backends and has been successfully deployed in production by several customers.
References: JIAR Ticket <https://issues.apache.org/jira/browse/SPARK-47773> SPIP Doc <https://docs.google.com/document/d/1v7sndtIHIBdzc4YvLPI8InXxhI7SnnAQ5HvmM2DGjVE/edit?usp=sharing> Your feedback and comments are welcome and appreciated. Thanks. Thanks, Jia Ke
