GitHub user andygrove closed a discussion: How does Comet compare to Gluten? Are there any plans to integrate with Gluten?
Comet and Gluten have similar architectures: - They both replace Spark's physical plan with a custom plan - They both use an Intermediate Representation to encode the plan. Gluten uses substrait, and Comet uses a simpler protobuf representation that is tightly coupled to Spark. - Gluten typically delegates to Velox when used to accelerate Spark, and Comet delegates to DataFusion The Comet project started before Gluten was an Apache project. There are no plans to integrate Comet with Gluten as part of the Comet project. However, there is no reason why Gluten could not add Comet/DataFusion as a backend query engine (DataFusion already supports Substrait, and Comet provides the Spark-compatible operators and expressions). GitHub link: https://github.com/apache/datafusion-comet/discussions/994 ---- This is an automatically sent email for github@datafusion.apache.org. To unsubscribe, please send an email to: github-unsubscr...@datafusion.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
