pedrumj2 commented on issue #13014: URL: https://github.com/apache/gluten/issues/13014#issuecomment-5683217028
> The initial design is to make Gluten exactly compatible with Vanilla Spark's SQL or pyspark code. Gluten's first design principle is that user needn't change any line of Spark SQL or pyspark code. While your new proposal is to create a new use case for native Velox offloading only. Thanks for sharing the context around this @FelixYBW > 1. Legacy Spark code `CREATE TEMPORARY FUNCTION ` doesn't break. Internally, we may reuse Spark's registration or overwrite it as a nop then inferring. But functionally it should behave exactly like Spark does. > 2. Document that if customers use only native UDF functions, they won't have the fallback path. After all, if they didn't create their Java implementation, Gluten definitely won't fall back. We can raise exceptions directly. I created a PR for the issue here: https://github.com/apache/gluten/pull/13016 It ensures that existing behavior around `CREATE TEMPORARY FUNCTION` is preserved I've also updated the documentation and added a section about how defining UDFs in this new manner will not have the spark fallback. Please let me know if there is anything else we should consider here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
