FelixYBW commented on issue #13014: URL: https://github.com/apache/gluten/issues/13014#issuecomment-5671695167
The initial design is to make Gluten exactly compatible with Vanilla Spark's SQL or pyspark code. Gluten's first design principle is that user needn't change any line of Spark SQL or pyspark code. While your new proposal is to create a new use case for native Velox offloading only. I think it's OK that we have both explicit declaration and inferring the Java signature from the Velox C++ UDF. We just need to make sure 1. Legacy Spark code `CREATE TEMPORARY FUNCTION ` doesn't break. Internally, we may reuse Spark's registration or overwrite it as a nop then inferring. But functionally it should behave exactly like Spark does. 2. Document that if customers use only native UDF functions, they won't have the fallback path. After all, if they didn't create their Java implementation, Gluten definitely won't fall back. We can raise exceptions directly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
