marin-ma commented on issue #13014: URL: https://github.com/apache/gluten/issues/13014#issuecomment-5662692588
Hi @pedrumj2 Thanks for bringing this up. Gluten used to register native udf without requiring a Java matching function and the `CREATE FUNCTION`/`CREATE TEMPORARY FUNCTION ` in the early implementation, but we added the restriction based on the user habit and the spark-compatibility concern. It's an uncommon use case that the user only needs a native UDF without having the java implementation. > From a user experience point of view, user can then write a Velox C++ udf once and have it available to different backends out of the box (i.e they get spark out of the box for free). The user who writes the SQL that uses the UDF doesn't have to be the one who implements the UDF. In Spark SQL applications, users need to register the UDF before using it, and Gluten runs compatible with Spark SQL engine. Removing the restrictions will break the Spark SQL registration requirement. Another concern is that not having the java registration restriction will also break Gluten's fallback mechanism, as Gluten doesn't know which function to fall back once the native UDF fails the validation process. cc: @FelixYBW -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
