pedrumj2 commented on issue #13014: URL: https://github.com/apache/gluten/issues/13014#issuecomment-5670854097
Gluten used to register native udf without requiring a Java matching function and the `CREATE FUNCTION`/`CREATE TEMPORARY FUNCTION ` in the early implementation, but we added the restriction based on the user habit and the spark-compatibility concern. It's an uncommon use case that the user only needs a native UDF without having the java implementation. Hi @marin-ma thanks for sharing the context around this. If I'm not mistake this was the PR that removed it? [PR7016](https://github.com/apache/gluten/pull/7016/changes) > Another concern is that not having the java registration restriction will also break Gluten's fallback mechanism, as Gluten doesn't know which function to fall back once the native UDF fails the validation process. In Spark SQL applications, users need to register the UDF before using it, and Gluten runs compatible with Spark SQL engine. Removing the restrictions will break the Spark SQL registration requirement. I see that makes sense. Would it make sense to perhaps gate this behind the [spark.gluten.sql.columnar.failOnFallback](https://github.com/apache/gluten/blob/6ade09d0b1e4ab2e91a849516d94a900774bbf02/gluten-substrait/src/main/scala/org/apache/gluten/config/GlutenConfig.scala#L1495-L1502 ) config? Seems there is a config in place that allows the no fallback path. Maybe I can share a bit of context why I'm hoping to see if its possible to relax this constrain a bit. Velox is a backend agnostic engine. The idea is we develop UDFs and Velox operators once and they would be applicable to all backends. In our organization we have 4 backends currently (Spark, Presto, an internal streaming backend and internal ML pre-proc backend) which are using Velox. When a user writes a Velox UDF ideally they would write it once and the same code would be applicable to all backends. Now if each backend requires re-writing the same code in the backend specific format that defeats the purpose of velox (write once and have it applicable to all backends). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
