comphead commented on PR #4223: URL: https://github.com/apache/datafusion-comet/pull/4223#issuecomment-4383101714
> > > Are we planning to merge it asap or wait DF 54.0? > > > > > > I would love to get the JVM UDF framework in (once reviewed). > > There are many applications where it can help us get acceleration by default rather than opt-in > > > > * lambdas > > * JSON > > * regex > > > > What would be the advantage of waiting for DF 54? Does that give us 100% compatibility for array_exists with lambdas? > > I could split the JVM UDF work out into a separate PR but there would be no tests if we don't have an example of an expression using it Having no tests for lambda is fine IMO as we do not expose the feature to users right away. > Does that give us 100% compatibility for array_exists with lambdas? Thats the entire intention of `datafusion-spark` module, but the compatibility is not always true unfortunately. My main concern we could end up with multiple lambda implementation in DF and in Comet and might cause confusion and conflicts. The small poc PR shown the `array_exists` works with basic lambda(no column capture, no nested lambdas) https://github.com/apache/datafusion-comet/pull/4127/changes#diff-7411f5845a2488bb1509b95d8ad1e014422e21d70e0b802bd7624eabc4621c66 For customers we can build another branch on top of DF54 migration branch and including lambda functions there, WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
