Re: [I] Separable Python and Rust components [datafusion-python]
timsaucer commented on issue #1193: URL: https://github.com/apache/datafusion-python/issues/1193#issuecomment-4216029400 Closing since there was no additional feedback/requests after the update of the UDFs. We also continue to expose more and more extension points. If there is a specific need that isn't addressed in the current release please feel free to re-open this ticket with a reproducible use case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Separable Python and Rust components [datafusion-python]
timsaucer closed issue #1193: Separable Python and Rust components URL: https://github.com/apache/datafusion-python/issues/1193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Separable Python and Rust components [datafusion-python]
timsaucer commented on issue #1193: URL: https://github.com/apache/datafusion-python/issues/1193#issuecomment-3134263515 Ok, I think I see what is happening then. I do think this is a problem of the unstable ABI. Are all of the extension points you're trying to get into python UDFs? If so it may be not too difficult. I just put up [this repo](https://github.com/timsaucer/datafusion_extension_type_examples) for the purpose of demonstrating extension types, but I think it may be a good example for your use as well. Specifically as long as you use 48.0.0 (both rust and python) you should be able to expose your UDFs like in [this library](https://github.com/timsaucer/datafusion_extension_type_examples/blob/main/python/src/lib.rs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Separable Python and Rust components [datafusion-python]
awhyte commented on issue #1193:
URL:
https://github.com/apache/datafusion-python/issues/1193#issuecomment-3134235415
Thanks very much for the quick response, Tim.
Sadly passing `col("a").expr` as `my_lib_function(col("a").expr)` also fails
with the same message. Perhaps the build configuration was too different - I'll
take a look at that to check.
I'm not quite sure I understand what you mean by wrapping the PyExpr?
Thank you for the link and the great write-up there - I'm not sure how I
missed that previously. Would a helpful contribution here be for me to write
`FFI_` and `Foreign` implementations for expressions? The current
`datafusion-ffi` crate seems to handle expressions via protobuf serialisation
from `datafusion-proto` which doesn't seem so nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [I] Separable Python and Rust components [datafusion-python]
timsaucer commented on issue #1193: URL: https://github.com/apache/datafusion-python/issues/1193#issuecomment-3133448537 I think this might be a duplicate of https://github.com/apache/datafusion-python/issues/853 which may need to be reopened. I suspect your immediate problem is resolved by passing `.expr` on the calls you are trying to make or wrapping your PyExpr. More generally I would also be careful about trying to share these binaries across different builds as there can be problems with the non stable ABI. There is additional information here: https://datafusion.apache.org/python/contributor-guide/ffi.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
