Re: [I] Spike: evaluate if cuDF can be used with datafusion-python [datafusion-python]
JigaoLuo commented on issue #936: URL: https://github.com/apache/datafusion-python/issues/936#issuecomment-3127576345 > polars blog gives quite a lot of details about CUDA integration https://pola.rs/posts/gpu-engine-release/ . Looking the code, looks like they have integrated it in the python layer not in rust. As an external contributor to cuDF, I encountered this issue firsthand. I could confirm that cuDF and Polars interact exclusively at the Python level, as there’s currently no binding between cuDF’s C++ backend and Polars' Rust core. One point worth highlighting is the role of pylibcudf: https://docs.rapids.ai/api/cudf/stable/pylibcudf/. This Python wrapper aims to expose the entire libcudf API with minimal overhead, enabling high-performance data operations in Python. Its design allows efficient bridging between Python and C++ without incurring the usual latency associated with Python bindings. However, pylibcudf still lacks full coverage of libcudf’s functionality, and its documentation is not complete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Spike: evaluate if cuDF can be used with datafusion-python [datafusion-python]
milenkovicm commented on issue #936: URL: https://github.com/apache/datafusion-python/issues/936#issuecomment-3127193788 looks like they have used bindgen https://github.com/rapidsai/cuvs/blob/1155a3a427cd1d1bfaf8fe74a937ed6dfa797ec7/rust/cuvs-sys/build.rs#L75 to generate rust bindings -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Spike: evaluate if cuDF can be used with datafusion-python [datafusion-python]
milenkovicm commented on issue #936: URL: https://github.com/apache/datafusion-python/issues/936#issuecomment-3127187038 polars blog gives quite a lot of details about CUDA integration https://pola.rs/posts/gpu-engine-release/ . Looking the code, looks like they have integrated it in the python layer not in rust. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Spike: evaluate if cuDF can be used with datafusion-python [datafusion-python]
paleolimbot commented on issue #936: URL: https://github.com/apache/datafusion-python/issues/936#issuecomment-2916913205 Just two things I was involved in that may be useful here: - `cudf::from_arrow()`: https://github.com/rapidsai/cudf/blob/2789fa83d943649b982493d68bbba852f848d82c/cpp/include/cudf/interop.hpp#L677-L698 - https://github.com/apache/arrow-nanoarrow/blob/fe0e3a2745952c7bd415fdf09bda2a869bf1390a/src/nanoarrow/nanoarrow_device.h#L278-L288 Probably integrating at a higher level than those makes sense (e.g., converting expressions), since the trick of optimizing data transfer to/from the GPU is sort of insane. The nanoarrow version also includes a (probably oudated) Apple Metal backend, which is potentially easier to integrate because you can use the same memory addresses space for the buffers (or I could, the last time I tried this). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Spike: evaluate if cuDF can be used with datafusion-python [datafusion-python]
deanm commented on issue #936: URL: https://github.com/apache/datafusion-python/issues/936#issuecomment-2725385324 polars has a gpu engine mode. I haven't looked at how they implement it but it's probably worth checking out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Spike: evaluate if cuDF can be used with datafusion-python [datafusion-python]
drauschenbach commented on issue #936: URL: https://github.com/apache/datafusion-python/issues/936#issuecomment-2455994000 > This repo (datafusion-python) once contained a prototype of translating DataFusion logical plan to cuDF operations (all in Python). It is still there in the history somewhere. Possibly #602. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Spike: evaluate if cuDF can be used with datafusion-python [datafusion-python]
andygrove commented on issue #936: URL: https://github.com/apache/datafusion-python/issues/936#issuecomment-2441991685 I see that there is now one RAPIDS library that provides Rust bindings: https://docs.rapids.ai/api/cuvs/nightly/rust_api/ so it may be interesting to see what approach they took to wrap C++ in this case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Spike: evaluate if cuDF can be used with datafusion-python [datafusion-python]
andygrove commented on issue #936: URL: https://github.com/apache/datafusion-python/issues/936#issuecomment-2441982883 I have some experience in this area. While at NVIDIA, I created a POC with Rust bindings around cuDF and then provided interoperability with arrow-rs. Unfortunately, that code was internal and not open-source. I used cxx to create the bindings. This repo (datafusion-python) once contained a prototype of translating DataFusion logical plan to cuDF operations. It is still there in the history somewhere. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
