gabotechs commented on issue #1612: URL: https://github.com/apache/datafusion-python/issues/1612#issuecomment-4885489317
Gathering some thoughts and learnings from a private conversation with @timsaucer. There are two diametrically opposite ways of doing this: **Option 1 — Bundled feature build** `datafusion-distributed` is added as a Rust dependency of `datafusion-python`, gated behind an off-by-default cargo feature, and compiled into the same wheel. There's no ABI boundary, so the internal wiring (distributed planner, worker server, codecs) is done in plain Rust exactly as the library expects. Users opt in by installing the feature-enabled build; those who don't pay nothing. The cost is that `datafusion-python` then is responsible for maintaining all the wiring. It would also imply treating `datafusion-distributed` as "special", as it's not scalable that all external projects follow this approach. **Option 2 — Foreign plugin** `datafusion-distributed` remains its own separately-built package that plugs into `datafusion-python` at runtime through datafusion-ffi's stable ABI (PyCapsule). `datafusion-python` takes no dependency on it, keeping the two projects fully decoupled in versioning and governance. The cost is that every interaction must be expressed as an FFI-stable type, and anything the FFI layer can't carry simply can't cross. This is for example the approach Ballista took. The amount of customization options `datafusion-distributed` offers to users can make this approach complicated though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
