gabotechs commented on issue #1612: URL: https://github.com/apache/datafusion-python/issues/1612#issuecomment-4885592723
The perceived end-result from a user standpoint is very good with Option 1, as distribution would be seamlessly integrated, and it would look like it works "out-of-the-box". The sum of the maintenance burden in all the involved projects (`datafusion`, `datafusion-distributed` and `datafusion-python`) is as small as it can get. The complexity of exposing all the customization options of `datafusion-distributed` to users is minimal (e.g., building custom distributed plans injecting network boundaries manually, customizing how custom plans are distributed across workers, etc...). On the other hand, it'd imply treating `datafusion-distributed` as special, it would transfer the maintenance burden to `datafusion-python` maintainers, and it would open the door for a bunch of other external projects to want to do the same. Still, the argument about seamless integration and end-user UX sounds like the most appealing. At the end of the day, I doubt `datafusion-python` users care about the ownership model of the different projects involved, most likely they really just want to get the job done with a setup as simple and consistent possible. So, rather than constructing from the implementation, it might be worth starting from the user experience. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
