jiangzhx commented on pull request #2131: URL: https://github.com/apache/arrow-datafusion/pull/2131#issuecomment-1086533235
> > Currently I believe the "plugin_dir is a local dir, I think it is better to support distributed file systems(HDFS/Object store) so that both the Executors and Scheduler can load the plugin files from a single place. > > Alternatively, users could package up dependencies in a Docker container and deploy that way. This could be more efficient in the case where multiple executors are running on the same node since the image will be downloaded once and cached. It also provides better version control - all executors will be guaranteed to be running the same code (assume a specific version of the image is deployed). > > I would be interested to hear more about the use case of loading dependencies from object store though. What would be the motivation of this approach? maybe in the future, we can support create custom udf&udaf like hive. ```sql CREATE FUNCTION myfunc AS 'myclass' USING JAR 'hdfs:///path/to/jar'; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
