mach-kernel commented on PR #1333:
URL:
https://github.com/apache/datafusion-ballista/pull/1333#issuecomment-3446832487
> I don't disagree with what you say, we only differ in opinion who should
provide this. I believe you could extend current ballista to do this for you
(or you could do it if we add ability to register additional gprc services as I
have mentioned)
I have one tiny disagreement here: providing an RPC extension hook is fine,
but it doesn't change the developer experience much. In other words I would
still need to compile/vendor a scheduler server with my custom extension to be
able to do this with the client.
Would we be able to meet in the middle where you accept the additional
scheduler RPC only (`get_catalog`)? I think this would be a net benefit for
everyone:
- IMO the scheduler exposing some catalog metadata isn't a big maintenance
burden, and useful to have
- It can be gated by a config setting, off by default, etc.
- The rest of this functionality (the codec, remote table provider,
"populate session catalog") can live in a separate user-provided library
- And notably, if all Ballista schedulers have this RPC endpoint, then the
UX for consumers of the client libraries would be easier. e.g., in the Python
lib case, maybe they can install the `ballista` library and also a
`ballista-scheduler-catalog` then call a function to opt in to this behavior.
But they wouldn't need to compile their own scheduler.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
