pan3793 commented on issue #5383:
URL: https://github.com/apache/kyuubi/issues/5383#issuecomment-2062861067
@tgravescs that's a good question, and we did have an offline discussion
about it.
TL;DR, your assumption will be the ultimate version, but not at the
beginning.
As you know the current main flow of Kyuubi is:
```
===[http]
client ===[thrift]====> Server ===[thrift]===> Engine
===[etc.] ---[thrift]---> STS/HS2/Impala (we know
someone implemented such a feature internally)
```
The engine itself is kind of a regular Spark app that basically only
consumes Spark's public API, making it easily compatible with multiple Spark
versions. As connect is a new feature and `connect-server` is not supposed to
be exposed to the user directly(I suppose only gRPC API is public API in this
case), pulling `connect-server` in the current Spark engine module directly
would break the current assumption. So in the experimental phase we are going
to create a dedicated engine module for the `connect` engine, we may call it
`SPARK_CONNECT`(the current one is SPARK_SQL)。
Another important case is `Server ===[thrift]===> Engine`, currently, we use
Thrift(more specifically, the HiveServer2 Thrift protocol) as the internal RPC
protocol, but for connect, obviously gRPC should be used, and keep two internal
RPC protocol is quite complex and redundant, we tend to create a dedicated
experimental server that keeps similar architecture but rewrite the RPC
implementation.
Once the PoC is completed, we can consider merging servers and engines to
achieve the final vision as you said.
```
===[http]
===[grpc]
client ===[thrift]====> Server ===[grpc]===> Engine
===[etc.] --[thrift]--> STS/HS2/Impala (we know someone
implemented such a feature internally)
```
Maybe @yaooqinn can share more information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
