davidyuan1223 commented on issue #5383: URL: https://github.com/apache/kyuubi/issues/5383#issuecomment-2064610454
@pan3793 @yaooqinn @ulysses-you @tgravescs Hello, I have analyzed the processing flow of spark-connect, as shown in the following figure.  1. SparkSession.builder.remote(host:port).getOrCreate() to create a SparkConnectClient(RPCClient) 2. spark.sql(xxx), acutually, this method is build a rpcRequest then use RPClient to process with Spark-Connect-Server 3. Then Spark-Connect-Server receive the request and process it with local sparkSession, finally, return the rpcResponse 4. The client sparkSession receive the rpcResponse will resolve it then return As mentioned above, I believe that in the RPC request process of kyuubi based on SparkConnect, we no longer need the involvement of SparkSession, so I have designed the following process:  1. We will implement a KyuubiSparkConnectClient(RPCClient, based on SparkConnectClient). It will be created when we use EngineRef.getOrCreate to create a KyuubiSparkConnectEngine 2. Examples, like beeline, when we use beeline to execute sql, it will create a thrift request to the KyuubiSparkConnectFrontendService 3. The frontendService will not do any thing, just like other engine, then the frontendService will post request to KyuubiSparkConnectService(client: KyuubiSparkConnectClient) 4. The backendService also like other engine, it will use corresponding operation to handle the request 5. The operation will process like the follow 5.1 Process the thrift request and tranform it to rpc request 5.2 Call client method to process the request 5.3 Receive the rpc response from the Spark-Connect-Server 5.4 Tranform the rpc response to thrift response Based the rpc client, we don't need create sparkSession(i.e. local environment don't need spark !!!!) What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
