Thank Shengkai for driving this effort, I think this is an essential addition to Flink Batch.
I have some small suggestions: 1) Kyuubi provides three ways to configure Hive metastore [1]. Could we provide similar abilities? Especially with the JDBC Connection URL, users can visit different Hive metastore server instances. 2) I think we can improve the "HiveServer2 Compatibility" section. We need to figure out two compatibility matrices. One is SQL Gateway with different versions of Hive metastore, and the other is different versions of Hive client (e.g., Hive JDBC) with SQL Gateway. We need to clarify what metastore and client versions we support and how users configure the versions. Best, Jark [1]: https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <fskm...@gmail.com> wrote: > Hi, Jiang. > > Thanks for your feedback! > > > Integrating the Hive ecosystem should not require changing the service > interface > > I move the API change to the FLIP-91. But I think it's possible we add more > interfaces to intergrate the new endpoints in the future because every > endpoints's functionality is different. For example, the REST endpoint > doen't support to fetch operation-level logs but the hiveserver2 endpoint > supports. In this case, we need to modify the shared GatewayService to > support the functionality exposed by the new endpint. > > > How to support different Hive versions? > > Do you means to support the different HiveServer2 version? The HiveServer2 > uses the version to guarantee the compatibility. During the openSession, > the client and server will determine the protocol version(minimun(client > version, hiveendpoint version)). After that the client and the server uses > the determined version to communicate. In the HiveServer2 endpoint, it > determines how the endpoint deserialize the results and the result schema. > I add a section about HiveServer2 compatiblity. > > > Could you please fully provide its definition including input parameters > and the corresponding return value schema? > > Because we implements the interface exposed by the Hive. So I add the file > link to the HiveServer2 interfaces[1], which contains all input parameters > and the results. Considering the file doesn't contain the output for the > Operation, I add the output schema for all the supported Operation in the > FLIP, which is not covered in the link. Hope these can address your > question. > > Best, > Shengkai > > [1] > > https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227 > > > > > > Nicholas Jiang <nicholasji...@apache.org> 于2022年4月22日周五 16:43写道： > > > Hi Shengkai. > > > > Thanks for driving the proposal of HiveServer2 Endpoint support. For the > > "GatewayService API Change", I don't think the motivation for supporting > > HiveServer2 endpoint need to change the GatewayService API, in other > words, > > integrating the Hive ecosystem should not require changing the service > > interface. If you confirm to change GatewayService interface, IMO, the > > proposal could be discussed in FLIP-91 because the public interfaces are > > defined in FLIP-91. > > > > In addtion, how to support different Hive versions and how to guarantee > > compatibility is not mentioned in the design. What's the behavior of the > > compatibility? > > > > Finally, for the public interfaces, could you please fully provide its > > definition including input parameters and the corresponding return value > > schema? > > > > Thanks, > > Nicholas Jiang > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote: > > > Hi, Flink developers. > > > > > > I want to start a discussion about the FLIP-223: Support HiveServer2 > > > Endpoint[1]. The Endpoint will implement the thrift interface exposed > by > > > the HiveServer2, and users' BI, CLI and other tools based on the > > > HiveServer2 can also be seamlessly migrated to the Flink SQL Gateway. > > After > > > the FLIP finishes, the users can have almost the same experience in the > > > Flink SQL Gateway with the HiveServer2 endpoint as in the HiveServer2. > > > > > > > > > I am glad that you can give some feedback about FLIP-223. > > > > > > Best, > > > Shengkai > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint > > > > > >
