Hi all, I'm not too familiar with Hive and HiveServer2, but I do have a couple of questions/concerns:
1. What is the relationship between this FLIP and FLIP-91? My assumption would be that this FLIP (and therefore the HiveServer2) implementation would need to be integrated in the REST Gateway, is that correct? If so, I would prefer to first complete the discussion and vote on FLIP-91, else we'll have two moving FLIPs who have a direct relationship with each other. 2. While I understand that Hive is important (in the Chinese ecosystem, not so much in Europe and the US), I still have maintenance concerns on this topic. We know that the current Hive integration isn't exactly ideal and requires a lot of work to get in better shape. At the same time, Hive still doesn't support Java 11 while we need (and should, given the premier support has ended already) to move away from Java 8. Best regards, Martijn Visser https://twitter.com/MartijnVisser82 https://github.com/MartijnVisser On Mon, 25 Apr 2022 at 12:13, Jark Wu <imj...@gmail.com> wrote: > Thank Shengkai for driving this effort, > I think this is an essential addition to Flink Batch. > > I have some small suggestions: > 1) Kyuubi provides three ways to configure Hive metastore [1]. Could we > provide similar abilities? > Especially with the JDBC Connection URL, users can visit different Hive > metastore server instances. > > 2) I think we can improve the "HiveServer2 Compatibility" section. > We need to figure out two compatibility matrices. One is SQL Gateway with > different versions of Hive metastore, > and the other is different versions of Hive client (e.g., Hive JDBC) with > SQL Gateway. We need to clarify > what metastore and client versions we support and how users configure the > versions. > > Best, > Jark > > > [1]: > > https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <fskm...@gmail.com> wrote: > > > Hi, Jiang. > > > > Thanks for your feedback! > > > > > Integrating the Hive ecosystem should not require changing the service > > interface > > > > I move the API change to the FLIP-91. But I think it's possible we add > more > > interfaces to intergrate the new endpoints in the future because every > > endpoints's functionality is different. For example, the REST endpoint > > doen't support to fetch operation-level logs but the hiveserver2 endpoint > > supports. In this case, we need to modify the shared GatewayService to > > support the functionality exposed by the new endpint. > > > > > How to support different Hive versions? > > > > Do you means to support the different HiveServer2 version? The > HiveServer2 > > uses the version to guarantee the compatibility. During the openSession, > > the client and server will determine the protocol version(minimun(client > > version, hiveendpoint version)). After that the client and the server > uses > > the determined version to communicate. In the HiveServer2 endpoint, it > > determines how the endpoint deserialize the results and the result > schema. > > I add a section about HiveServer2 compatiblity. > > > > > Could you please fully provide its definition including input > parameters > > and the corresponding return value schema? > > > > Because we implements the interface exposed by the Hive. So I add the > file > > link to the HiveServer2 interfaces[1], which contains all input > parameters > > and the results. Considering the file doesn't contain the output for the > > Operation, I add the output schema for all the supported Operation in the > > FLIP, which is not covered in the link. Hope these can address your > > question. > > > > Best, > > Shengkai > > > > [1] > > > > > https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227 > > > > > > > > > > > > Nicholas Jiang <nicholasji...@apache.org> 于2022年4月22日周五 16:43写道： > > > > > Hi Shengkai. > > > > > > Thanks for driving the proposal of HiveServer2 Endpoint support. For > the > > > "GatewayService API Change", I don't think the motivation for > supporting > > > HiveServer2 endpoint need to change the GatewayService API, in other > > words, > > > integrating the Hive ecosystem should not require changing the service > > > interface. If you confirm to change GatewayService interface, IMO, the > > > proposal could be discussed in FLIP-91 because the public interfaces > are > > > defined in FLIP-91. > > > > > > In addtion, how to support different Hive versions and how to guarantee > > > compatibility is not mentioned in the design. What's the behavior of > the > > > compatibility? > > > > > > Finally, for the public interfaces, could you please fully provide its > > > definition including input parameters and the corresponding return > value > > > schema? > > > > > > Thanks, > > > Nicholas Jiang > > > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote: > > > > Hi, Flink developers. > > > > > > > > I want to start a discussion about the FLIP-223: Support HiveServer2 > > > > Endpoint[1]. The Endpoint will implement the thrift interface exposed > > by > > > > the HiveServer2, and users' BI, CLI and other tools based on the > > > > HiveServer2 can also be seamlessly migrated to the Flink SQL Gateway. > > > After > > > > the FLIP finishes, the users can have almost the same experience in > the > > > > Flink SQL Gateway with the HiveServer2 endpoint as in the > HiveServer2. > > > > > > > > > > > > I am glad that you can give some feedback about FLIP-223. > > > > > > > > Best, > > > > Shengkai > > > > > > > > [1] > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint > > > > > > > > > >
