Hi. Godfrey. Nice to hear the comments from you.
> Could you give a whole architecture about the Ecosystem of HiveServers > and the SqlGateway, such as JDBC driver, Beeline, etc. > Which is more clear for users. Yes. I have updated the FLIP and added the architecture of the Gateway with the HiveServer2 endpoint. > How To Use >> Could you give a complete example to describe an end-to-end case? Yes. I have updated the FLIP. The beeline users can just use the connect command to connect to the SQLGateway with the HiveServer2 endpoint. For example, users just inputs "!connect jdbc:hive2://<host>:<port>/<db>;auth=noSasl hiveuser pass" into the terminal to connect to the SQLGateway. > Is the streaming SQL supported? What's the behavior if I submit a streaming query or I change the dialect to 'default'? Yes. We don't limit the usage here. Users can switch to the streaming mode or use the default dialect. But we don't suggest users use the hive dialect in the streaming mode. As far as I know, it has some problems that are not fixed yet, e.g. you may get errors for SQL that works in the batch mode. I added a section to mention this. > Considering the different users may have different requirements to connect to different meta stores, > they can use the DDL to register the HiveCatalog that satisfies their requirements. >> Could you give some examples to explain it more? Hive supports setting multiple metastore addresses via the config option "hive.metastore.urls". Here I just mean users can switch to connect to different metastore instances using the CREATE CATALOG DDL. I updated the FLIP to make it more clear. Best, Shengkai godfrey he <godfre...@gmail.com> 于2022年6月6日周一 13:45写道： > Hi Shengkai, > > Thanks for driving this. > > I have a few comments: > > Could you give a whole architecture about the Ecosystem of HiveServers > and the SqlGateway, such as JDBC driver, Beeline, etc. > Which is more clear for users. > > > Considering the different users may have different requirements to > connect to different meta stores, > > they can use the DDL to register the HiveCatalog that satisfies their > requirements. > Could you give some examples to explain it more? > > > How To Use > Could you a complete example to describe an end-to-end case? > > Is the streaming sql supported? What's the behavior if I submit streaming > query > or I change the dialect to 'default'? > > Best, > Godfrey > > Shengkai Fang <fskm...@gmail.com> 于2022年6月1日周三 21:13写道： > > > > Hi, Jingsong. > > > > Thanks for your feedback. > > > > > I've read the FLIP and it's not quite clear what the specific > unsupported > > items are > > > > Yes. I have added a section named Difference with HiveServer2 and list > the > > difference between the SQL Gateway with HiveServer2 endpoint and > > HiveServer2. > > > > > Support multiple metastore clients in one gateway? > > > > Yes. It may cause class conflicts when using the different versions of > Hive > > Catalog at the same time. I add a section named "How to use" to remind > the > > users don't use HiveCatalog with different versions together. > > > > > Hive versions and setup > > > > Considering the HiveServer2 endpoint binds to the HiveCatalog, we will > not > > introduce a new module about the HiveServer2 endpoint. The current > > dependencies in the hive connector should be enough for the HiveServer2 > > Endpoint except for the hive-service-RPC(it contains the HiveServer2 > > interface). In this way, the hive connector jar will contain an > endpoint. I > > add a section named "Merge HiveServer2 Endpoint into Hive Connector > > Module". > > > > For usage, the user can just add the hive connector jar into the > classpath > > and use the sql-gateway.sh to start the SQL Gateway with the hiveserver2 > > endpoint. You can refer to the section "How to use" for more details. > > > > Best, > > Shengkai > > > > Jingsong Li <jingsongl...@gmail.com> 于2022年6月1日周三 15:04写道： > > > > > Hi Shengkai, > > > > > > Thanks for driving. > > > > > > I have a few comments: > > > > > > ## Unsupported features > > > > > > I've read the FLIP and it's not quite clear what the specific > unsupported > > > items are? > > > - For example, security related, is it not supported. > > > - For example, is there a loss of precision for types > > > - For example, the FetchResults are not the same > > > > > > ## Support multiple metastore clients in one gateway? > > > > > > > During the setup, the HiveServer2 tires to load the config in the > > > hive-site.xml to initialize the Hive metastore client. In the Flink, > we use > > > the Catalog interface to connect to the Hive Metastore, which is > allowed to > > > communicate with different Hive Metastore[1]. Therefore, we allows the > user > > > to specify the path of the hive-site.xml as the endpoint parameters, > which > > > will used to create the default HiveCatalog in the Flink. Considering > the > > > different users may have different requirements to connect to different > > > meta stores, they can use the DDL to register the HiveCatalog that > > > satisfies their requirements. > > > > > > I understand it is difficult. You really want to support? > > > > > > ## Hive versions and setup > > > > > > I saw jark also commented, but FLIP does not seem to have been > modified, > > > how should the user setup, which jar to add, which hive metastore > version > > > to support? How to setup to support? > > > > > > Best, > > > Jingsong > > > > > > On Tue, May 24, 2022 at 11:57 AM Shengkai Fang <fskm...@gmail.com> > wrote: > > > > > > > Hi, all. > > > > > > > > Considering we start to vote for FLIP-91 for a while, I think we can > > > > restart the discussion about the FLIP-223. > > > > > > > > I am glad that you can give some feedback about FLIP-223. > > > > > > > > Best, > > > > Shengkai > > > > > > > > > > > > Martijn Visser <mart...@ververica.com> 于2022年5月6日周五 19:10写道： > > > > > > > > > Hi Shengkai, > > > > > > > > > > Thanks for clarifying. > > > > > > > > > > Best regards, > > > > > > > > > > Martijn > > > > > > > > > > On Fri, 6 May 2022 at 08:40, Shengkai Fang <fskm...@gmail.com> > wrote: > > > > > > > > > > > Hi Martijn. > > > > > > > > > > > > > So this implementation would not rely in any way on Hive, only > on > > > > > Thrift? > > > > > > > > > > > > Yes. The dependency is light. We also can just copy the iface > file > > > > from > > > > > > the Hive repo and maintain by ourselves. > > > > > > > > > > > > Best, > > > > > > Shengkai > > > > > > > > > > > > Martijn Visser <martijnvis...@apache.org> 于2022年5月4日周三 21:44写道： > > > > > > > > > > > > > Hi Shengkai, > > > > > > > > > > > > > > > Actually we will only rely on the API in the Hive, which only > > > > > contains > > > > > > > the thrift file and the generated code > > > > > > > > > > > > > > So this implementation would not rely in any way on Hive, only > on > > > > > Thrift? > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Martijn Visser > > > > > > > https://twitter.com/MartijnVisser82 > > > > > > > https://github.com/MartijnVisser > > > > > > > > > > > > > > > > > > > > > On Fri, 29 Apr 2022 at 05:16, Shengkai Fang <fskm...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > > > Hi, Jark and Martijn > > > > > > > > > > > > > > > > Thanks for your feedback. > > > > > > > > > > > > > > > > > Kyuubi provides three ways to configure Hive metastore [1]. > > > Could > > > > > we > > > > > > > > provide similar abilities? > > > > > > > > > > > > > > > > Yes. I have updated the FLIP about this and it takes some > time to > > > > > > figure > > > > > > > > out how the jdbc driver works. I added the section about how > to > > > use > > > > > the > > > > > > > > hive JDBC to configure the session-level catalog. > > > > > > > > > > > > > > > > > I think we can improve the "HiveServer2 Compatibility" > section. > > > > > > > > > > > > > > > > Yes. I have updated the FLIP and added more details about the > > > > > > > > compatibility. > > > > > > > > > > > > > > > > > Prefer to first complete the discussion and vote on > FLIP-91 > > > then > > > > > > > discuss > > > > > > > > FLIP-223 > > > > > > > > > > > > > > > > Of course. We can wait until the discussion of the FLIP-91 > > > > finishes. > > > > > > > > > > > > > > > > > Maintenance concerns about the hive > > > > > > > > > > > > > > > > Actually we will only rely on the API in the Hive, which only > > > > > contains > > > > > > > the > > > > > > > > thrift file and the generated code[1]. I think it will not > > > > influence > > > > > us > > > > > > > to > > > > > > > > upgrade the java version. > > > > > > > > > > > > > > > > [1] https://github.com/apache/hive/tree/master/service-rpc > > > > > > > > > > > > > > > > Best, > > > > > > > > Shengkai > > > > > > > > > > > > > > > > Martijn Visser <martijnvis...@apache.org> 于2022年4月26日周二 > 20:44写道： > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > I'm not too familiar with Hive and HiveServer2, but I do > have a > > > > > > couple > > > > > > > of > > > > > > > > > questions/concerns: > > > > > > > > > > > > > > > > > > 1. What is the relationship between this FLIP and FLIP-91? > My > > > > > > > assumption > > > > > > > > > would be that this FLIP (and therefore the HiveServer2) > > > > > > implementation > > > > > > > > > would need to be integrated in the REST Gateway, is that > > > correct? > > > > > If > > > > > > > so, > > > > > > > > I > > > > > > > > > would prefer to first complete the discussion and vote on > > > > FLIP-91, > > > > > > else > > > > > > > > > we'll have two moving FLIPs who have a direct relationship > with > > > > > each > > > > > > > > other. > > > > > > > > > > > > > > > > > > 2. While I understand that Hive is important (in the > Chinese > > > > > > ecosystem, > > > > > > > > not > > > > > > > > > so much in Europe and the US), I still have maintenance > > > concerns > > > > on > > > > > > > this > > > > > > > > > topic. We know that the current Hive integration isn't > exactly > > > > > ideal > > > > > > > and > > > > > > > > > requires a lot of work to get in better shape. At the same > > > time, > > > > > Hive > > > > > > > > still > > > > > > > > > doesn't support Java 11 while we need (and should, given > the > > > > > premier > > > > > > > > > support has ended already) to move away from Java 8. > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > Martijn Visser > > > > > > > > > https://twitter.com/MartijnVisser82 > > > > > > > > > https://github.com/MartijnVisser > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 25 Apr 2022 at 12:13, Jark Wu <imj...@gmail.com> > > > wrote: > > > > > > > > > > > > > > > > > > > Thank Shengkai for driving this effort, > > > > > > > > > > I think this is an essential addition to Flink Batch. > > > > > > > > > > > > > > > > > > > > I have some small suggestions: > > > > > > > > > > 1) Kyuubi provides three ways to configure Hive metastore > > > [1]. > > > > > > Could > > > > > > > we > > > > > > > > > > provide similar abilities? > > > > > > > > > > Especially with the JDBC Connection URL, users can visit > > > > > different > > > > > > > Hive > > > > > > > > > > metastore server instances. > > > > > > > > > > > > > > > > > > > > 2) I think we can improve the "HiveServer2 Compatibility" > > > > > section. > > > > > > > > > > We need to figure out two compatibility matrices. One is > SQL > > > > > > Gateway > > > > > > > > with > > > > > > > > > > different versions of Hive metastore, > > > > > > > > > > and the other is different versions of Hive client (e.g., > > > Hive > > > > > > JDBC) > > > > > > > > with > > > > > > > > > > SQL Gateway. We need to clarify > > > > > > > > > > what metastore and client versions we support and how > users > > > > > > configure > > > > > > > > the > > > > > > > > > > versions. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Jark > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations > > > > > > > > > > > > > > > > > > > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang < > > > fskm...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi, Jiang. > > > > > > > > > > > > > > > > > > > > > > Thanks for your feedback! > > > > > > > > > > > > > > > > > > > > > > > Integrating the Hive ecosystem should not require > > > changing > > > > > the > > > > > > > > > service > > > > > > > > > > > interface > > > > > > > > > > > > > > > > > > > > > > I move the API change to the FLIP-91. But I think it's > > > > possible > > > > > > we > > > > > > > > add > > > > > > > > > > more > > > > > > > > > > > interfaces to intergrate the new endpoints in the > future > > > > > because > > > > > > > > every > > > > > > > > > > > endpoints's functionality is different. For example, > the > > > REST > > > > > > > > endpoint > > > > > > > > > > > doen't support to fetch operation-level logs but the > > > > > hiveserver2 > > > > > > > > > endpoint > > > > > > > > > > > supports. In this case, we need to modify the shared > > > > > > GatewayService > > > > > > > > to > > > > > > > > > > > support the functionality exposed by the new endpint. > > > > > > > > > > > > > > > > > > > > > > > How to support different Hive versions? > > > > > > > > > > > > > > > > > > > > > > Do you means to support the different HiveServer2 > version? > > > > The > > > > > > > > > > HiveServer2 > > > > > > > > > > > uses the version to guarantee the compatibility. > During the > > > > > > > > > openSession, > > > > > > > > > > > the client and server will determine the protocol > > > > > > > > > version(minimun(client > > > > > > > > > > > version, hiveendpoint version)). After that the client > and > > > > the > > > > > > > server > > > > > > > > > > uses > > > > > > > > > > > the determined version to communicate. In the > HiveServer2 > > > > > > endpoint, > > > > > > > > it > > > > > > > > > > > determines how the endpoint deserialize the results > and the > > > > > > result > > > > > > > > > > schema. > > > > > > > > > > > I add a section about HiveServer2 compatiblity. > > > > > > > > > > > > > > > > > > > > > > > Could you please fully provide its definition > including > > > > input > > > > > > > > > > parameters > > > > > > > > > > > and the corresponding return value schema? > > > > > > > > > > > > > > > > > > > > > > Because we implements the interface exposed by the > Hive. > > > So I > > > > > add > > > > > > > the > > > > > > > > > > file > > > > > > > > > > > link to the HiveServer2 interfaces[1], which contains > all > > > > input > > > > > > > > > > parameters > > > > > > > > > > > and the results. Considering the file doesn't contain > the > > > > > output > > > > > > > for > > > > > > > > > the > > > > > > > > > > > Operation, I add the output schema for all the > supported > > > > > > Operation > > > > > > > in > > > > > > > > > the > > > > > > > > > > > FLIP, which is not covered in the link. Hope these can > > > > address > > > > > > your > > > > > > > > > > > question. > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Shengkai > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nicholas Jiang <nicholasji...@apache.org> > 于2022年4月22日周五 > > > > > 16:43写道： > > > > > > > > > > > > > > > > > > > > > > > Hi Shengkai. > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving the proposal of HiveServer2 > Endpoint > > > > > > support. > > > > > > > > For > > > > > > > > > > the > > > > > > > > > > > > "GatewayService API Change", I don't think the > motivation > > > > for > > > > > > > > > > supporting > > > > > > > > > > > > HiveServer2 endpoint need to change the > GatewayService > > > API, > > > > > in > > > > > > > > other > > > > > > > > > > > words, > > > > > > > > > > > > integrating the Hive ecosystem should not require > > > changing > > > > > the > > > > > > > > > service > > > > > > > > > > > > interface. If you confirm to change GatewayService > > > > interface, > > > > > > > IMO, > > > > > > > > > the > > > > > > > > > > > > proposal could be discussed in FLIP-91 because the > public > > > > > > > > interfaces > > > > > > > > > > are > > > > > > > > > > > > defined in FLIP-91. > > > > > > > > > > > > > > > > > > > > > > > > In addtion, how to support different Hive versions > and > > > how > > > > to > > > > > > > > > guarantee > > > > > > > > > > > > compatibility is not mentioned in the design. What's > the > > > > > > behavior > > > > > > > > of > > > > > > > > > > the > > > > > > > > > > > > compatibility? > > > > > > > > > > > > > > > > > > > > > > > > Finally, for the public interfaces, could you please > > > fully > > > > > > > provide > > > > > > > > > its > > > > > > > > > > > > definition including input parameters and the > > > corresponding > > > > > > > return > > > > > > > > > > value > > > > > > > > > > > > schema? > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Nicholas Jiang > > > > > > > > > > > > > > > > > > > > > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote: > > > > > > > > > > > > > Hi, Flink developers. > > > > > > > > > > > > > > > > > > > > > > > > > > I want to start a discussion about the FLIP-223: > > > Support > > > > > > > > > HiveServer2 > > > > > > > > > > > > > Endpoint[1]. The Endpoint will implement the thrift > > > > > interface > > > > > > > > > exposed > > > > > > > > > > > by > > > > > > > > > > > > > the HiveServer2, and users' BI, CLI and other tools > > > based > > > > > on > > > > > > > the > > > > > > > > > > > > > HiveServer2 can also be seamlessly migrated to the > > > Flink > > > > > SQL > > > > > > > > > Gateway. > > > > > > > > > > > > After > > > > > > > > > > > > > the FLIP finishes, the users can have almost the > same > > > > > > > experience > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > > Flink SQL Gateway with the HiveServer2 endpoint as > in > > > the > > > > > > > > > > HiveServer2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am glad that you can give some feedback about > > > FLIP-223. > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > Shengkai > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
