Re: Table API thrift support

Hi Chen, If you want to split these two things, then "We suggest user upgrade hive metastore version to match thrift version given keeping two thrift version leads to hive connector dependencies complicated to manage." seems like something that won't work. This reads like in order for you to intro

Re: Table API thrift support

Hi Martjin, We would like to propose split externalizing flink-connector-hive (connector) from supporting thrift format (new format encode/decode). As we discussed, Externalized+Connector+development , Hive conn

Re: Table API thrift support

Hi Chen, Everything on connector externalization is documented at https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development, including the links to the relevant discussions on that topic in the community. Thanks, Martijn On Mon, Nov 28, 2022 at 11:59 AM Chen Qin wrot

Re: Table API thrift support

Hi Martijn, I feel our proposal “shading libthrift in hive connector” seems pinching a new problem “how to externalization connectors”. I assume there might be some discussion in community already. If so please kindly pass some contexts. I would incline take back shading proposal at this point.

Re: Table API thrift support

Hi Chen, While I agree that Hive Metastore is a crucial component for a lot of companies, this isn't the case for all companies. Right now it sounds like Flink has to take on tech debt because users of Flink are running on older versions of the Hive Metastore. I don't think that's a good idea at a

Re: Table API thrift support

Hi Martjin, "shading Thrift libraries from the Hive connector" Hivemetastore is foundational software running in many companies used by Spark/Flink... etc. Upgrading the hive metastore touches many pieces of data engineering. If the user updates flink job jar dependency to the latest 0.17, it woul

Re: Table API thrift support

Hi Chen, I'm a bit skeptical of shading Thrift libraries from the Hive connector, especially with the plans to externalize connectors (including Hive). Have we considered getting the versions in sync to avoid the need of any shading? The FLIP also shows a version of Thrift (0.5.0-p6) that I don't

Re: Table API thrift support

Hi Yuxia, Martijin, Thanks for your feedback on FLIP-237! My understanding is that FLIP-237 better focuses on thrift encoding/decoding in Datastream/Table API/ Pyflink. To address feedbacks, I made follow changes to FLIP-237

Re: Table API thrift support

On Mon, May 30, 2022 at 7:35 AM Martijn Visser wrote: > Hi Chen, > > I think the best starting point would be to create a FLIP [1]. One of the > important topics from my point of view is to make sure that such changes > are not only available for SQL users, but are also being considered for > Tab

Re: Table API thrift support

Thanks for raising it. But I wonder what do you mean by saying "dynamic overwrite schema type information loaded from HiveCatalog". For such case, what does the HiveCatalog store? Best regards, Yuxia - 原始邮件 - 发件人: "Chen Qin" 收件人: "dev" 发送时间: 星期一, 2022年 5 月 30日 上午 1:04:29 主题: Table API

Re: Table API thrift support

Hi Chen, I think the best starting point would be to create a FLIP [1]. One of the important topics from my point of view is to make sure that such changes are not only available for SQL users, but are also being considered for Table API, DataStream and/or Python. There might be reasons why not to