HI, Peter. Thanks for your work. Flink devs must benefit from it. +1 for make it a separate part so that other flink connectors can also make use of it witthout including hive connector. But since we try to externalize the external connectors from Flink repo and it's related to HMS, I think it's expected be a external repo maintained by Flink community.
Best regards, Yuxia ----- 原始邮件 ----- 发件人: "Péter Váry" <peter.vary.apa...@gmail.com> 收件人: "dev" <dev@flink.apache.org> 发送时间: 星期二, 2023年 1 月 10日 下午 6:17:48 主题: Re: [Discussion] externalize Hive connector Hi Team, Somewhat, but not strictly related: - We would like to use delegation tokens to connect from the IcebergFilesCommitter tasks to kerberized Hive Metastore servers when committing changes in the Iceberg connector [1]. Gabor Somogyi is working on generalizing token support [2]. I am working on creating an implementation for the DelegationTokenProvider / DelegationTokenReceiver interfaces for HMS. This could be useful for other connectors too which are using HMS and would like to use the general delegation token framework to avoid connecting to the Kerberos server every time when a Task has to connect to the HMS. Also, as a next step, the community could implement the same for kerberized HiveServer2 too. I am still looking for a place where this flink-hms-delegation-token-provider could fit in the Flink ecosystem. My current feeling is: - It could be part of the Hive connector code base - It could be a different artifact than the real connector itself - so Iceberg, or other connectors who do not need the full Hive connector functionality could use it without including the whole Hive connector and its dependencies What do you think about this? Thanks, Peter [1] https://iceberg.apache.org/docs/latest/flink-connector/ [2] https://issues.apache.org/jira/browse/FLINK-29918 Chen Qin <qinnc...@gmail.com> ezt írta (időpont: 2023. jan. 10., K, 4:48): > Hi Martijin, > > Thank you for sharing your thoughts. In my opinion, FLINK-26603 is no > longer blocking FLINK-30064. > That being said, we could consider keeping FLINK-26603 as follow-up work > and breaking it down into smaller tasks as we proceed. > > [PR] https://github.com/apache/flink-connector-hive/pull/3 > > Chen > > On Mon, Jan 9, 2023 at 7:45 AM Martijn Visser <martijnvis...@apache.org> > wrote: > > > Hi Chen, > > > > Thanks for bringing this up! I think it would be great if the Hive > > connector is externalized. We've already previously established [1] that > it > > should be externalized. I believe the only reason why this hasn't been > done > > yet is because it's blocked by > > https://issues.apache.org/jira/browse/FLINK-26603. Is that still the > case? > > > > Best regards, > > > > Martijn > > > > [1] https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr > > > > On Mon, Jan 9, 2023 at 4:22 PM Chen Qin <qinnc...@gmail.com> wrote: > > > > > Hi there, > > > > > > Following community guidance Externalized+Connector+development > > > < > > > https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development > >, > > We > > > would like to initiate discussion on moving connector/hive to > > > apache/flink-connector-hive > > > <https://github.com/apache/flink-connector-hive>. > > > > > > Currently proposed changes includes > > > > > > - cleanup dependencies introduced from hive/yarn dependencies with > > > latest package version stated in properties section in POM file > > > - add FlinkPlannerCalciteShim to handle PlannerCalcite API signature > > > changes from 1.16 v.s 1.17-SNAPSHOT > > > - add PackageITTests and ProductionArchitectureTests > > > - [bonus] adding docker e2e tests with list of supported Hive/HMS > > > versions > > > > > > Risk associated with this change includes not being able to release > until > > > 1.17 release, so we would have to keep cherry-pick changes from > > > flink/connectors/hive for a period of time. > > > > > > Looking forward to hearing community feedback. > > > > > > Chen > > > > > >
