Yuxia, thanks for details context, I see with your point on maintainability. If your don’t mind, I would like to help continue the work you mentioned.
Chen On Tue, Jan 10, 2023 at 03:47 yuxia <luoyu...@alumni.sjtu.edu.cn> wrote: > Hi, Chen. > Appreciated your efforts. > I would like to shared my thonghts about it. > > I do think it's not a good timing to externalize Hive connector right now. > As you know, the Hive connector relies some internal methods in > flink-table-planner. The Flink devs can > modify them with more freedom as they are internal methods with providing > any compatibility gurantee. > If externalize it now, it may well happen that some methods relied by Hive > connector have been removed/renamed from flink-table-planner, which will > then cause the Hive connector fail. > It'll bring much burden to the connector maintainer, the maintainer will > need to take much care of these changes and do the adaption. As I see, > it'll always end with only some users report this issue, did the maintainer > notice that. It'll be definitely a bad user experience. > > It's a histrical technical debt that Hive connector relies the internal > methods in flink-table-planner, and we have created FLINK-26603[1] to > resolve it. > I'm intended to finish it in Flink 1.17, but may well fail to catch up > with Flink 1.17. Feel sorry for that for I'm busy with other stuff. If you > have interest about it, appreciated that you can also take part in this > jira. > > Of course you can just create a seperate Hive connector now, and then > cherry pick changes from the Flink repo before the Hive connector can be > removed from Flink repo. But I'm afraid that it may still take much > efforts. And IIUC, the external Hive connector can't be released as offical > Hive connector unless we resolve the problem I refered above. > So, why not wait any thing is ready before externalize the Hive connector? > > Best regards, > Yuxia > > ----- 原始邮件 ----- > 发件人: "Chen Qin" <qinnc...@gmail.com> > 收件人: "dev" <dev@flink.apache.org> > 发送时间: 星期二, 2023年 1 月 10日 上午 11:47:02 > 主题: Re: [Discussion] externalize Hive connector > > Hi Martijin, > > Thank you for sharing your thoughts. In my opinion, FLINK-26603 is no > longer blocking FLINK-30064. > That being said, we could consider keeping FLINK-26603 as follow-up work > and breaking it down into smaller tasks as we proceed. > > [PR] https://github.com/apache/flink-connector-hive/pull/3 > > Chen > > On Mon, Jan 9, 2023 at 7:45 AM Martijn Visser <martijnvis...@apache.org> > wrote: > > > Hi Chen, > > > > Thanks for bringing this up! I think it would be great if the Hive > > connector is externalized. We've already previously established [1] that > it > > should be externalized. I believe the only reason why this hasn't been > done > > yet is because it's blocked by > > https://issues.apache.org/jira/browse/FLINK-26603. Is that still the > case? > > > > Best regards, > > > > Martijn > > > > [1] https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr > > > > On Mon, Jan 9, 2023 at 4:22 PM Chen Qin <qinnc...@gmail.com> wrote: > > > > > Hi there, > > > > > > Following community guidance Externalized+Connector+development > > > < > > > https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development > >, > > We > > > would like to initiate discussion on moving connector/hive to > > > apache/flink-connector-hive > > > <https://github.com/apache/flink-connector-hive>. > > > > > > Currently proposed changes includes > > > > > > - cleanup dependencies introduced from hive/yarn dependencies with > > > latest package version stated in properties section in POM file > > > - add FlinkPlannerCalciteShim to handle PlannerCalcite API signature > > > changes from 1.16 v.s 1.17-SNAPSHOT > > > - add PackageITTests and ProductionArchitectureTests > > > - [bonus] adding docker e2e tests with list of supported Hive/HMS > > > versions > > > > > > Risk associated with this change includes not being able to release > until > > > 1.17 release, so we would have to keep cherry-pick changes from > > > flink/connectors/hive for a period of time. > > > > > > Looking forward to hearing community feedback. > > > > > > Chen > > > > > >
