Hi, all. I would like to help to do some work about this issue. Because some classes in flink-connector-base are supposed to be used inside the user jar directly, FLINK-25927[1] has been reverted by FLINK-26701[2].
And the final solution is as follows. - package flink-connector-base into flink-dist - the external connectors will not bundle the connector-base module, which is written in the Externalized Connector development docs[3] But the most external connectors still bundle the connector-base module now. I will check this problem and stop bundling connector-base module in every externalized connector[4]. Best, Hang [1] https://issues.apache.org/jira/browse/FLINK-25927 [2] https://issues.apache.org/jira/browse/FLINK-26701 [3] https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development [4] https://issues.apache.org/jira/browse/FLINK-30400 Alexander Fedulov <alexan...@ververica.com> 于2022年2月14日周一 21:34写道： > Hi, > > Thomas, Chesnay, thank you for your input. Below I will try to capture two > actionable alternatives together with their benefits and downsides: > > Alternative #1: Package flink-connector-base into flink-dist > > Downsides: > - breaks existing CI/IDE setup that previously neither relied on flink-dist > nor added flink-connector-base as a dependency > - could break existing connectors due to conflicts between > flink-connector-base of different version (if they did not relocate it) > - more work: flink-dist needs publishing to maven central to provide a > solution for CI/IDE setups (this is currently not done) > - flink-dist is heavy: currently about 118MB, which could be potentially > reduced to ~70MB by removing parts that are not directly related to > interfaces, like flink-kubernetes, but this needs more work > > Benefits: > - consistency: flink-connector-base does not get "special treatment" when > compared to other Flink APIs > - makes it easier for connector base to use utilities of Flink (evolve > together) > - makes it easier to evolve dependency on core, table-commons (only source > compatibility required, not binary) > > > Alternative #2: shade and relocate flink-connector-base in every connector > > Downsides: > - will break connectors that were previously transitively pulling it in via > flink-connector-files/flink-table uber jar > - treats this API differently than the other Flink APIs > - increased API compatibility surface: everything that flink-connector-base > relies on (flink-core, flink-table-commons) has to be binary compatible > between the versions, not just the flink-connector-base itself > > Benefits: > - less work from the implementation perspective - flink-dist does not need > to be published > - does not break existing CI/IDE setups > - also no need to pull in the sizeable flink-dist dependency for running in > IDEs and CI > > > All in all, the issue seems to boil down to the question of API > compatibility guarantees, as has already been rightly pointed out in this > thread. The main difference between the approaches is were the > compatibility guarantee emphasis is put: > > 1: connector -> *COMPATIBLE* -> connector-base -> [core, table-common] > 2: connector -> connector-base -> *COMPATIBLE* -> [core, table-common] > > As you see, both approaches are not ideal and have their downsides. A > better solution could be the one where users rely on a single lightweight > module that encapsulates all public APIs. This module could then evolve in > sync and with strict @Public compatibility guarantees. Such an approach is > a significant effort and, as Thomas mentioned, is only hinted at in > FLIP-196 as the eventual goal. To move forwards while minimizing the > potential to break existing connectors and setups, we could try to reap the > benefits and to mitigate the downsides by combining Alternative #1 and > Alternative #2, i.e.: > > - shade and relocate all dependencies to flink-connector-base for the > connectors maintained within Flink > - add a documentation notice which asks external connector developers to > also shade and relocate flink-connector-base in their implementations > - package flink-connector-base into flink-dist > > This would allow both not to break the existing CI/IDE setups > (flink-connector-base remains included into connectors) while also not > break the connectors that were previously pulling in flink-connector-base > via flink-connector-files/flink-table. > > The mixed solution is not meant to be a permanent one, and we should > revisit the API compatibility topic in 1.16. > > Let me know what you think. > > Thanks, > Alexander Fedulov > > On Mon, Feb 14, 2022 at 10:01 AM Chesnay Schepler <ches...@apache.org> > wrote: > > > Letting connectors bundle it doesn't necessarily make it harder to > > achieve; that all depends on how we approach it; > > e.g., everything that connector-base uses from the core Flink could be > > required to also be annotated with Public(Evolving). > > (i.e., treat it as if it were externalized) > > > > On 13/02/2022 02:12, Thomas Weise wrote: > > > Hi Chesnay, > > > > > > My understanding is that source compatibility is the initial step > > > towards a stronger guarantee that will reduce the pain downstream. In > > > that spirit, I would anticipate that we are not taking steps to make > > > the long term goal harder to achieve? > > > > > > The FLIP [1] states: > > > > > > "There is no official guarantee that a program compiled against an > > > earlier version can be executed on a newer Flink cluster (no ABI > > > backwards compatibility). But eventually we should try provide this > > > guarantee." > > > > > > Cheers, > > > Thomas > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-196%3A+Source+API+stability+guarantees > > > > > > On Fri, Feb 11, 2022 at 12:26 AM Chesnay Schepler <ches...@apache.org> > > wrote: > > >> The conclusion in FLIP-196 was that we only provide source > > compatibility for Public(Evolving) APIs, which means that _everything_ > must > > be compiled against the Flink version you are using it with. > > >> > > >> Doesn't that mean that such dependency conflicts are then user errors? > > >> > > >> On 10/02/2022 20:19, Thomas Weise wrote: > > >> > > >> Hi Alexander, > > >> > > >> It is beneficial for users to be able to replace/choose a connector as > > >> part of their application. When flink-connector-base is included in > > >> dist, that becomes more difficult. We can run into hard to understand > > >> dependency conflicts such as [1]. Depending on the Flink platform > > >> used, it may also be hard to update something that is part of dist. I > > >> would prefer to keep the module outside dist. > > >> > > >> Thanks, > > >> Thomas > > >> > > >> [1] https://lists.apache.org/thread/0ht5y2tyzpt16ft36zm428182dxfs3zx > > >> > > >> On Wed, Feb 9, 2022 at 3:26 AM Alexander Fedulov > > >> <alexan...@ververica.com> wrote: > > >> > > >> Hi everyone, > > >> > > >> I would like to discuss the best approach to address the issue raised > > >> in FLINK-25927 [1]. It can be summarized as follows: > > >> > > >> flink-connector-base is currently inconsistently used in connectors > > >> > > >> (directly shaded in some and transitively pulled in via > > >> flink-connector-files which is itself shaded in the table uber jar) > > >> > > >> FLINK-24687 [2] moved flink-connector-files out of the flink-table > uber > > >> > > >> jar > > >> > > >> It is necessary to make usage of flink-connector-base consistent > across > > >> > > >> all connectors > > >> > > >> One approach is to stop shading flink-connector-files in all > connectors > > and > > >> instead package it in flink-dist, making it a part of Flink-wide > > provided > > >> public API. This approach is investigated in the following PoC PR: > 18545 > > >> [3]. The issue with this approach is that it breaks any existing CI > and > > >> IDE setups that do not directly rely on flink-dist and also do not > > include > > >> flink-connector-files as an explicit dependency. > > >> > > >> In theory, a nice alternative would be to make it a part of a > dependency > > >> that is ubiquitously provided, for instance, flink-streaming-java. > Doing > > >> that for flink-streaming-java would, however, introduce a dependency > > cycle > > >> and is currently not feasible. > > >> > > >> It would be great to hear your opinions on what could be the best way > > >> forward here. > > >> > > >> [1] https://issues.apache.org/jira/browse/FLINK-25927 > > >> [2] https://issues.apache.org/jira/browse/FLINK-24687 > > >> [3] https://github.com/apache/flink/pull/18545 > > >> > > >> > > >> Thanks, > > >> Alexander Fedulov > > >> > > >> > > > > >
