Hi Alexander, It is beneficial for users to be able to replace/choose a connector as part of their application. When flink-connector-base is included in dist, that becomes more difficult. We can run into hard to understand dependency conflicts such as [1]. Depending on the Flink platform used, it may also be hard to update something that is part of dist. I would prefer to keep the module outside dist.
Thanks, Thomas [1] https://lists.apache.org/thread/0ht5y2tyzpt16ft36zm428182dxfs3zx On Wed, Feb 9, 2022 at 3:26 AM Alexander Fedulov <alexan...@ververica.com> wrote: > > Hi everyone, > > I would like to discuss the best approach to address the issue raised > in FLINK-25927 [1]. It can be summarized as follows: > > > flink-connector-base is currently inconsistently used in connectors > (directly shaded in some and transitively pulled in via > flink-connector-files which is itself shaded in the table uber jar) > > > FLINK-24687 [2] moved flink-connector-files out of the flink-table uber > jar > > > It is necessary to make usage of flink-connector-base consistent across > all connectors > > One approach is to stop shading flink-connector-files in all connectors and > instead package it in flink-dist, making it a part of Flink-wide provided > public API. This approach is investigated in the following PoC PR: 18545 > [3]. The issue with this approach is that it breaks any existing CI and > IDE setups that do not directly rely on flink-dist and also do not include > flink-connector-files as an explicit dependency. > > In theory, a nice alternative would be to make it a part of a dependency > that is ubiquitously provided, for instance, flink-streaming-java. Doing > that for flink-streaming-java would, however, introduce a dependency cycle > and is currently not feasible. > > It would be great to hear your opinions on what could be the best way > forward here. > > [1] https://issues.apache.org/jira/browse/FLINK-25927 > [2] https://issues.apache.org/jira/browse/FLINK-24687 > [3] https://github.com/apache/flink/pull/18545 > > > Thanks, > Alexander Fedulov
