Letting connectors bundle it doesn't necessarily make it harder to
achieve; that all depends on how we approach it;
e.g., everything that connector-base uses from the core Flink could be
required to also be annotated with Public(Evolving).
(i.e., treat it as if it were externalized)
On 13/02/2022 02:12, Thomas Weise wrote:
Hi Chesnay,
My understanding is that source compatibility is the initial step
towards a stronger guarantee that will reduce the pain downstream. In
that spirit, I would anticipate that we are not taking steps to make
the long term goal harder to achieve?
The FLIP [1] states:
"There is no official guarantee that a program compiled against an
earlier version can be executed on a newer Flink cluster (no ABI
backwards compatibility). But eventually we should try provide this
guarantee."
Cheers,
Thomas
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-196%3A+Source+API+stability+guarantees
On Fri, Feb 11, 2022 at 12:26 AM Chesnay Schepler <ches...@apache.org> wrote:
The conclusion in FLIP-196 was that we only provide source compatibility for
Public(Evolving) APIs, which means that _everything_ must be compiled against
the Flink version you are using it with.
Doesn't that mean that such dependency conflicts are then user errors?
On 10/02/2022 20:19, Thomas Weise wrote:
Hi Alexander,
It is beneficial for users to be able to replace/choose a connector as
part of their application. When flink-connector-base is included in
dist, that becomes more difficult. We can run into hard to understand
dependency conflicts such as [1]. Depending on the Flink platform
used, it may also be hard to update something that is part of dist. I
would prefer to keep the module outside dist.
Thanks,
Thomas
[1] https://lists.apache.org/thread/0ht5y2tyzpt16ft36zm428182dxfs3zx
On Wed, Feb 9, 2022 at 3:26 AM Alexander Fedulov
<alexan...@ververica.com> wrote:
Hi everyone,
I would like to discuss the best approach to address the issue raised
in FLINK-25927 [1]. It can be summarized as follows:
flink-connector-base is currently inconsistently used in connectors
(directly shaded in some and transitively pulled in via
flink-connector-files which is itself shaded in the table uber jar)
FLINK-24687 [2] moved flink-connector-files out of the flink-table uber
jar
It is necessary to make usage of flink-connector-base consistent across
all connectors
One approach is to stop shading flink-connector-files in all connectors and
instead package it in flink-dist, making it a part of Flink-wide provided
public API. This approach is investigated in the following PoC PR: 18545
[3]. The issue with this approach is that it breaks any existing CI and
IDE setups that do not directly rely on flink-dist and also do not include
flink-connector-files as an explicit dependency.
In theory, a nice alternative would be to make it a part of a dependency
that is ubiquitously provided, for instance, flink-streaming-java. Doing
that for flink-streaming-java would, however, introduce a dependency cycle
and is currently not feasible.
It would be great to hear your opinions on what could be the best way
forward here.
[1] https://issues.apache.org/jira/browse/FLINK-25927
[2] https://issues.apache.org/jira/browse/FLINK-24687
[3] https://github.com/apache/flink/pull/18545
Thanks,
Alexander Fedulov
