Dear community,
Today I would like to kickstart a series of discussions around creating an
external connector repository. The main idea is to decouple the release
cycle of Flink with the release cycles of the connectors. This is a common
approach in other big data analytics projects and seems to s
Hi everyone,
I created the flink-connectors repo [1] to advance the topic. We would
create a proof-of-concept in the next few weeks as a special branch that
I'd then use for discussions. If the community agrees with the approach,
that special branch will become the master. If not, we can reiterate
Hi everyone,
we are currently in the process of setting up the flink-connectors repo [1]
for new connectors but we hit a wall that we currently cannot take:
branching model.
To reiterate the original motivation of the external connector repo: We
want to decouple the release cycle of a connector wi
-connectors/releases/tag/v0.10.1
[4] https://search.maven.org/search?q=pravega-connectors-flink
Best Regards,
Brian
Internal Use - Confidential
-Original Message-
From: Arvid Heise
Sent: Friday, November 19, 2021 4:12 PM
To: dev
Subject: Re: [DISCUSS] Creating an external connector
https://github.com/pravega/flink-connectors/releases/tag/v0.10.1
> [4] https://search.maven.org/search?q=pravega-connectors-flink
>
> Best Regards,
> Brian
>
>
> Internal Use - Confidential
>
> -Original Message-
> From: Arvid Heise
> Sent: Friday, November
ttps://github.com/pravega/flink-connectors/wiki/Versioning-strategy-for-Flink-connector
> > [3] https://github.com/pravega/flink-connectors/releases/tag/v0.10.1
> > [4] https://search.maven.org/search?q=pravega-connectors-flink
> >
> > Best Regards,
> > Brian
> &g
-connectors/releases/tag/v0.10.1
[4] https://search.maven.org/search?q=pravega-connectors-flink
Best Regards,
Brian
Internal Use - Confidential
-Original Message-
From: Arvid Heise
Sent: Friday, November 19, 2021 4:12 PM
To: dev
Subject: Re: [DISCUSS] Creating an external connecto
On the
> >>> contrary, we can also do the opposite way to align with Flink version
> and
> >>> maintain several branches for different system version.
> >>>
> >>> I would say this is only a fairly-OK solution because it is a bit
> painful
> &g
will
> > publish
> > >>> snapshots for all these 3 branches).
> > >>> For example, recently we have 0.10.1 release[3], and in maven central
> > we
> > >>> need to upload three artifacts(For Flink 1.13, 1.12, 1.11) for 0.10.1
> > >>
hou, Brian
> wrote:
> > > >>
> > > >>> Hi Arvid,
> > > >>>
> > > >>> For branching model, the Pravega Flink connector has some
> experience
> > > what
> > > >>> I would like to share. Here[1][2]
t; >> better than having just one artifact that happens to run with
>> multiple
>> > > >> Flink versions. I guess it depends on what dependencies we are
>> > > exposing. If
>> > > >> the connector uses flink-connector-base, then we probably
t;> > > >>>
> >> > > >> I'd like to give connector devs a simple way to express to which
> >> Flink
> >> > > >> versions the current branch is compatible. From there we can
> >> generate
> >> > > the
> >> > > >>
t; This certainly would mean that there is little value in
> coupling
> > >> > > connector
> > >> > > >> versions. So it's making a good case for having separate
> connector
> > >> > > repos.
> > >> > > >>
ctors#compatibility-matrix
[2]
https://github.com/pravega/flink-connectors/wiki/Versioning-strategy-for-Flink-connector
[3]
https://github.com/pravega/flink-connectors/releases/tag/v0.10.1
[4]
https://search.maven.org/search?q=pravega-connectors-flink
Best Regards,
Brian
Internal Use -
) with all other connectors the repo
> >>>>> contains.
> >>>>>> But
> >>>>>>>> for connectors that get changed frequently, having a dedicated
> >>>>>> repository
> >>>>>>>> that allows indepen
easier and more frequent, I think that coupling different
>> >> connector
>> >>>>>>>> releases might be counter-productive.
>> >>>>>>>>
>> >>>>>>>> To me it sounds not very practical to mainly use a
Hi Arvid,
In general I think breaking up the big repo would be a good move with many
benefits (which you have outlined already). One concern would be how to
proceed with our docs / examples if we were to really separate out all
connectors.
1. More real-life examples would essentially now depend o
My opinion of splitting the Flink repositories hasn't changed; I'm still
in favor of it.
While it would technically be possible to release individual connectors
even if they are part of the Flink repo,
it is quite a hassle to do so and error prone due to the current branch
structure.
A split
Thanks for driving this discussion Arvid! I think this will be one giant leap
for Flink community. Externalizing connectors would give connector developers
more freedom in developing, releasing and maintaining, which can attract more
developers for contributing their connectors and expand the Fl
We are mostly talking about the freedom this would bring to the connector
authors, but we still don't have answers for the important topics:
- How exactly are we going to maintain the high quality standard of the
connectors?
- How would the connector release cycle to look like? Is this going to
af
Hi, all
I understand very well that the maintainers of the community want to move the
connector to an external system. Indeed, the development and maintenance of the
connector requires a lot of energy, and these do not involve the Flink core
framework, which can reduce the maintenance pressure
Hi folks,
thanks for joining the discussion. I'd like to give some ideas on how
certain concerns are going to be addressed:
Ingo:
> In general I think breaking up the big repo would be a good move with many
> benefits (which you have outlined already). One concern would be how to
> proceed with o
I think you're misinterpreting my comment.
Independent from the repo split we should only keep the connectors in
the Flink project that we actively maintain.
The rest we might as well just drop.
If some external people are interested in maintaining these connectors
then there's nothing stoppin
Generally, the issues are reproducibility and control.
Stuffs completely broken on the Flink side for a week? Well then so are
the connector repos.
(As-is) You can't go back to a previous version of the snapshot. Which
also means that checking out older commits can be problematic because
you'd
Thanks for initiating this discussion.
There are definitely a few things that are not optimal with our
current management of connectors. I would not necessarily characterize
it as a "mess" though. As the points raised so far show, it isn't easy
to find a solution that balances competing requiremen
Hi all,
I think it would be a huge benefit if we can achieve more frequent releases
of connectors, which are not bound to the release cycle of Flink itself. I
agree that in order to get there, we need to have stable interfaces which
are trustworthy and reliable, so they can be safely used by those
Okay I think it is clear that the majority would like to keep connectors
under the Apache Flink umbrella. That means we will not be able to have
per-connector repositories and project management, automatic dependency
bumping with Dependabot, or semi-automatic releases.
So then I'm assuming the dir
Thank you, Arvid & team, for working on this.
I would also favor one connector repository under the ASF. This will
already force us to provide better tools and more stable APIs, which
connectors developed outside of Apache Flink will benefit from, too.
Besides simplifying the formal release proce
Hey all,
I don't have much to add to the general discussion. Just a single
comment on:
that we could adjust the bylaws for the connectors such that we need
fewer PMCs to approve a release. Would it be enough to have one PMC
vote per connector release?
I think it's not an option. This
TBH I think you're overestimating how much work it is to create a
non-Flink release. Having done most of the flink-shaded releases, I
really don't see an issue of even doing weekly releases with that process.
We can not reduce the number of votes AFAIK; the ASF seems very clear on
that matter
Could you clarify what release cadence you're thinking of? There's quite
a big range that fits "more frequent than Flink" (per-commit, daily,
weekly, bi-weekly, monthly, even bi-monthly).
On 19/10/2021 14:15, Martijn Visser wrote:
Hi all,
I think it would be a huge benefit if we can achieve m
Hi,
I think Thomas raised very good questions and would like to know your
opinions if we want to move connectors out of flink in this version.
(1) is the connector API already stable?
> Separate releases would only make sense if the core Flink surface is
> fairly stable though. As evident from Ic
Hi everyone,
regarding the stability of the APIs. I think everyone agrees that
connector APIs which are stable across minor versions (1.13->1.14) are the
mid-term goal. But:
a) These APIs are still quite young, and we shouldn't make them @Public
prematurely either.
b) Isn't this *mostly* orthogo
Hi Konstantin,
> the connectors need to be adopted and require at least one release per
Flink minor release.
However, this will make the releases of connectors slower, e.g. maintain
features for multiple branches and release multiple branches.
I think the main purpose of having an external connect
Hi,
I see the stable core Flink API as a prerequisite for modularity. And
for connectors it is not just the source and sink API (source being
stable as of 1.14), but everything that is required to build and
maintain a connector downstream, such as the test utilities and
infrastructure.
Without th
Hi all,
My name is Kyle and I’m an open source developer primarily focused on
Apache Iceberg.
I’m happy to help clarify or elaborate on any aspect of our experience
working on a relatively decoupled connector that is downstream and pretty
popular.
I’d also love to be able to contribute or assist
Hi folks,
I think some questions came up and I'd like to address the question of the
timing.
Could you clarify what release cadence you're thinking of? There's quite
> a big range that fits "more frequent than Flink" (per-commit, daily,
> weekly, bi-weekly, monthly, even bi-monthly).
The short a
Thank you all, for the nice discussion!
>From my point of view, I very much like the idea of putting connectors in a
separate repository. But I would argue it should be part of Apache Flink,
similar to flink-statefun, flink-ml, etc.
I share many of the reasons for that:
- As argued many times,
Hi everyone,
>From the discussion, it seems to me that we have different opinions
whether to have an ASF umbrella repository or to host them outside of the
ASF. It also seems that this is not really the problem to solve. Since
there are many good arguments for either approach, we could simply star
39 matches
Mail list logo
