Hi everyone, I wanted to summarise the email thread and see if there are any open items that still need to be discussed, before we can finalise the discussion in this email thread:
1. About having multi connectors in one repo or each connector in its own repository As explained by @Arvid Heise <ar...@apache.org> we ultimately propose to have a single repository per connector, which seems to be favoured in the community. 2. About having the connector repositories under ASF or not. The consensus is that all connectors would remain under the ASF. I think we can categorise the questions or concerns that are brought forward as the following one: 3. How would we set up the testing? We need to make sure that we provide a proper testing framework, which means that we provide a public Source- and Sink testing framework. As mentioned extensively in the thread, we need to make sure that the necessary interfaces are properly annotated and at least @PublicEvolving. This also includes the test infrastructure, like MiniCluster. For the latter, we don't know exactly yet how to balance having publicly available test infrastructure vs being able to iterate inside of Flink, but we can all agree this has to be solved. For testing infrastructure, we would like to use Github Actions. In the current state, it probably makes sense for a connector repo to follow the branching strategy of Flink. That will ensure a match between the released connector and Flink version. This should change when all the Flink interfaces have stabilised so you can use a connector with multiple Flink versions. That means that we should have a nightly build test for: - The `main` branch of the connector (which would be the unreleased version) against the `master` branch of Flink (the unreleased version of Flink). - Any supported `release-X.YY` branch of the connector against the `release-X.YY` branch of Flink. We should also have a smoke test E2E tests in Flink (one for DataStream, one for Table, one for SQL, one for Python) which loads all the connectors and does an arbitrary test (post data on source, load into Flink, sink output and compare that output is as expected. 4. How would we integrate documentation? Documentation for a connector should probably end up in the connector repository. The Flink website should contain one entrance to all connectors (so not the current approach where we have connectors per DataStream API, Table API etc). Each connector documentation should end up as one menu item in connectors, containing all necessary information for all DataStream, Table, SQL and Python implementations. 5. Which connectors should end up in the external connector repo? I'll open up a separate thread on this topic to have a parallel discussion on that. We should reach consensus on both threads before we can move forward on this topic as a whole. Best regards, Martijn On Fri, 10 Dec 2021 at 04:47, Thomas Weise <t...@apache.org> wrote: > +1 for repo per connector from my side also > > Thanks for trying out the different approaches. > > Where would the common/infra pieces live? In a separate repository > with its own release? > > Thomas > > On Thu, Dec 9, 2021 at 12:42 PM Till Rohrmann <trohrm...@apache.org> > wrote: > > > > Sorry if I was a bit unclear. +1 for the single repo per connector > approach. > > > > Cheers, > > Till > > > > On Thu, Dec 9, 2021 at 5:41 PM Till Rohrmann <trohrm...@apache.org> > wrote: > > > > > +1 for the single repo approach. > > > > > > Cheers, > > > Till > > > > > > On Thu, Dec 9, 2021 at 3:54 PM Martijn Visser <mart...@ververica.com> > > > wrote: > > > > > >> I also agree that it feels more natural to go with a repo for each > > >> individual connector. Each repository can be made available at > > >> flink-packages.org so users can find them, next to referring to them > in > > >> documentation. +1 from my side. > > >> > > >> On Thu, 9 Dec 2021 at 15:38, Arvid Heise <ar...@apache.org> wrote: > > >> > > >> > Hi all, > > >> > > > >> > We tried out Chesnay's proposal and went with Option 2. > Unfortunately, > > >> we > > >> > experienced tough nuts to crack and feel like we hit a dead end: > > >> > - The main pain point with the outlined Frankensteinian connector > repo > > >> is > > >> > how to handle shared code / infra code. If we have it in some > <common> > > >> > branch, then we need to merge the common branch in the connector > branch > > >> on > > >> > update. However, it's unclear to me how improvements in the common > > >> branch > > >> > that naturally appear while working on a specific connector go back > into > > >> > the common branch. You can't use a pull request from your branch or > else > > >> > your connector code would poison the connector-less common branch. > So > > >> you > > >> > would probably manually copy the files over to a common branch and > > >> create a > > >> > PR branch for that. > > >> > - A weird solution could be to have the common branch as a > submodule in > > >> the > > >> > repo itself (if that's even possible). I'm sure that this setup > would > > >> blow > > >> > up the minds of all newcomers. > > >> > - Similarly, it's mandatory to have safeguards against code from > > >> connector > > >> > A poisoning connector B, common, or main. I had some similar setup > in > > >> the > > >> > past and code from two "distinct" branch types constantly swept > over. > > >> > - We could also say that we simply release <common> independently > and > > >> just > > >> > have a maven (SNAPSHOT) dependency on it. But that would create a > weird > > >> > flow if you need to change in common where you need to constantly > switch > > >> > branches back and forth. > > >> > - In general, Frankensteinian's approach is very switch intensive. > If > > >> you > > >> > maintain 3 connectors and need to fix 1 build stability each at the > same > > >> > time (quite common nowadays for some reason) and you have 2 review > > >> rounds, > > >> > you need to switch branches 9 times ignoring changes to common. > > >> > > > >> > Additionally, we still have the rather user/dev unfriendly main > that is > > >> > mostly empty. I'm also not sure we can generate an overview > README.md to > > >> > make it more friendly here because in theory every connector branch > > >> should > > >> > be based on main and we would get merge conflicts. > > >> > > > >> > I'd like to propose once again to go with individual repositories. > > >> > - The only downside that we discussed so far is that we have more > > >> initial > > >> > setup to do. Since we organically grow the number of > > >> connector/repositories > > >> > that load is quite distributed. We can offer templates after > finding a > > >> good > > >> > approach that can even be used by outside organizations. > > >> > - Regarding secrets, I think it's actually an advantage that the > Kafka > > >> > connector has no access to the AWS secrets. If there are secrets to > be > > >> > shared across connectors, we can and should use Azure's Variable > Groups > > >> (I > > >> > have used it in the past to share Nexus creds across repos). That > would > > >> > also make rotation easy. > > >> > - Working on different connectors would be rather easy as all > modern IDE > > >> > support multiple repo setups in the same project. You still need to > do > > >> > multiple releases in case you update common code (either accessed > > >> through > > >> > Nexus or git submodule) and you want to release your connector. > > >> > - There is no difference in respect to how many CI runs there in > both > > >> > approaches. > > >> > - Individual repositories also have the advantage of allowing > external > > >> > incubation. Let's assume someone builds connector A and hosts it in > > >> their > > >> > organization (very common setup). If they want to contribute the > code to > > >> > Flink, we could simply transfer the repository into ASF after > ensuring > > >> > Flink coding standards. Then we retain git history and Github > issues. > > >> > > > >> > Is there any point that I'm missing? > > >> > > > >> > On Fri, Nov 26, 2021 at 1:32 PM Chesnay Schepler < > ches...@apache.org> > > >> > wrote: > > >> > > > >> > > For sharing workflows we should be able to use composite actions. > We'd > > >> > > have the main definition files in the flink-connectors repo, that > we > > >> > > also need to tag/release, which other branches/repos can then > import. > > >> > > These are also versioned, so we don't have to worry about > accidentally > > >> > > breaking stuff. > > >> > > These could also be used to enforce certain standards / interfaces > > >> such > > >> > > that we can automate more things (e.g., integration into the Flink > > >> > > documentation). > > >> > > > > >> > > It is true that Option 2) and dedicated repositories share a lot > of > > >> > > properties. While I did say in an offline conversation that we in > that > > >> > > case might just as well use separate repositories, I'm not so sure > > >> > > anymore. One repo would make administration a bit easier, for > example > > >> > > secrets wouldn't have to be applied to each repo (we wouldn't want > > >> > > certain secrets to be set up organization-wide). > > >> > > I overall also like that one repo would present a single access > point; > > >> > > you can't "miss" a connector repo, and I would hope that having > it as > > >> > > one repo would nurture more collaboration between the connectors, > > >> which > > >> > > after all need to solve similar problems. > > >> > > > > >> > > It is a fair point that the branching model would be quite weird, > but > > >> I > > >> > > think that would subside pretty quickly. > > >> > > > > >> > > Personally I'd go with Option 2, and if that doesn't work out we > can > > >> > > still split the repo later on. (Which should then be a trivial > matter > > >> of > > >> > > copying all <connector>/* branches and renaming them). > > >> > > > > >> > > On 26/11/2021 12:47, Till Rohrmann wrote: > > >> > > > Hi Arvid, > > >> > > > > > >> > > > Thanks for updating this thread with the latest findings. The > > >> described > > >> > > > limitations for a single connector repo sound suboptimal to me. > > >> > > > > > >> > > > * Option 2. sounds as if we try to simulate multi connector > repos > > >> > inside > > >> > > of > > >> > > > a single repo. I also don't know how we would share code > between the > > >> > > > different branches (sharing infrastructure would probably be > easier > > >> > > > though). This seems to have the same limitations as dedicated > repos > > >> > with > > >> > > > the downside of having a not very intuitive branching model. > > >> > > > * Isn't option 1. kind of a degenerated version of option 2. > where > > >> we > > >> > > have > > >> > > > some unrelated code from other connectors in the individual > > >> connector > > >> > > > branches? > > >> > > > * Option 3. has the downside that someone creating a release > has to > > >> > > release > > >> > > > all connectors. This means that she either has to sync with the > > >> > different > > >> > > > connector maintainers or has to be able to release all > connectors on > > >> > her > > >> > > > own. We are already seeing in the Flink community that releases > > >> require > > >> > > > quite good communication/coordination between the different > people > > >> > > working > > >> > > > on different Flink components. Given our goals to make connector > > >> > releases > > >> > > > easier and more frequent, I think that coupling different > connector > > >> > > > releases might be counter-productive. > > >> > > > > > >> > > > To me it sounds not very practical to mainly use a mono > repository > > >> w/o > > >> > > > having some more advanced build infrastructure that, for > example, > > >> > allows > > >> > > to > > >> > > > have different git roots in different connector directories. > Maybe > > >> the > > >> > > mono > > >> > > > repo can be a catch all repository for connectors that want to > be > > >> > > released > > >> > > > in lock-step (Option 3.) with all other connectors the repo > > >> contains. > > >> > But > > >> > > > for connectors that get changed frequently, having a dedicated > > >> > repository > > >> > > > that allows independent releases sounds preferable to me. > > >> > > > > > >> > > > What utilities and infrastructure code do you intend to share? > Using > > >> > git > > >> > > > submodules can definitely be one option to share code. However, > it > > >> > might > > >> > > > also be ok to depend on flink-connector-common artifacts which > could > > >> > make > > >> > > > things easier. Where I am unsure is whether git submodules can > be > > >> used > > >> > to > > >> > > > share infrastructure code (e.g. the .github/workflows) because > you > > >> need > > >> > > > these files in the repo to trigger the CI infrastructure. > > >> > > > > > >> > > > Cheers, > > >> > > > Till > > >> > > > > > >> > > > On Thu, Nov 25, 2021 at 1:59 PM Arvid Heise <ar...@apache.org> > > >> wrote: > > >> > > > > > >> > > >> Hi Brian, > > >> > > >> > > >> > > >> Thank you for sharing. I think your approach is very valid and > is > > >> in > > >> > > line > > >> > > >> with what I had in mind. > > >> > > >> > > >> > > >> Basically Pravega community aligns the connector releases with > the > > >> > > Pravega > > >> > > >>> mainline release > > >> > > >>> > > >> > > >> This certainly would mean that there is little value in > coupling > > >> > > connector > > >> > > >> versions. So it's making a good case for having separate > connector > > >> > > repos. > > >> > > >> > > >> > > >> > > >> > > >>> and maintains the connector with the latest 3 Flink > versions(CI > > >> will > > >> > > >>> publish snapshots for all these 3 branches) > > >> > > >>> > > >> > > >> I'd like to give connector devs a simple way to express to > which > > >> Flink > > >> > > >> versions the current branch is compatible. From there we can > > >> generate > > >> > > the > > >> > > >> compatibility matrix automatically and optionally also create > > >> > different > > >> > > >> releases per supported Flink version. Not sure if the latter is > > >> indeed > > >> > > >> better than having just one artifact that happens to run with > > >> multiple > > >> > > >> Flink versions. I guess it depends on what dependencies we are > > >> > > exposing. If > > >> > > >> the connector uses flink-connector-base, then we probably need > > >> > separate > > >> > > >> artifacts with poms anyways. > > >> > > >> > > >> > > >> Best, > > >> > > >> > > >> > > >> Arvid > > >> > > >> > > >> > > >> On Fri, Nov 19, 2021 at 10:55 AM Zhou, Brian <b.z...@dell.com> > > >> wrote: > > >> > > >> > > >> > > >>> Hi Arvid, > > >> > > >>> > > >> > > >>> For branching model, the Pravega Flink connector has some > > >> experience > > >> > > what > > >> > > >>> I would like to share. Here[1][2] is the compatibility matrix > and > > >> > wiki > > >> > > >>> explaining the branching model and releases. Basically Pravega > > >> > > community > > >> > > >>> aligns the connector releases with the Pravega mainline > release, > > >> and > > >> > > >>> maintains the connector with the latest 3 Flink versions(CI > will > > >> > > publish > > >> > > >>> snapshots for all these 3 branches). > > >> > > >>> For example, recently we have 0.10.1 release[3], and in maven > > >> central > > >> > > we > > >> > > >>> need to upload three artifacts(For Flink 1.13, 1.12, 1.11) for > > >> 0.10.1 > > >> > > >>> version[4]. > > >> > > >>> > > >> > > >>> There are some alternatives. Another solution that we once > > >> discussed > > >> > > but > > >> > > >>> finally got abandoned is to have a independent version just > like > > >> the > > >> > > >>> current CDC connector, and then give a big compatibility > matrix to > > >> > > users. > > >> > > >>> We think it would be too confusing when the connector > develops. On > > >> > the > > >> > > >>> contrary, we can also do the opposite way to align with Flink > > >> version > > >> > > and > > >> > > >>> maintain several branches for different system version. > > >> > > >>> > > >> > > >>> I would say this is only a fairly-OK solution because it is a > bit > > >> > > painful > > >> > > >>> for maintainers as cherry-picks are very common and releases > would > > >> > > >> require > > >> > > >>> much work. However, if neither systems do not have a nice > backward > > >> > > >>> compatibility, there seems to be no comfortable solution to > the > > >> their > > >> > > >>> connector. > > >> > > >>> > > >> > > >>> [1] > > >> https://github.com/pravega/flink-connectors#compatibility-matrix > > >> > > >>> [2] > > >> > > >>> > > >> > > >> > > >> > > > > >> > > > >> > https://github.com/pravega/flink-connectors/wiki/Versioning-strategy-for-Flink-connector > > >> > > >>> [3] > > >> https://github.com/pravega/flink-connectors/releases/tag/v0.10.1 > > >> > > >>> [4] > https://search.maven.org/search?q=pravega-connectors-flink > > >> > > >>> > > >> > > >>> Best Regards, > > >> > > >>> Brian > > >> > > >>> > > >> > > >>> > > >> > > >>> Internal Use - Confidential > > >> > > >>> > > >> > > >>> -----Original Message----- > > >> > > >>> From: Arvid Heise <ar...@apache.org> > > >> > > >>> Sent: Friday, November 19, 2021 4:12 PM > > >> > > >>> To: dev > > >> > > >>> Subject: Re: [DISCUSS] Creating an external connector > repository > > >> > > >>> > > >> > > >>> > > >> > > >>> [EXTERNAL EMAIL] > > >> > > >>> > > >> > > >>> Hi everyone, > > >> > > >>> > > >> > > >>> we are currently in the process of setting up the > flink-connectors > > >> > repo > > >> > > >>> [1] for new connectors but we hit a wall that we currently > cannot > > >> > take: > > >> > > >>> branching model. > > >> > > >>> To reiterate the original motivation of the external connector > > >> repo: > > >> > We > > >> > > >>> want to decouple the release cycle of a connector with Flink. > > >> > However, > > >> > > if > > >> > > >>> we want to support semantic versioning in the connectors with > the > > >> > > ability > > >> > > >>> to introduce breaking changes through major version bumps and > > >> support > > >> > > >>> bugfixes on old versions, then we need release branches > similar to > > >> > how > > >> > > >>> Flink core operates. > > >> > > >>> Consider two connectors, let's call them kafka and hbase. We > have > > >> > kafka > > >> > > >> in > > >> > > >>> version 1.0.X, 1.1.Y (small improvement), 2.0.Z (config > option) > > >> > change > > >> > > >> and > > >> > > >>> hbase only on 1.0.A. > > >> > > >>> > > >> > > >>> Now our current assumption was that we can work with a > mono-repo > > >> > under > > >> > > >> ASF > > >> > > >>> (flink-connectors). Then, for release-branches, we found 3 > > >> options: > > >> > > >>> 1. We would need to create some ugly mess with the cross > product > > >> of > > >> > > >>> connector and version: so you have kafka-release-1.0, > > >> > > kafka-release-1.1, > > >> > > >>> kafka-release-2.0, hbase-release-1.0. The main issue is not > the > > >> > amount > > >> > > of > > >> > > >>> branches (that's something that git can handle) but there the > > >> state > > >> > of > > >> > > >>> kafka is undefined in hbase-release-1.0. That's a call for > > >> desaster > > >> > and > > >> > > >>> makes releasing connectors very cumbersome (CI would only > execute > > >> and > > >> > > >>> publish hbase SNAPSHOTS on hbase-release-1.0). > > >> > > >>> 2. We could avoid the undefined state by having an empty > master > > >> and > > >> > > each > > >> > > >>> release branch really only holds the code of the connector. > But > > >> > that's > > >> > > >> also > > >> > > >>> not great: any user that looks at the repo and sees no > connector > > >> > would > > >> > > >>> assume that it's dead. > > >> > > >>> 3. We could have synced releases similar to the CDC connectors > > >> [2]. > > >> > > That > > >> > > >>> means that if any connector introduces a breaking change, all > > >> > > connectors > > >> > > >>> get a new major. I find that quite confusing to a user if > hbase > > >> gets > > >> > a > > >> > > >> new > > >> > > >>> release without any change because kafka introduced a breaking > > >> > change. > > >> > > >>> > > >> > > >>> To fully decouple release cycles and CI of connectors, we > could > > >> add > > >> > > >>> individual repositories under ASF (flink-connector-kafka, > > >> > > >>> flink-connector-hbase). Then we can apply the same branching > > >> model as > > >> > > >>> before. I quickly checked if there are precedences in the > apache > > >> > > >> community > > >> > > >>> for that approach and just by scanning alphabetically I found > > >> cordova > > >> > > >> with > > >> > > >>> 70 and couchdb with 77 apache repos respectively. So it > certainly > > >> > seems > > >> > > >>> like other projects approached our problem in that way and the > > >> apache > > >> > > >>> organization is okay with that. I currently expect max 20 > > >> additional > > >> > > >> repos > > >> > > >>> for connectors and in the future 10 max each for formats and > > >> > > filesystems > > >> > > >> if > > >> > > >>> we would also move them out at some point in time. So we > would be > > >> at > > >> > a > > >> > > >>> total of 50 repos. > > >> > > >>> > > >> > > >>> Note for all options, we need to provide a compability matrix > > >> that we > > >> > > aim > > >> > > >>> to autogenerate. > > >> > > >>> > > >> > > >>> Now for the potential downsides that we internally discussed: > > >> > > >>> - How can we ensure common infra structure code, utilties, and > > >> > quality? > > >> > > >>> I propose to add a flink-connector-common that contains all > these > > >> > > things > > >> > > >>> and is added as a git submodule/subtree to the repos. > > >> > > >>> - Do we implicitly discourage connector developers to maintain > > >> more > > >> > > than > > >> > > >>> one connector with a fragmented code base? > > >> > > >>> That is certainly a risk. However, I currently also see few > devs > > >> > > working > > >> > > >>> on more than one connector. However, it may actually help > keeping > > >> the > > >> > > >> devs > > >> > > >>> that maintain a specific connector on the hook. We could use > > >> github > > >> > > >> issues > > >> > > >>> to track bugs and feature requests and a dev can focus his > limited > > >> > time > > >> > > >> on > > >> > > >>> getting that one connector right. > > >> > > >>> > > >> > > >>> So WDYT? Compared to some intermediate suggestions with split > > >> repos, > > >> > > the > > >> > > >>> big difference is that everything remains under Apache > umbrella > > >> and > > >> > the > > >> > > >>> Flink community. > > >> > > >>> > > >> > > >>> [1] > > >> > > >>> > > >> > > >> > > >> > > > > >> > > > >> > https://urldefense.com/v3/__https://github.com/apache/flink-connectors__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzxxweh4$ > > >> > > >>> [github[.]com] [2] > > >> > > >>> > > >> > > >> > > >> > > > > >> > > > >> > https://urldefense.com/v3/__https://github.com/ververica/flink-cdc-connectors/__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzgoPGA8$ > > >> > > >>> [github[.]com] > > >> > > >>> > > >> > > >>> On Fri, Nov 12, 2021 at 3:39 PM Arvid Heise <ar...@apache.org > > > > >> > wrote: > > >> > > >>> > > >> > > >>>> Hi everyone, > > >> > > >>>> > > >> > > >>>> I created the flink-connectors repo [1] to advance the > topic. We > > >> > would > > >> > > >>>> create a proof-of-concept in the next few weeks as a special > > >> branch > > >> > > >>>> that I'd then use for discussions. If the community agrees > with > > >> the > > >> > > >>>> approach, that special branch will become the master. If > not, we > > >> can > > >> > > >>>> reiterate over it or create competing POCs. > > >> > > >>>> > > >> > > >>>> If someone wants to try things out in parallel, just make > sure > > >> that > > >> > > >>>> you are not accidentally pushing POCs to the master. > > >> > > >>>> > > >> > > >>>> As a reminder: We will not move out any current connector > from > > >> Flink > > >> > > >>>> at this point in time, so everything in Flink will remain as > is > > >> and > > >> > be > > >> > > >>>> maintained there. > > >> > > >>>> > > >> > > >>>> Best, > > >> > > >>>> > > >> > > >>>> Arvid > > >> > > >>>> > > >> > > >>>> [1] > > >> > > >>>> > > >> > > > > >> > https://urldefense.com/v3/__https://github.com/apache/flink-connectors > > >> > > >>>> > > >> > > __;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzxxweh4 > > >> > > >>>> $ [github[.]com] > > >> > > >>>> > > >> > > >>>> On Fri, Oct 29, 2021 at 6:57 PM Till Rohrmann < > > >> trohrm...@apache.org > > >> > > > > >> > > >>>> wrote: > > >> > > >>>> > > >> > > >>>>> Hi everyone, > > >> > > >>>>> > > >> > > >>>>> From the discussion, it seems to me that we have different > > >> > opinions > > >> > > >>>>> whether to have an ASF umbrella repository or to host them > > >> outside > > >> > of > > >> > > >>>>> the ASF. It also seems that this is not really the problem > to > > >> > solve. > > >> > > >>>>> Since there are many good arguments for either approach, we > > >> could > > >> > > >>>>> simply start with an ASF umbrella repository and see how > people > > >> > adopt > > >> > > >>>>> it. If the individual connectors cannot move fast enough or > if > > >> > people > > >> > > >>>>> prefer to not buy into the more heavy-weight ASF processes, > then > > >> > they > > >> > > >>>>> can host the code also somewhere else. We simply need to > make > > >> sure > > >> > > >>>>> that these connectors are discoverable (e.g. via > > >> flink-packages). > > >> > > >>>>> > > >> > > >>>>> The more important problem seems to be to provide common > tooling > > >> > > >>>>> (testing, infrastructure, documentation) that can easily be > > >> reused. > > >> > > >>>>> Similarly, it has become clear that the Flink community > needs to > > >> > > >>>>> improve on providing stable APIs. I think it is not > realistic to > > >> > > >>>>> first complete these tasks before starting to move > connectors to > > >> > > >>>>> dedicated repositories. As Stephan said, creating a > connector > > >> > > >>>>> repository will force us to pay more attention to API > stability > > >> and > > >> > > >>>>> also to think about which testing tools are required. > Hence, I > > >> > > >>>>> believe that starting to add connectors to a different > > >> repository > > >> > > >>>>> than apache/flink will help improve our connector tooling > > >> > (declaring > > >> > > >>>>> testing classes as public, creating a common test utility > repo, > > >> > > >>>>> creating a repo > > >> > > >>>>> template) and vice versa. Hence, I like Arvid's proposed > > >> process as > > >> > > >>>>> it will start kicking things off w/o letting this effort > fizzle > > >> > out. > > >> > > >>>>> > > >> > > >>>>> Cheers, > > >> > > >>>>> Till > > >> > > >>>>> > > >> > > >>>>> On Thu, Oct 28, 2021 at 11:44 AM Stephan Ewen < > se...@apache.org > > >> > > > >> > > >> wrote: > > >> > > >>>>>> Thank you all, for the nice discussion! > > >> > > >>>>>> > > >> > > >>>>>> From my point of view, I very much like the idea of > putting > > >> > > >>>>>> connectors > > >> > > >>>>> in a > > >> > > >>>>>> separate repository. But I would argue it should be part of > > >> Apache > > >> > > >>>>> Flink, > > >> > > >>>>>> similar to flink-statefun, flink-ml, etc. > > >> > > >>>>>> > > >> > > >>>>>> I share many of the reasons for that: > > >> > > >>>>>> - As argued many times, reduces complexity of the Flink > > >> repo, > > >> > > >>>>> increases > > >> > > >>>>>> response times of CI, etc. > > >> > > >>>>>> - Much lower barrier of contribution, because an > unstable > > >> > > >>>>>> connector > > >> > > >>>>> would > > >> > > >>>>>> not de-stabilize the whole build. Of course, we would need > to > > >> make > > >> > > >>>>>> sure > > >> > > >>>>> we > > >> > > >>>>>> set this up the right way, with connectors having > individual CI > > >> > > >>>>>> runs, > > >> > > >>>>> build > > >> > > >>>>>> status, etc. But it certainly seems possible. > > >> > > >>>>>> > > >> > > >>>>>> > > >> > > >>>>>> I would argue some points a bit different than some cases > made > > >> > > >> before: > > >> > > >>>>>> (a) I believe the separation would increase connector > > >> stability. > > >> > > >>>>> Because it > > >> > > >>>>>> really forces us to work with the connectors against the > APIs > > >> like > > >> > > >>>>>> any external developer. A mono repo is somehow the wrong > thing > > >> if > > >> > > >>>>>> you in practice want to actually guarantee stable internal > > >> APIs at > > >> > > >>> some layer. > > >> > > >>>>>> Because the mono repo makes it easy to just change > something on > > >> > > >>>>>> both > > >> > > >>>>> sides > > >> > > >>>>>> of the API (provider and consumer) seamlessly. > > >> > > >>>>>> > > >> > > >>>>>> Major refactorings in Flink need to keep all connector API > > >> > > >>>>>> contracts intact, or we need to have a new version of the > > >> > connector > > >> > > >>> API. > > >> > > >>>>>> (b) We may even be able to go towards more lightweight and > > >> > > >>>>>> automated releases over time, even if we stay in Apache > Flink > > >> with > > >> > > >>> that repo. > > >> > > >>>>>> This isn't yet fully aligned with the Apache release > policies, > > >> > yet, > > >> > > >>>>>> but there are board discussions about whether there can be > > >> > > >>>>>> bot-triggered releases (by dependabot) and how that could > fit > > >> into > > >> > > >>> the Apache process. > > >> > > >>>>>> This doesn't seem to be quite there just yet, but seeing > that > > >> > those > > >> > > >>>>> start > > >> > > >>>>>> is a good sign, and there is a good chance we can do some > > >> things > > >> > > >>> there. > > >> > > >>>>>> I am not sure whether we should let bots trigger releases, > > >> because > > >> > > >>>>>> a > > >> > > >>>>> final > > >> > > >>>>>> human look at things isn't a bad thing, especially given > the > > >> > > >>>>>> popularity > > >> > > >>>>> of > > >> > > >>>>>> software supply chain attacks recently. > > >> > > >>>>>> > > >> > > >>>>>> > > >> > > >>>>>> I do share Chesnay's concerns about complexity in tooling, > > >> though. > > >> > > >>>>>> Both release tooling and test tooling. They are not > > >> incompatible > > >> > > >>>>>> with that approach, but they are a task we need to tackle > > >> during > > >> > > >>>>>> this change which will add additional work. > > >> > > >>>>>> > > >> > > >>>>>> > > >> > > >>>>>> > > >> > > >>>>>> On Tue, Oct 26, 2021 at 10:31 AM Arvid Heise < > ar...@apache.org > > >> > > > >> > > >>> wrote: > > >> > > >>>>>>> Hi folks, > > >> > > >>>>>>> > > >> > > >>>>>>> I think some questions came up and I'd like to address the > > >> > > >>>>>>> question of > > >> > > >>>>>> the > > >> > > >>>>>>> timing. > > >> > > >>>>>>> > > >> > > >>>>>>> Could you clarify what release cadence you're thinking of? > > >> > > >>>>>>> There's > > >> > > >>>>> quite > > >> > > >>>>>>>> a big range that fits "more frequent than Flink" > (per-commit, > > >> > > >>>>>>>> daily, weekly, bi-weekly, monthly, even bi-monthly). > > >> > > >>>>>>> The short answer is: as often as needed: > > >> > > >>>>>>> - If there is a CVE in a dependency and we need to bump > it - > > >> > > >>>>>>> release immediately. > > >> > > >>>>>>> - If there is a new feature merged, release soonish. We > may > > >> > > >>>>>>> collect a > > >> > > >>>>> few > > >> > > >>>>>>> successive features before a release. > > >> > > >>>>>>> - If there is a bugfix, release immediately or soonish > > >> depending > > >> > > >>>>>>> on > > >> > > >>>>> the > > >> > > >>>>>>> severity and if there are workarounds available. > > >> > > >>>>>>> > > >> > > >>>>>>> We should not limit ourselves; the whole idea of > independent > > >> > > >>>>>>> releases > > >> > > >>>>> is > > >> > > >>>>>>> exactly that you release as needed. There is no release > > >> planning > > >> > > >>>>>>> or anything needed, you just go with a release as if it > was an > > >> > > >>>>>>> external artifact. > > >> > > >>>>>>> > > >> > > >>>>>>> (1) is the connector API already stable? > > >> > > >>>>>>>> From another discussion thread [1], connector API is far > > >> from > > >> > > >>>>> stable. > > >> > > >>>>>>>> Currently, it's hard to build connectors against multiple > > >> Flink > > >> > > >>>>>> versions. > > >> > > >>>>>>>> There are breaking API changes both in 1.12 -> 1.13 and > 1.13 > > >> -> > > >> > > >>>>>>>> 1.14 > > >> > > >>>>>> and > > >> > > >>>>>>>> maybe also in the future versions, because Table > related > > >> APIs > > >> > > >>>>>>>> are > > >> > > >>>>>> still > > >> > > >>>>>>>> @PublicEvolving and new Sink API is still @Experimental. > > >> > > >>>>>>>> > > >> > > >>>>>>> The question is: what is stable in an evolving system? We > > >> > > >>>>>>> recently discovered that the old SourceFunction needed to > be > > >> > > >>>>>>> refined such that cancellation works correctly [1]. So > that > > >> > > >>>>>>> interface is in Flink since > > >> > > >>>>> 7 > > >> > > >>>>>>> years, heavily used also outside, and we still had to > change > > >> the > > >> > > >>>>> contract > > >> > > >>>>>>> in a way that I'd expect any implementer to recheck their > > >> > > >>>>> implementation. > > >> > > >>>>>>> It might not be necessary to change anything and you can > > >> probably > > >> > > >>>>> change > > >> > > >>>>>>> the the code for all Flink versions but still, the > interface > > >> was > > >> > > >>>>>>> not > > >> > > >>>>>> stable > > >> > > >>>>>>> in the closest sense. > > >> > > >>>>>>> > > >> > > >>>>>>> If we focus just on API changes on the unified interfaces, > > >> then > > >> > > >>>>>>> we > > >> > > >>>>> expect > > >> > > >>>>>>> one more change to Sink API to support compaction. For > Table > > >> API, > > >> > > >>>>> there > > >> > > >>>>>>> will most likely also be some changes in 1.15. So we could > > >> wait > > >> > > >>>>>>> for > > >> > > >>>>> 1.15. > > >> > > >>>>>>> But I'm questioning if that's really necessary because we > will > > >> > > >>>>>>> add > > >> > > >>>>> more > > >> > > >>>>>>> functionality beyond 1.15 without breaking API. For > example, > > >> we > > >> > > >>>>>>> may > > >> > > >>>>> add > > >> > > >>>>>>> more unified connector metrics. If you want to use it in > your > > >> > > >>>>> connector, > > >> > > >>>>>>> you have to support multiple Flink versions anyhow. So > rather > > >> > > >>>>>>> then > > >> > > >>>>>> focusing > > >> > > >>>>>>> the discussion on "when is stuff stable", I'd rather > focus on > > >> > > >>>>>>> "how > > >> > > >>>>> can we > > >> > > >>>>>>> support building connectors against multiple Flink > versions" > > >> and > > >> > > >>>>>>> make > > >> > > >>>>> it > > >> > > >>>>>> as > > >> > > >>>>>>> painless as possible. > > >> > > >>>>>>> > > >> > > >>>>>>> Chesnay pointed out to use different branches for > different > > >> Flink > > >> > > >>>>>> versions > > >> > > >>>>>>> which sounds like a good suggestion. With a mono-repo, we > > >> can't > > >> > > >>>>>>> use branches differently anyways (there is no way to have > > >> release > > >> > > >>>>>>> branches > > >> > > >>>>>> per > > >> > > >>>>>>> connector without chaos). In these branches, we could > provide > > >> > > >>>>>>> shims to simulate future features in older Flink versions > such > > >> > > >>>>>>> that code-wise, > > >> > > >>>>> the > > >> > > >>>>>>> source code of a specific connector may not diverge > (much). > > >> For > > >> > > >>>>> example, > > >> > > >>>>>> to > > >> > > >>>>>>> register unified connector metrics, we could simulate the > > >> current > > >> > > >>>>>> approach > > >> > > >>>>>>> also in some utility package of the mono-repo. > > >> > > >>>>>>> > > >> > > >>>>>>> I see the stable core Flink API as a prerequisite for > > >> modularity. > > >> > > >>>>>>> And > > >> > > >>>>>>>> for connectors it is not just the source and sink API > (source > > >> > > >>>>>>>> being stable as of 1.14), but everything that is > required to > > >> > > >>>>>>>> build and maintain a connector downstream, such as the > test > > >> > > >>>>>>>> utilities and infrastructure. > > >> > > >>>>>>>> > > >> > > >>>>>>> That is a very fair point. I'm actually surprised to see > that > > >> > > >>>>>>> MiniClusterWithClientResource is not public. I see it > being > > >> used > > >> > > >>>>>>> in > > >> > > >>>>> all > > >> > > >>>>>>> connectors, especially outside of Flink. I fear that as > long > > >> as > > >> > > >>>>>>> we do > > >> > > >>>>> not > > >> > > >>>>>>> have connectors outside, we will not properly annotate and > > >> > > >>>>>>> maintain > > >> > > >>>>> these > > >> > > >>>>>>> utilties in a classic hen-and-egg-problem. I will outline > an > > >> idea > > >> > > >>>>>>> at > > >> > > >>>>> the > > >> > > >>>>>>> end. > > >> > > >>>>>>> > > >> > > >>>>>>>> the connectors need to be adopted and require at least > one > > >> > > >>>>>>>> release > > >> > > >>>>> per > > >> > > >>>>>>>> Flink minor release. > > >> > > >>>>>>>> However, this will make the releases of connectors > slower, > > >> e.g. > > >> > > >>>>>> maintain > > >> > > >>>>>>>> features for multiple branches and release multiple > branches. > > >> > > >>>>>>>> I think the main purpose of having an external connector > > >> > > >>>>>>>> repository > > >> > > >>>>> is > > >> > > >>>>>> in > > >> > > >>>>>>>> order to have "faster releases of connectors"? > > >> > > >>>>>>>> > > >> > > >>>>>>>> Imagine a project with a complex set of dependencies. > Let's > > >> say > > >> > > >>>>> Flink > > >> > > >>>>>>>> version A plus Flink reliant dependencies released by > other > > >> > > >>>>>>>> projects (Flink-external connectors, Beam, Iceberg, Hudi, > > >> ..). > > >> > > >>>>>>>> We don't want > > >> > > >>>>> a > > >> > > >>>>>>>> situation where we bump the core Flink version to B and > > >> things > > >> > > >>>>>>>> fall apart (interface changes, utilities that were > useful but > > >> > > >>>>>>>> not public, transitive dependencies etc.). > > >> > > >>>>>>>> > > >> > > >>>>>>> Yes, that's why I wanted to automate the processes more > which > > >> is > > >> > > >>>>>>> not > > >> > > >>>>> that > > >> > > >>>>>>> easy under ASF. Maybe we automate the source provision > across > > >> > > >>>>> supported > > >> > > >>>>>>> versions and have 1 vote thread for all versions of a > > >> connector? > > >> > > >>>>>>> > > >> > > >>>>>>> From the perspective of CDC connector maintainers, the > > >> biggest > > >> > > >>>>> advantage > > >> > > >>>>>> of > > >> > > >>>>>>>> maintaining it outside of the Flink project is that: > > >> > > >>>>>>>> 1) we can have a more flexible and faster release cycle > > >> > > >>>>>>>> 2) we can be more liberal with committership for > connector > > >> > > >>>>> maintainers > > >> > > >>>>>>>> which can also attract more committers to help the > release. > > >> > > >>>>>>>> > > >> > > >>>>>>>> Personally, I think maintaining one connector repository > > >> under > > >> > > >>>>>>>> the > > >> > > >>>>> ASF > > >> > > >>>>>>> may > > >> > > >>>>>>>> not have the above benefits. > > >> > > >>>>>>>> > > >> > > >>>>>>> Yes, I also feel that ASF is too restrictive for our > needs. > > >> But > > >> > > >>>>>>> it > > >> > > >>>>> feels > > >> > > >>>>>>> like there are too many that see it differently and I > think we > > >> > > >>>>>>> need > > >> > > >>>>>>> > > >> > > >>>>>>> (2) Flink testability without connectors. > > >> > > >>>>>>>> This is a very good question. How can we guarantee the > new > > >> > > >>>>>>>> Source > > >> > > >>>>> and > > >> > > >>>>>>> Sink > > >> > > >>>>>>>> API are stable with only test implementation? > > >> > > >>>>>>>> > > >> > > >>>>>>> We can't and shouldn't. Since the connector repo is > managed by > > >> > > >>>>>>> Flink, > > >> > > >>>>> a > > >> > > >>>>>>> Flink release manager needs to check if the Flink > connectors > > >> are > > >> > > >>>>> actually > > >> > > >>>>>>> working prior to creating an RC. That's similar to how > > >> > > >>>>>>> flink-shaded > > >> > > >>>>> and > > >> > > >>>>>>> flink core are related. > > >> > > >>>>>>> > > >> > > >>>>>>> > > >> > > >>>>>>> So here is one idea that I had to get things rolling. We > are > > >> > > >>>>>>> going to address the external repo iteratively without > > >> > > >>>>>>> compromising what we > > >> > > >>>>>> already > > >> > > >>>>>>> have: > > >> > > >>>>>>> 1.Phase, add new contributions to external repo. We use > that > > >> time > > >> > > >>>>>>> to > > >> > > >>>>>> setup > > >> > > >>>>>>> infra accordingly and optimize release processes. We will > > >> > > >>>>>>> identify > > >> > > >>>>> test > > >> > > >>>>>>> utilities that are not yet public/stable and fix that. > > >> > > >>>>>>> 2.Phase, add ports to the new unified interfaces of > existing > > >> > > >>>>> connectors. > > >> > > >>>>>>> That requires a previous Flink release to make utilities > > >> stable. > > >> > > >>>>>>> Keep > > >> > > >>>>> old > > >> > > >>>>>>> interfaces in flink-core. > > >> > > >>>>>>> 3.Phase, remove old interfaces in flink-core of some > > >> connectors > > >> > > >>>>>>> (tbd > > >> > > >>>>> at a > > >> > > >>>>>>> later point). > > >> > > >>>>>>> 4.Phase, optionally move all remaining connectors (tbd at > a > > >> later > > >> > > >>>>> point). > > >> > > >>>>>>> I'd envision having ~3 months between the starting the > > >> different > > >> > > >>>>> phases. > > >> > > >>>>>>> WDYT? > > >> > > >>>>>>> > > >> > > >>>>>>> > > >> > > >>>>>>> [1] > > >> > > >>>>>>> > > >> > https://urldefense.com/v3/__https://issues.apache.org/jira/browse > > >> > > >>>>>>> > > >> /FLINK-23527__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgd > > >> > > >>>>>>> ke_-XjpYgX2sIvAP4$ [issues[.]apache[.]org] > > >> > > >>>>>>> > > >> > > >>>>>>> On Thu, Oct 21, 2021 at 7:12 AM Kyle Bendickson < > > >> k...@tabular.io > > >> > > > > >> > > >>>>> wrote: > > >> > > >>>>>>>> Hi all, > > >> > > >>>>>>>> > > >> > > >>>>>>>> My name is Kyle and I’m an open source developer > primarily > > >> > > >>>>>>>> focused > > >> > > >>>>> on > > >> > > >>>>>>>> Apache Iceberg. > > >> > > >>>>>>>> > > >> > > >>>>>>>> I’m happy to help clarify or elaborate on any aspect of > our > > >> > > >>>>> experience > > >> > > >>>>>>>> working on a relatively decoupled connector that is > > >> downstream > > >> > > >>>>>>>> and > > >> > > >>>>>> pretty > > >> > > >>>>>>>> popular. > > >> > > >>>>>>>> > > >> > > >>>>>>>> I’d also love to be able to contribute or assist in any > way I > > >> > > >> can. > > >> > > >>>>>>>> I don’t mean to thread jack, but are there any meetings > or > > >> > > >>>>>>>> community > > >> > > >>>>>> sync > > >> > > >>>>>>>> ups, specifically around the connector APIs, that I might > > >> join > > >> > > >>>>>>>> / be > > >> > > >>>>>>> invited > > >> > > >>>>>>>> to? > > >> > > >>>>>>>> > > >> > > >>>>>>>> I did want to add that even though I’ve experienced some > of > > >> the > > >> > > >>>>>>>> pain > > >> > > >>>>>>> points > > >> > > >>>>>>>> of integrating with an evolving system / API (catalog > support > > >> > > >>>>>>>> is > > >> > > >>>>>>> generally > > >> > > >>>>>>>> speaking pretty new everywhere really in this space), I > also > > >> > > >>>>>>>> agree personally that you shouldn’t slow down development > > >> > > >>>>>>>> velocity too > > >> > > >>>>> much > > >> > > >>>>>> for > > >> > > >>>>>>>> the sake of external connector. Getting to a performant > and > > >> > > >>>>>>>> stable > > >> > > >>>>>> place > > >> > > >>>>>>>> should be the primary goal, and slowing that down to > support > > >> > > >>>>> stragglers > > >> > > >>>>>>>> will (in my personal opinion) always be a losing game. > Some > > >> > > >>>>>>>> folks > > >> > > >>>>> will > > >> > > >>>>>>>> simply stay behind on versions regardless until they > have to > > >> > > >>>>> upgrade. > > >> > > >>>>>>>> I am working on ensuring that the Iceberg community stays > > >> > > >>>>>>>> within 1-2 versions of Flink, so that we can help provide > > >> more > > >> > > >>>>>>>> feedback or > > >> > > >>>>>>> contribute > > >> > > >>>>>>>> things that might make our ability to support multiple > Flink > > >> > > >>>>> runtimes / > > >> > > >>>>>>>> versions with one project / codebase and minimal to no > > >> > > >>>>>>>> reflection > > >> > > >>>>> (our > > >> > > >>>>>>>> desired goal). > > >> > > >>>>>>>> > > >> > > >>>>>>>> If there’s anything I can do or any way I can be of > > >> assistance, > > >> > > >>>>> please > > >> > > >>>>>>>> don’t hesitate to reach out. Or find me on ASF slack 😀 > > >> > > >>>>>>>> > > >> > > >>>>>>>> I greatly appreciate your general concern for the needs > of > > >> > > >>>>> downstream > > >> > > >>>>>>>> connector integrators! > > >> > > >>>>>>>> > > >> > > >>>>>>>> Cheers > > >> > > >>>>>>>> Kyle Bendickson (GitHub: kbendick) Open Source Developer > kyle > > >> > > >>>>>>>> [at] tabular [dot] io > > >> > > >>>>>>>> > > >> > > >>>>>>>> On Wed, Oct 20, 2021 at 11:35 AM Thomas Weise < > > >> t...@apache.org> > > >> > > >>>>> wrote: > > >> > > >>>>>>>>> Hi, > > >> > > >>>>>>>>> > > >> > > >>>>>>>>> I see the stable core Flink API as a prerequisite for > > >> > > >>> modularity. > > >> > > >>>>> And > > >> > > >>>>>>>>> for connectors it is not just the source and sink API > > >> (source > > >> > > >>>>> being > > >> > > >>>>>>>>> stable as of 1.14), but everything that is required to > build > > >> > > >>>>>>>>> and maintain a connector downstream, such as the test > > >> > > >>>>>>>>> utilities and infrastructure. > > >> > > >>>>>>>>> > > >> > > >>>>>>>>> Without the stable surface of core Flink, changes will > leak > > >> > > >>>>>>>>> into downstream dependencies and force lock step > updates. > > >> > > >>>>>>>>> Refactoring across N repos is more painful than a single > > >> > > >>>>>>>>> repo. Those with experience developing downstream of > Flink > > >> > > >>>>>>>>> will know the pain, and > > >> > > >>>>>> that > > >> > > >>>>>>>>> isn't limited to connectors. I don't remember a Flink > "minor > > >> > > >>>>> version" > > >> > > >>>>>>>>> update that was just a dependency version change and > did not > > >> > > >>>>>>>>> force other downstream changes. > > >> > > >>>>>>>>> > > >> > > >>>>>>>>> Imagine a project with a complex set of dependencies. > Let's > > >> > > >>>>>>>>> say > > >> > > >>>>> Flink > > >> > > >>>>>>>>> version A plus Flink reliant dependencies released by > other > > >> > > >>>>> projects > > >> > > >>>>>>>>> (Flink-external connectors, Beam, Iceberg, Hudi, ..). We > > >> > > >>>>>>>>> don't > > >> > > >>>>> want a > > >> > > >>>>>>>>> situation where we bump the core Flink version to B and > > >> > > >>>>>>>>> things > > >> > > >>>>> fall > > >> > > >>>>>>>>> apart (interface changes, utilities that were useful > but not > > >> > > >>>>> public, > > >> > > >>>>>>>>> transitive dependencies etc.). > > >> > > >>>>>>>>> > > >> > > >>>>>>>>> The discussion here also highlights the benefits of > keeping > > >> > > >>>>> certain > > >> > > >>>>>>>>> connectors outside Flink. Whether that is due to > difference > > >> > > >>>>>>>>> in developer community, maturity of the connectors, > their > > >> > > >>>>>>>>> specialized/limited usage etc. I would like to see that > as a > > >> > > >>>>>>>>> sign > > >> > > >>>>> of > > >> > > >>>>>> a > > >> > > >>>>>>>>> growing ecosystem and most of the ideas that Arvid has > put > > >> > > >>>>>>>>> forward would benefit further growth of the connector > > >> > > >> ecosystem. > > >> > > >>>>>>>>> As for keeping connectors within Apache Flink: I prefer > that > > >> > > >>>>>>>>> as > > >> > > >>>>> the > > >> > > >>>>>>>>> path forward for "essential" connectors like FileSource, > > >> > > >>>>> KafkaSource, > > >> > > >>>>>>>>> ... And we can still achieve a more flexible and faster > > >> > > >>>>>>>>> release > > >> > > >>>>>> cycle. > > >> > > >>>>>>>>> Thanks, > > >> > > >>>>>>>>> Thomas > > >> > > >>>>>>>>> > > >> > > >>>>>>>>> > > >> > > >>>>>>>>> > > >> > > >>>>>>>>> > > >> > > >>>>>>>>> > > >> > > >>>>>>>>> On Wed, Oct 20, 2021 at 3:32 AM Jark Wu < > imj...@gmail.com> > > >> > > >>> wrote: > > >> > > >>>>>>>>>> Hi Konstantin, > > >> > > >>>>>>>>>> > > >> > > >>>>>>>>>>> the connectors need to be adopted and require at least > > >> > > >>>>>>>>>>> one > > >> > > >>>>>> release > > >> > > >>>>>>>> per > > >> > > >>>>>>>>>> Flink minor release. > > >> > > >>>>>>>>>> However, this will make the releases of connectors > slower, > > >> > > >>> e.g. > > >> > > >>>>>>>> maintain > > >> > > >>>>>>>>>> features for multiple branches and release multiple > > >> > > >> branches. > > >> > > >>>>>>>>>> I think the main purpose of having an external > connector > > >> > > >>>>> repository > > >> > > >>>>>>> is > > >> > > >>>>>>>> in > > >> > > >>>>>>>>>> order to have "faster releases of connectors"? > > >> > > >>>>>>>>>> > > >> > > >>>>>>>>>> > > >> > > >>>>>>>>>> From the perspective of CDC connector maintainers, the > > >> > > >>>>>>>>>> biggest > > >> > > >>>>>>>> advantage > > >> > > >>>>>>>>> of > > >> > > >>>>>>>>>> maintaining it outside of the Flink project is that: > > >> > > >>>>>>>>>> 1) we can have a more flexible and faster release cycle > > >> > > >>>>>>>>>> 2) we can be more liberal with committership for > connector > > >> > > >>>>>>> maintainers > > >> > > >>>>>>>>>> which can also attract more committers to help the > release. > > >> > > >>>>>>>>>> > > >> > > >>>>>>>>>> Personally, I think maintaining one connector > repository > > >> > > >>>>>>>>>> under > > >> > > >>>>> the > > >> > > >>>>>>> ASF > > >> > > >>>>>>>>> may > > >> > > >>>>>>>>>> not have the above benefits. > > >> > > >>>>>>>>>> > > >> > > >>>>>>>>>> Best, > > >> > > >>>>>>>>>> Jark > > >> > > >>>>>>>>>> > > >> > > >>>>>>>>>> On Wed, 20 Oct 2021 at 15:14, Konstantin Knauf < > > >> > > >>>>> kna...@apache.org> > > >> > > >>>>>>>>> wrote: > > >> > > >>>>>>>>>>> Hi everyone, > > >> > > >>>>>>>>>>> > > >> > > >>>>>>>>>>> regarding the stability of the APIs. I think everyone > > >> > > >>>>>>>>>>> agrees > > >> > > >>>>> that > > >> > > >>>>>>>>>>> connector APIs which are stable across minor versions > > >> > > >>>>>> (1.13->1.14) > > >> > > >>>>>>>> are > > >> > > >>>>>>>>> the > > >> > > >>>>>>>>>>> mid-term goal. But: > > >> > > >>>>>>>>>>> > > >> > > >>>>>>>>>>> a) These APIs are still quite young, and we shouldn't > > >> > > >>>>>>>>>>> make > > >> > > >>>>> them > > >> > > >>>>>>>> @Public > > >> > > >>>>>>>>>>> prematurely either. > > >> > > >>>>>>>>>>> > > >> > > >>>>>>>>>>> b) Isn't this *mostly* orthogonal to where the > connector > > >> > > >>>>>>>>>>> code > > >> > > >>>>>>> lives? > > >> > > >>>>>>>>> Yes, > > >> > > >>>>>>>>>>> as long as there are breaking changes, the connectors > > >> > > >>>>>>>>>>> need to > > >> > > >>>>> be > > >> > > >>>>>>>>> adopted > > >> > > >>>>>>>>>>> and require at least one release per Flink minor > release. > > >> > > >>>>>>>>>>> Documentation-wise this can be addressed via a > > >> > > >>>>>>>>>>> compatibility > > >> > > >>>>>> matrix > > >> > > >>>>>>>> for > > >> > > >>>>>>>>>>> each connector as Arvid suggested. IMO we shouldn't > block > > >> > > >>>>>>>>>>> this > > >> > > >>>>>>> effort > > >> > > >>>>>>>>> on > > >> > > >>>>>>>>>>> the stability of the APIs. > > >> > > >>>>>>>>>>> > > >> > > >>>>>>>>>>> Cheers, > > >> > > >>>>>>>>>>> > > >> > > >>>>>>>>>>> Konstantin > > >> > > >>>>>>>>>>> > > >> > > >>>>>>>>>>> > > >> > > >>>>>>>>>>> > > >> > > >>>>>>>>>>> On Wed, Oct 20, 2021 at 8:56 AM Jark Wu > > >> > > >>>>>>>>>>> <imj...@gmail.com> > > >> > > >>>>>> wrote: > > >> > > >>>>>>>>>>>> Hi, > > >> > > >>>>>>>>>>>> > > >> > > >>>>>>>>>>>> I think Thomas raised very good questions and would > like > > >> > > >>>>>>>>>>>> to > > >> > > >>>>> know > > >> > > >>>>>>>> your > > >> > > >>>>>>>>>>>> opinions if we want to move connectors out of flink > in > > >> > > >>>>>>>>>>>> this > > >> > > >>>>>>> version. > > >> > > >>>>>>>>>>>> (1) is the connector API already stable? > > >> > > >>>>>>>>>>>>> Separate releases would only make sense if the core > > >> > > >>>>>>>>>>>>> Flink > > >> > > >>>>>>> surface > > >> > > >>>>>>>> is > > >> > > >>>>>>>>>>>>> fairly stable though. As evident from Iceberg (and > > >> > > >>>>>>>>>>>>> also > > >> > > >>>>> Beam), > > >> > > >>>>>>>>> that's > > >> > > >>>>>>>>>>>>> not the case currently. We should probably focus on > > >> > > >>>>> addressing > > >> > > >>>>>>> the > > >> > > >>>>>>>>>>>>> stability first, before splitting code. A success > > >> > > >>>>>>>>>>>>> criteria > > >> > > >>>>>> could > > >> > > >>>>>>>> be > > >> > > >>>>>>>>>>>>> that we are able to build Iceberg and Beam against > > >> > > >>>>>>>>>>>>> multiple > > >> > > >>>>>>> Flink > > >> > > >>>>>>>>>>>>> versions w/o the need to change code. The goal would > > >> > > >>>>>>>>>>>>> be > > >> > > >>>>> that > > >> > > >>>>>> no > > >> > > >>>>>>>>>>>>> connector breaks when we make changes to Flink core. > > >> > > >>>>>>>>>>>>> Until > > >> > > >>>>>>> that's > > >> > > >>>>>>>>> the > > >> > > >>>>>>>>>>>>> case, code separation creates a setup where 1+1 or > N+1 > > >> > > >>>>>>>> repositories > > >> > > >>>>>>>>>>>>> need to move lock step. > > >> > > >>>>>>>>>>>> From another discussion thread [1], connector API > is far > > >> > > >>>>>>>>>>>> from > > >> > > >>>>>>>> stable. > > >> > > >>>>>>>>>>>> Currently, it's hard to build connectors against > > >> > > >>>>>>>>>>>> multiple > > >> > > >>>>> Flink > > >> > > >>>>>>>>> versions. > > >> > > >>>>>>>>>>>> There are breaking API changes both in 1.12 -> 1.13 > and > > >> > > >>>>>>>>>>>> 1.13 > > >> > > >>>>> -> > > >> > > >>>>>>> 1.14 > > >> > > >>>>>>>>> and > > >> > > >>>>>>>>>>>> maybe also in the future versions, because Table > > >> > > >>>>>>>>>>>> related > > >> > > >>>>> APIs > > >> > > >>>>>>> are > > >> > > >>>>>>>>> still > > >> > > >>>>>>>>>>>> @PublicEvolving and new Sink API is still > @Experimental. > > >> > > >>>>>>>>>>>> > > >> > > >>>>>>>>>>>> > > >> > > >>>>>>>>>>>> (2) Flink testability without connectors. > > >> > > >>>>>>>>>>>>> Flink w/o Kafka connector (and few others) isn't > > >> > > >>>>>>>>>>>>> viable. Testability of Flink was already brought up, > > >> > > >>>>>>>>>>>>> can we > > >> > > >>>>>>> really > > >> > > >>>>>>>>>>>>> certify a Flink core release without Kafka > connector? > > >> > > >>>>>>>>>>>>> Maybe > > >> > > >>>>>>> those > > >> > > >>>>>>>>>>>>> connectors that are used in Flink e2e tests to > > >> > > >>>>>>>>>>>>> validate > > >> > > >>>>>>>>> functionality > > >> > > >>>>>>>>>>>>> of core Flink should not be broken out? > > >> > > >>>>>>>>>>>> This is a very good question. How can we guarantee > the > > >> > > >>>>>>>>>>>> new > > >> > > >>>>>> Source > > >> > > >>>>>>>> and > > >> > > >>>>>>>>> Sink > > >> > > >>>>>>>>>>>> API are stable with only test implementation? > > >> > > >>>>>>>>>>>> > > >> > > >>>>>>>>>>>> > > >> > > >>>>>>>>>>>> Best, > > >> > > >>>>>>>>>>>> Jark > > >> > > >>>>>>>>>>>> > > >> > > >>>>>>>>>>>> > > >> > > >>>>>>>>>>>> > > >> > > >>>>>>>>>>>> > > >> > > >>>>>>>>>>>> > > >> > > >>>>>>>>>>>> On Tue, 19 Oct 2021 at 23:56, Chesnay Schepler < > > >> > > >>>>>>> ches...@apache.org> > > >> > > >>>>>>>>>>>> wrote: > > >> > > >>>>>>>>>>>> > > >> > > >>>>>>>>>>>>> Could you clarify what release cadence you're > thinking > > >> > > >>> of? > > >> > > >>>>>>> There's > > >> > > >>>>>>>>> quite > > >> > > >>>>>>>>>>>>> a big range that fits "more frequent than Flink" > > >> > > >>>>> (per-commit, > > >> > > >>>>>>>> daily, > > >> > > >>>>>>>>>>>>> weekly, bi-weekly, monthly, even bi-monthly). > > >> > > >>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>> On 19/10/2021 14:15, Martijn Visser wrote: > > >> > > >>>>>>>>>>>>>> Hi all, > > >> > > >>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>> I think it would be a huge benefit if we can > achieve > > >> > > >>>>>>>>>>>>>> more > > >> > > >>>>>>>> frequent > > >> > > >>>>>>>>>>>>> releases > > >> > > >>>>>>>>>>>>>> of connectors, which are not bound to the release > > >> > > >>>>>>>>>>>>>> cycle > > >> > > >>>>> of > > >> > > >>>>>>> Flink > > >> > > >>>>>>>>>>>> itself. > > >> > > >>>>>>>>>>>>> I > > >> > > >>>>>>>>>>>>>> agree that in order to get there, we need to have > > >> > > >>>>>>>>>>>>>> stable > > >> > > >>>>>>>>> interfaces > > >> > > >>>>>>>>>>>> which > > >> > > >>>>>>>>>>>>>> are trustworthy and reliable, so they can be safely > > >> > > >>>>>>>>>>>>>> used > > >> > > >>>>> by > > >> > > >>>>>>>> those > > >> > > >>>>>>>>>>>>>> connectors. I do think that work still needs to be > > >> > > >>>>>>>>>>>>>> done > > >> > > >>>>> on > > >> > > >>>>>>> those > > >> > > >>>>>>>>>>>>>> interfaces, but I am confident that we can get > there > > >> > > >>>>> from a > > >> > > >>>>>>>> Flink > > >> > > >>>>>>>>>>>>>> perspective. > > >> > > >>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>> I am worried that we would not be able to achieve > > >> > > >>>>>>>>>>>>>> those > > >> > > >>>>>>> frequent > > >> > > >>>>>>>>>>>> releases > > >> > > >>>>>>>>>>>>>> of connectors if we are putting these connectors > > >> > > >>>>>>>>>>>>>> under > > >> > > >>>>> the > > >> > > >>>>>>>> Apache > > >> > > >>>>>>>>>>>>> umbrella, > > >> > > >>>>>>>>>>>>>> because that means that for each connector release > > >> > > >>>>>>>>>>>>>> we > > >> > > >>>>> have > > >> > > >>>>>> to > > >> > > >>>>>>>>> follow > > >> > > >>>>>>>>>>>> the > > >> > > >>>>>>>>>>>>>> Apache release creation process. This requires a > lot > > >> > > >>>>>>>>>>>>>> of > > >> > > >>>>>> manual > > >> > > >>>>>>>>> steps > > >> > > >>>>>>>>>>>> and > > >> > > >>>>>>>>>>>>>> prohibits automation and I think it would be hard > to > > >> > > >>>>> scale > > >> > > >>>>>> out > > >> > > >>>>>>>>>>>> frequent > > >> > > >>>>>>>>>>>>>> releases of connectors. I'm curious how others > think > > >> > > >>>>>>>>>>>>>> this > > >> > > >>>>>>>>> challenge > > >> > > >>>>>>>>>>>> could > > >> > > >>>>>>>>>>>>>> be solved. > > >> > > >>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>> Best regards, > > >> > > >>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>> Martijn > > >> > > >>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>> On Mon, 18 Oct 2021 at 22:22, Thomas Weise < > > >> > > >>>>> t...@apache.org> > > >> > > >>>>>>>>> wrote: > > >> > > >>>>>>>>>>>>>>> Thanks for initiating this discussion. > > >> > > >>>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>>> There are definitely a few things that are not > > >> > > >>>>>>>>>>>>>>> optimal > > >> > > >>>>> with > > >> > > >>>>>>> our > > >> > > >>>>>>>>>>>>>>> current management of connectors. I would not > > >> > > >>>>> necessarily > > >> > > >>>>>>>>>>>> characterize > > >> > > >>>>>>>>>>>>>>> it as a "mess" though. As the points raised so far > > >> > > >>>>> show, it > > >> > > >>>>>>>> isn't > > >> > > >>>>>>>>>>>> easy > > >> > > >>>>>>>>>>>>>>> to find a solution that balances competing > > >> > > >>>>>>>>>>>>>>> requirements > > >> > > >>>>> and > > >> > > >>>>>>>>> leads to > > >> > > >>>>>>>>>>>> a > > >> > > >>>>>>>>>>>>>>> net improvement. > > >> > > >>>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>>> It would be great if we can find a setup that > > >> > > >>>>>>>>>>>>>>> allows for > > >> > > >>>>>>>>> connectors > > >> > > >>>>>>>>>>>> to > > >> > > >>>>>>>>>>>>>>> be released independently of core Flink and that > > >> > > >>>>>>>>>>>>>>> each > > >> > > >>>>>>> connector > > >> > > >>>>>>>>> can > > >> > > >>>>>>>>>>>> be > > >> > > >>>>>>>>>>>>>>> released separately. Flink already has separate > > >> > > >>>>>>>>>>>>>>> releases (flink-shaded), so that by itself isn't a > > >> > > >>> new thing. > > >> > > >>>>>>>>> Per-connector > > >> > > >>>>>>>>>>>>>>> releases would need to allow for more frequent > > >> > > >>>>>>>>>>>>>>> releases > > >> > > >>>>>>>> (without > > >> > > >>>>>>>>> the > > >> > > >>>>>>>>>>>>>>> baggage that a full Flink release comes with). > > >> > > >>>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>>> Separate releases would only make sense if the core > > >> > > >>>>> Flink > > >> > > >>>>>>>>> surface is > > >> > > >>>>>>>>>>>>>>> fairly stable though. As evident from Iceberg (and > > >> > > >>>>>>>>>>>>>>> also > > >> > > >>>>>>> Beam), > > >> > > >>>>>>>>> that's > > >> > > >>>>>>>>>>>>>>> not the case currently. We should probably focus > on > > >> > > >>>>>>> addressing > > >> > > >>>>>>>>> the > > >> > > >>>>>>>>>>>>>>> stability first, before splitting code. A success > > >> > > >>>>> criteria > > >> > > >>>>>>>> could > > >> > > >>>>>>>>> be > > >> > > >>>>>>>>>>>>>>> that we are able to build Iceberg and Beam against > > >> > > >>>>> multiple > > >> > > >>>>>>>> Flink > > >> > > >>>>>>>>>>>>>>> versions w/o the need to change code. The goal > > >> > > >>>>>>>>>>>>>>> would be > > >> > > >>>>>> that > > >> > > >>>>>>> no > > >> > > >>>>>>>>>>>>>>> connector breaks when we make changes to Flink > core. > > >> > > >>>>> Until > > >> > > >>>>>>>>> that's the > > >> > > >>>>>>>>>>>>>>> case, code separation creates a setup where 1+1 or > > >> > > >>>>>>>>>>>>>>> N+1 > > >> > > >>>>>>>>> repositories > > >> > > >>>>>>>>>>>>>>> need to move lock step. > > >> > > >>>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>>> Regarding some connectors being more important for > > >> > > >>>>>>>>>>>>>>> Flink > > >> > > >>>>>> than > > >> > > >>>>>>>>> others: > > >> > > >>>>>>>>>>>>>>> That's a fact. Flink w/o Kafka connector (and few > > > >> > > >>>>> others) > > >> > > >>>>>>> isn't > > >> > > >>>>>>>>>>>>>>> viable. Testability of Flink was already brought > > >> > > >>>>>>>>>>>>>>> up, > > >> > > >>>>> can we > > >> > > >>>>>>>>> really > > >> > > >>>>>>>>>>>>>>> certify a Flink core release without Kafka > > >> > > >> connector? > > >> > > >>>>> Maybe > > >> > > >>>>>>>> those > > >> > > >>>>>>>>>>>>>>> connectors that are used in Flink e2e tests to > > >> > > >>>>>>>>>>>>>>> validate > > >> > > >>>>>>>>> functionality > > >> > > >>>>>>>>>>>>>>> of core Flink should not be broken out? > > >> > > >>>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>>> Finally, I think that the connectors that move > into > > >> > > >>>>>> separate > > >> > > >>>>>>>>> repos > > >> > > >>>>>>>>>>>>>>> should remain part of the Apache Flink project. > > >> > > >>>>>>>>>>>>>>> Larger > > >> > > >>>>>>>>> organizations > > >> > > >>>>>>>>>>>>>>> tend to approve the use of and contribution to > open > > >> > > >>>>> source > > >> > > >>>>>> at > > >> > > >>>>>>>> the > > >> > > >>>>>>>>>>>>>>> project level. Sometimes it is everything ASF. > More > > >> > > >>>>> often > > >> > > >>>>>> it > > >> > > >>>>>>> is > > >> > > >>>>>>>>>>>>>>> "Apache Foo". It would be fatal to end up with a > > >> > > >>>>> patchwork > > >> > > >>>>>> of > > >> > > >>>>>>>>>>>> projects > > >> > > >>>>>>>>>>>>>>> with potentially different licenses and governance > > >> > > >>>>>>>>>>>>>>> to > > >> > > >>>>>> arrive > > >> > > >>>>>>>> at a > > >> > > >>>>>>>>>>>>>>> working Flink setup. This may mean we prioritize > > >> > > >>>>> usability > > >> > > >>>>>>> over > > >> > > >>>>>>>>>>>>>>> developer convenience, if that's in the best > > >> > > >>>>>>>>>>>>>>> interest of > > >> > > >>>>>>> Flink > > >> > > >>>>>>>>> as a > > >> > > >>>>>>>>>>>>>>> whole. > > >> > > >>>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>>> Thanks, > > >> > > >>>>>>>>>>>>>>> Thomas > > >> > > >>>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>>> On Mon, Oct 18, 2021 at 6:59 AM Chesnay Schepler < > > >> > > >>>>>>>>> ches...@apache.org > > >> > > >>>>>>>>>>>>>>> wrote: > > >> > > >>>>>>>>>>>>>>>> Generally, the issues are reproducibility and > > >> > > >>> control. > > >> > > >>>>>>>>>>>>>>>> Stuffs completely broken on the Flink side for a > > >> > > >>> week? > > >> > > >>>>>> Well > > >> > > >>>>>>>>> then so > > >> > > >>>>>>>>>>>> are > > >> > > >>>>>>>>>>>>>>>> the connector repos. > > >> > > >>>>>>>>>>>>>>>> (As-is) You can't go back to a previous version > of > > >> > > >>>>>>>>>>>>>>>> the > > >> > > >>>>>>>> snapshot. > > >> > > >>>>>>>>>>>> Which > > >> > > >>>>>>>>>>>>>>>> also means that checking out older commits can be > > >> > > >>>>>>> problematic > > >> > > >>>>>>>>>>>> because > > >> > > >>>>>>>>>>>>>>>> you'd still work against the latest snapshots, > and > > >> > > >>>>>>>>>>>>>>>> they > > >> > > >>>>>> not > > >> > > >>>>>>> be > > >> > > >>>>>>>>>>>>>>>> compatible with each other. > > >> > > >>>>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>>>> > > >> > > >>>>>>>>>>>>>>>> On 18/10/2021 15:22, Arvid Heise wrote: > > >> > > >>>>>>>>>>>>>>>>> I was actually betting on snapshots versions. > > >> > > >>>>>>>>>>>>>>>>> What are > > >> > > >>>>>> the > > >> > > >>>>>>>>> limits? > > >> > > >>>>>>>>>>>>>>>>> Obviously, we can only do a release of a 1.15 > > >> > > >>>>> connector > > >> > > >>>>>>> after > > >> > > >>>>>>>>> 1.15 > > >> > > >>>>>>>>>>>> is > > >> > > >>>>>>>>>>>>>>>>> release. > > >> > > >>>>>>>>>>>>> > > >> > > >>>>>>>>>>> > > >> > > >>>>>>>>>>> -- > > >> > > >>>>>>>>>>> > > >> > > >>>>>>>>>>> Konstantin Knauf > > >> > > >>>>>>>>>>> > > >> > > >>>>>>>>>>> > https://urldefense.com/v3/__https://twitter.com/snntrable > > >> > > >>>>>>>>>>> > __;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_- > > >> > > >>>>>>>>>>> XjpYgX5MUy9M4$ [twitter[.]com] > > >> > > >>>>>>>>>>> > > >> > > >>>>>>>>>>> > https://urldefense.com/v3/__https://github.com/knaufk__;! > > >> > > >>>>>>>>>>> > !LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpY > > >> > > >>>>>>>>>>> gXyX8u50S$ [github[.]com] > > >> > > >>>>>>>>>>> > > >> > > > > >> > > > > >> > > > >> > > > >
