+1 for repo per connector from my side also Thanks for trying out the different approaches.
Where would the common/infra pieces live? In a separate repository with its own release? Thomas On Thu, Dec 9, 2021 at 12:42 PM Till Rohrmann <trohrm...@apache.org> wrote: > > Sorry if I was a bit unclear. +1 for the single repo per connector approach. > > Cheers, > Till > > On Thu, Dec 9, 2021 at 5:41 PM Till Rohrmann <trohrm...@apache.org> wrote: > > > +1 for the single repo approach. > > > > Cheers, > > Till > > > > On Thu, Dec 9, 2021 at 3:54 PM Martijn Visser <mart...@ververica.com> > > wrote: > > > >> I also agree that it feels more natural to go with a repo for each > >> individual connector. Each repository can be made available at > >> flink-packages.org so users can find them, next to referring to them in > >> documentation. +1 from my side. > >> > >> On Thu, 9 Dec 2021 at 15:38, Arvid Heise <ar...@apache.org> wrote: > >> > >> > Hi all, > >> > > >> > We tried out Chesnay's proposal and went with Option 2. Unfortunately, > >> we > >> > experienced tough nuts to crack and feel like we hit a dead end: > >> > - The main pain point with the outlined Frankensteinian connector repo > >> is > >> > how to handle shared code / infra code. If we have it in some <common> > >> > branch, then we need to merge the common branch in the connector branch > >> on > >> > update. However, it's unclear to me how improvements in the common > >> branch > >> > that naturally appear while working on a specific connector go back into > >> > the common branch. You can't use a pull request from your branch or else > >> > your connector code would poison the connector-less common branch. So > >> you > >> > would probably manually copy the files over to a common branch and > >> create a > >> > PR branch for that. > >> > - A weird solution could be to have the common branch as a submodule in > >> the > >> > repo itself (if that's even possible). I'm sure that this setup would > >> blow > >> > up the minds of all newcomers. > >> > - Similarly, it's mandatory to have safeguards against code from > >> connector > >> > A poisoning connector B, common, or main. I had some similar setup in > >> the > >> > past and code from two "distinct" branch types constantly swept over. > >> > - We could also say that we simply release <common> independently and > >> just > >> > have a maven (SNAPSHOT) dependency on it. But that would create a weird > >> > flow if you need to change in common where you need to constantly switch > >> > branches back and forth. > >> > - In general, Frankensteinian's approach is very switch intensive. If > >> you > >> > maintain 3 connectors and need to fix 1 build stability each at the same > >> > time (quite common nowadays for some reason) and you have 2 review > >> rounds, > >> > you need to switch branches 9 times ignoring changes to common. > >> > > >> > Additionally, we still have the rather user/dev unfriendly main that is > >> > mostly empty. I'm also not sure we can generate an overview README.md to > >> > make it more friendly here because in theory every connector branch > >> should > >> > be based on main and we would get merge conflicts. > >> > > >> > I'd like to propose once again to go with individual repositories. > >> > - The only downside that we discussed so far is that we have more > >> initial > >> > setup to do. Since we organically grow the number of > >> connector/repositories > >> > that load is quite distributed. We can offer templates after finding a > >> good > >> > approach that can even be used by outside organizations. > >> > - Regarding secrets, I think it's actually an advantage that the Kafka > >> > connector has no access to the AWS secrets. If there are secrets to be > >> > shared across connectors, we can and should use Azure's Variable Groups > >> (I > >> > have used it in the past to share Nexus creds across repos). That would > >> > also make rotation easy. > >> > - Working on different connectors would be rather easy as all modern IDE > >> > support multiple repo setups in the same project. You still need to do > >> > multiple releases in case you update common code (either accessed > >> through > >> > Nexus or git submodule) and you want to release your connector. > >> > - There is no difference in respect to how many CI runs there in both > >> > approaches. > >> > - Individual repositories also have the advantage of allowing external > >> > incubation. Let's assume someone builds connector A and hosts it in > >> their > >> > organization (very common setup). If they want to contribute the code to > >> > Flink, we could simply transfer the repository into ASF after ensuring > >> > Flink coding standards. Then we retain git history and Github issues. > >> > > >> > Is there any point that I'm missing? > >> > > >> > On Fri, Nov 26, 2021 at 1:32 PM Chesnay Schepler <ches...@apache.org> > >> > wrote: > >> > > >> > > For sharing workflows we should be able to use composite actions. We'd > >> > > have the main definition files in the flink-connectors repo, that we > >> > > also need to tag/release, which other branches/repos can then import. > >> > > These are also versioned, so we don't have to worry about accidentally > >> > > breaking stuff. > >> > > These could also be used to enforce certain standards / interfaces > >> such > >> > > that we can automate more things (e.g., integration into the Flink > >> > > documentation). > >> > > > >> > > It is true that Option 2) and dedicated repositories share a lot of > >> > > properties. While I did say in an offline conversation that we in that > >> > > case might just as well use separate repositories, I'm not so sure > >> > > anymore. One repo would make administration a bit easier, for example > >> > > secrets wouldn't have to be applied to each repo (we wouldn't want > >> > > certain secrets to be set up organization-wide). > >> > > I overall also like that one repo would present a single access point; > >> > > you can't "miss" a connector repo, and I would hope that having it as > >> > > one repo would nurture more collaboration between the connectors, > >> which > >> > > after all need to solve similar problems. > >> > > > >> > > It is a fair point that the branching model would be quite weird, but > >> I > >> > > think that would subside pretty quickly. > >> > > > >> > > Personally I'd go with Option 2, and if that doesn't work out we can > >> > > still split the repo later on. (Which should then be a trivial matter > >> of > >> > > copying all <connector>/* branches and renaming them). > >> > > > >> > > On 26/11/2021 12:47, Till Rohrmann wrote: > >> > > > Hi Arvid, > >> > > > > >> > > > Thanks for updating this thread with the latest findings. The > >> described > >> > > > limitations for a single connector repo sound suboptimal to me. > >> > > > > >> > > > * Option 2. sounds as if we try to simulate multi connector repos > >> > inside > >> > > of > >> > > > a single repo. I also don't know how we would share code between the > >> > > > different branches (sharing infrastructure would probably be easier > >> > > > though). This seems to have the same limitations as dedicated repos > >> > with > >> > > > the downside of having a not very intuitive branching model. > >> > > > * Isn't option 1. kind of a degenerated version of option 2. where > >> we > >> > > have > >> > > > some unrelated code from other connectors in the individual > >> connector > >> > > > branches? > >> > > > * Option 3. has the downside that someone creating a release has to > >> > > release > >> > > > all connectors. This means that she either has to sync with the > >> > different > >> > > > connector maintainers or has to be able to release all connectors on > >> > her > >> > > > own. We are already seeing in the Flink community that releases > >> require > >> > > > quite good communication/coordination between the different people > >> > > working > >> > > > on different Flink components. Given our goals to make connector > >> > releases > >> > > > easier and more frequent, I think that coupling different connector > >> > > > releases might be counter-productive. > >> > > > > >> > > > To me it sounds not very practical to mainly use a mono repository > >> w/o > >> > > > having some more advanced build infrastructure that, for example, > >> > allows > >> > > to > >> > > > have different git roots in different connector directories. Maybe > >> the > >> > > mono > >> > > > repo can be a catch all repository for connectors that want to be > >> > > released > >> > > > in lock-step (Option 3.) with all other connectors the repo > >> contains. > >> > But > >> > > > for connectors that get changed frequently, having a dedicated > >> > repository > >> > > > that allows independent releases sounds preferable to me. > >> > > > > >> > > > What utilities and infrastructure code do you intend to share? Using > >> > git > >> > > > submodules can definitely be one option to share code. However, it > >> > might > >> > > > also be ok to depend on flink-connector-common artifacts which could > >> > make > >> > > > things easier. Where I am unsure is whether git submodules can be > >> used > >> > to > >> > > > share infrastructure code (e.g. the .github/workflows) because you > >> need > >> > > > these files in the repo to trigger the CI infrastructure. > >> > > > > >> > > > Cheers, > >> > > > Till > >> > > > > >> > > > On Thu, Nov 25, 2021 at 1:59 PM Arvid Heise <ar...@apache.org> > >> wrote: > >> > > > > >> > > >> Hi Brian, > >> > > >> > >> > > >> Thank you for sharing. I think your approach is very valid and is > >> in > >> > > line > >> > > >> with what I had in mind. > >> > > >> > >> > > >> Basically Pravega community aligns the connector releases with the > >> > > Pravega > >> > > >>> mainline release > >> > > >>> > >> > > >> This certainly would mean that there is little value in coupling > >> > > connector > >> > > >> versions. So it's making a good case for having separate connector > >> > > repos. > >> > > >> > >> > > >> > >> > > >>> and maintains the connector with the latest 3 Flink versions(CI > >> will > >> > > >>> publish snapshots for all these 3 branches) > >> > > >>> > >> > > >> I'd like to give connector devs a simple way to express to which > >> Flink > >> > > >> versions the current branch is compatible. From there we can > >> generate > >> > > the > >> > > >> compatibility matrix automatically and optionally also create > >> > different > >> > > >> releases per supported Flink version. Not sure if the latter is > >> indeed > >> > > >> better than having just one artifact that happens to run with > >> multiple > >> > > >> Flink versions. I guess it depends on what dependencies we are > >> > > exposing. If > >> > > >> the connector uses flink-connector-base, then we probably need > >> > separate > >> > > >> artifacts with poms anyways. > >> > > >> > >> > > >> Best, > >> > > >> > >> > > >> Arvid > >> > > >> > >> > > >> On Fri, Nov 19, 2021 at 10:55 AM Zhou, Brian <b.z...@dell.com> > >> wrote: > >> > > >> > >> > > >>> Hi Arvid, > >> > > >>> > >> > > >>> For branching model, the Pravega Flink connector has some > >> experience > >> > > what > >> > > >>> I would like to share. Here[1][2] is the compatibility matrix and > >> > wiki > >> > > >>> explaining the branching model and releases. Basically Pravega > >> > > community > >> > > >>> aligns the connector releases with the Pravega mainline release, > >> and > >> > > >>> maintains the connector with the latest 3 Flink versions(CI will > >> > > publish > >> > > >>> snapshots for all these 3 branches). > >> > > >>> For example, recently we have 0.10.1 release[3], and in maven > >> central > >> > > we > >> > > >>> need to upload three artifacts(For Flink 1.13, 1.12, 1.11) for > >> 0.10.1 > >> > > >>> version[4]. > >> > > >>> > >> > > >>> There are some alternatives. Another solution that we once > >> discussed > >> > > but > >> > > >>> finally got abandoned is to have a independent version just like > >> the > >> > > >>> current CDC connector, and then give a big compatibility matrix to > >> > > users. > >> > > >>> We think it would be too confusing when the connector develops. On > >> > the > >> > > >>> contrary, we can also do the opposite way to align with Flink > >> version > >> > > and > >> > > >>> maintain several branches for different system version. > >> > > >>> > >> > > >>> I would say this is only a fairly-OK solution because it is a bit > >> > > painful > >> > > >>> for maintainers as cherry-picks are very common and releases would > >> > > >> require > >> > > >>> much work. However, if neither systems do not have a nice backward > >> > > >>> compatibility, there seems to be no comfortable solution to the > >> their > >> > > >>> connector. > >> > > >>> > >> > > >>> [1] > >> https://github.com/pravega/flink-connectors#compatibility-matrix > >> > > >>> [2] > >> > > >>> > >> > > >> > >> > > > >> > > >> https://github.com/pravega/flink-connectors/wiki/Versioning-strategy-for-Flink-connector > >> > > >>> [3] > >> https://github.com/pravega/flink-connectors/releases/tag/v0.10.1 > >> > > >>> [4] https://search.maven.org/search?q=pravega-connectors-flink > >> > > >>> > >> > > >>> Best Regards, > >> > > >>> Brian > >> > > >>> > >> > > >>> > >> > > >>> Internal Use - Confidential > >> > > >>> > >> > > >>> -----Original Message----- > >> > > >>> From: Arvid Heise <ar...@apache.org> > >> > > >>> Sent: Friday, November 19, 2021 4:12 PM > >> > > >>> To: dev > >> > > >>> Subject: Re: [DISCUSS] Creating an external connector repository > >> > > >>> > >> > > >>> > >> > > >>> [EXTERNAL EMAIL] > >> > > >>> > >> > > >>> Hi everyone, > >> > > >>> > >> > > >>> we are currently in the process of setting up the flink-connectors > >> > repo > >> > > >>> [1] for new connectors but we hit a wall that we currently cannot > >> > take: > >> > > >>> branching model. > >> > > >>> To reiterate the original motivation of the external connector > >> repo: > >> > We > >> > > >>> want to decouple the release cycle of a connector with Flink. > >> > However, > >> > > if > >> > > >>> we want to support semantic versioning in the connectors with the > >> > > ability > >> > > >>> to introduce breaking changes through major version bumps and > >> support > >> > > >>> bugfixes on old versions, then we need release branches similar to > >> > how > >> > > >>> Flink core operates. > >> > > >>> Consider two connectors, let's call them kafka and hbase. We have > >> > kafka > >> > > >> in > >> > > >>> version 1.0.X, 1.1.Y (small improvement), 2.0.Z (config option) > >> > change > >> > > >> and > >> > > >>> hbase only on 1.0.A. > >> > > >>> > >> > > >>> Now our current assumption was that we can work with a mono-repo > >> > under > >> > > >> ASF > >> > > >>> (flink-connectors). Then, for release-branches, we found 3 > >> options: > >> > > >>> 1. We would need to create some ugly mess with the cross product > >> of > >> > > >>> connector and version: so you have kafka-release-1.0, > >> > > kafka-release-1.1, > >> > > >>> kafka-release-2.0, hbase-release-1.0. The main issue is not the > >> > amount > >> > > of > >> > > >>> branches (that's something that git can handle) but there the > >> state > >> > of > >> > > >>> kafka is undefined in hbase-release-1.0. That's a call for > >> desaster > >> > and > >> > > >>> makes releasing connectors very cumbersome (CI would only execute > >> and > >> > > >>> publish hbase SNAPSHOTS on hbase-release-1.0). > >> > > >>> 2. We could avoid the undefined state by having an empty master > >> and > >> > > each > >> > > >>> release branch really only holds the code of the connector. But > >> > that's > >> > > >> also > >> > > >>> not great: any user that looks at the repo and sees no connector > >> > would > >> > > >>> assume that it's dead. > >> > > >>> 3. We could have synced releases similar to the CDC connectors > >> [2]. > >> > > That > >> > > >>> means that if any connector introduces a breaking change, all > >> > > connectors > >> > > >>> get a new major. I find that quite confusing to a user if hbase > >> gets > >> > a > >> > > >> new > >> > > >>> release without any change because kafka introduced a breaking > >> > change. > >> > > >>> > >> > > >>> To fully decouple release cycles and CI of connectors, we could > >> add > >> > > >>> individual repositories under ASF (flink-connector-kafka, > >> > > >>> flink-connector-hbase). Then we can apply the same branching > >> model as > >> > > >>> before. I quickly checked if there are precedences in the apache > >> > > >> community > >> > > >>> for that approach and just by scanning alphabetically I found > >> cordova > >> > > >> with > >> > > >>> 70 and couchdb with 77 apache repos respectively. So it certainly > >> > seems > >> > > >>> like other projects approached our problem in that way and the > >> apache > >> > > >>> organization is okay with that. I currently expect max 20 > >> additional > >> > > >> repos > >> > > >>> for connectors and in the future 10 max each for formats and > >> > > filesystems > >> > > >> if > >> > > >>> we would also move them out at some point in time. So we would be > >> at > >> > a > >> > > >>> total of 50 repos. > >> > > >>> > >> > > >>> Note for all options, we need to provide a compability matrix > >> that we > >> > > aim > >> > > >>> to autogenerate. > >> > > >>> > >> > > >>> Now for the potential downsides that we internally discussed: > >> > > >>> - How can we ensure common infra structure code, utilties, and > >> > quality? > >> > > >>> I propose to add a flink-connector-common that contains all these > >> > > things > >> > > >>> and is added as a git submodule/subtree to the repos. > >> > > >>> - Do we implicitly discourage connector developers to maintain > >> more > >> > > than > >> > > >>> one connector with a fragmented code base? > >> > > >>> That is certainly a risk. However, I currently also see few devs > >> > > working > >> > > >>> on more than one connector. However, it may actually help keeping > >> the > >> > > >> devs > >> > > >>> that maintain a specific connector on the hook. We could use > >> github > >> > > >> issues > >> > > >>> to track bugs and feature requests and a dev can focus his limited > >> > time > >> > > >> on > >> > > >>> getting that one connector right. > >> > > >>> > >> > > >>> So WDYT? Compared to some intermediate suggestions with split > >> repos, > >> > > the > >> > > >>> big difference is that everything remains under Apache umbrella > >> and > >> > the > >> > > >>> Flink community. > >> > > >>> > >> > > >>> [1] > >> > > >>> > >> > > >> > >> > > > >> > > >> https://urldefense.com/v3/__https://github.com/apache/flink-connectors__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzxxweh4$ > >> > > >>> [github[.]com] [2] > >> > > >>> > >> > > >> > >> > > > >> > > >> https://urldefense.com/v3/__https://github.com/ververica/flink-cdc-connectors/__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzgoPGA8$ > >> > > >>> [github[.]com] > >> > > >>> > >> > > >>> On Fri, Nov 12, 2021 at 3:39 PM Arvid Heise <ar...@apache.org> > >> > wrote: > >> > > >>> > >> > > >>>> Hi everyone, > >> > > >>>> > >> > > >>>> I created the flink-connectors repo [1] to advance the topic. We > >> > would > >> > > >>>> create a proof-of-concept in the next few weeks as a special > >> branch > >> > > >>>> that I'd then use for discussions. If the community agrees with > >> the > >> > > >>>> approach, that special branch will become the master. If not, we > >> can > >> > > >>>> reiterate over it or create competing POCs. > >> > > >>>> > >> > > >>>> If someone wants to try things out in parallel, just make sure > >> that > >> > > >>>> you are not accidentally pushing POCs to the master. > >> > > >>>> > >> > > >>>> As a reminder: We will not move out any current connector from > >> Flink > >> > > >>>> at this point in time, so everything in Flink will remain as is > >> and > >> > be > >> > > >>>> maintained there. > >> > > >>>> > >> > > >>>> Best, > >> > > >>>> > >> > > >>>> Arvid > >> > > >>>> > >> > > >>>> [1] > >> > > >>>> > >> > > > >> https://urldefense.com/v3/__https://github.com/apache/flink-connectors > >> > > >>>> > >> > __;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzxxweh4 > >> > > >>>> $ [github[.]com] > >> > > >>>> > >> > > >>>> On Fri, Oct 29, 2021 at 6:57 PM Till Rohrmann < > >> trohrm...@apache.org > >> > > > >> > > >>>> wrote: > >> > > >>>> > >> > > >>>>> Hi everyone, > >> > > >>>>> > >> > > >>>>> From the discussion, it seems to me that we have different > >> > opinions > >> > > >>>>> whether to have an ASF umbrella repository or to host them > >> outside > >> > of > >> > > >>>>> the ASF. It also seems that this is not really the problem to > >> > solve. > >> > > >>>>> Since there are many good arguments for either approach, we > >> could > >> > > >>>>> simply start with an ASF umbrella repository and see how people > >> > adopt > >> > > >>>>> it. If the individual connectors cannot move fast enough or if > >> > people > >> > > >>>>> prefer to not buy into the more heavy-weight ASF processes, then > >> > they > >> > > >>>>> can host the code also somewhere else. We simply need to make > >> sure > >> > > >>>>> that these connectors are discoverable (e.g. via > >> flink-packages). > >> > > >>>>> > >> > > >>>>> The more important problem seems to be to provide common tooling > >> > > >>>>> (testing, infrastructure, documentation) that can easily be > >> reused. > >> > > >>>>> Similarly, it has become clear that the Flink community needs to > >> > > >>>>> improve on providing stable APIs. I think it is not realistic to > >> > > >>>>> first complete these tasks before starting to move connectors to > >> > > >>>>> dedicated repositories. As Stephan said, creating a connector > >> > > >>>>> repository will force us to pay more attention to API stability > >> and > >> > > >>>>> also to think about which testing tools are required. Hence, I > >> > > >>>>> believe that starting to add connectors to a different > >> repository > >> > > >>>>> than apache/flink will help improve our connector tooling > >> > (declaring > >> > > >>>>> testing classes as public, creating a common test utility repo, > >> > > >>>>> creating a repo > >> > > >>>>> template) and vice versa. Hence, I like Arvid's proposed > >> process as > >> > > >>>>> it will start kicking things off w/o letting this effort fizzle > >> > out. > >> > > >>>>> > >> > > >>>>> Cheers, > >> > > >>>>> Till > >> > > >>>>> > >> > > >>>>> On Thu, Oct 28, 2021 at 11:44 AM Stephan Ewen <se...@apache.org > >> > > >> > > >> wrote: > >> > > >>>>>> Thank you all, for the nice discussion! > >> > > >>>>>> > >> > > >>>>>> From my point of view, I very much like the idea of putting > >> > > >>>>>> connectors > >> > > >>>>> in a > >> > > >>>>>> separate repository. But I would argue it should be part of > >> Apache > >> > > >>>>> Flink, > >> > > >>>>>> similar to flink-statefun, flink-ml, etc. > >> > > >>>>>> > >> > > >>>>>> I share many of the reasons for that: > >> > > >>>>>> - As argued many times, reduces complexity of the Flink > >> repo, > >> > > >>>>> increases > >> > > >>>>>> response times of CI, etc. > >> > > >>>>>> - Much lower barrier of contribution, because an unstable > >> > > >>>>>> connector > >> > > >>>>> would > >> > > >>>>>> not de-stabilize the whole build. Of course, we would need to > >> make > >> > > >>>>>> sure > >> > > >>>>> we > >> > > >>>>>> set this up the right way, with connectors having individual CI > >> > > >>>>>> runs, > >> > > >>>>> build > >> > > >>>>>> status, etc. But it certainly seems possible. > >> > > >>>>>> > >> > > >>>>>> > >> > > >>>>>> I would argue some points a bit different than some cases made > >> > > >> before: > >> > > >>>>>> (a) I believe the separation would increase connector > >> stability. > >> > > >>>>> Because it > >> > > >>>>>> really forces us to work with the connectors against the APIs > >> like > >> > > >>>>>> any external developer. A mono repo is somehow the wrong thing > >> if > >> > > >>>>>> you in practice want to actually guarantee stable internal > >> APIs at > >> > > >>> some layer. > >> > > >>>>>> Because the mono repo makes it easy to just change something on > >> > > >>>>>> both > >> > > >>>>> sides > >> > > >>>>>> of the API (provider and consumer) seamlessly. > >> > > >>>>>> > >> > > >>>>>> Major refactorings in Flink need to keep all connector API > >> > > >>>>>> contracts intact, or we need to have a new version of the > >> > connector > >> > > >>> API. > >> > > >>>>>> (b) We may even be able to go towards more lightweight and > >> > > >>>>>> automated releases over time, even if we stay in Apache Flink > >> with > >> > > >>> that repo. > >> > > >>>>>> This isn't yet fully aligned with the Apache release policies, > >> > yet, > >> > > >>>>>> but there are board discussions about whether there can be > >> > > >>>>>> bot-triggered releases (by dependabot) and how that could fit > >> into > >> > > >>> the Apache process. > >> > > >>>>>> This doesn't seem to be quite there just yet, but seeing that > >> > those > >> > > >>>>> start > >> > > >>>>>> is a good sign, and there is a good chance we can do some > >> things > >> > > >>> there. > >> > > >>>>>> I am not sure whether we should let bots trigger releases, > >> because > >> > > >>>>>> a > >> > > >>>>> final > >> > > >>>>>> human look at things isn't a bad thing, especially given the > >> > > >>>>>> popularity > >> > > >>>>> of > >> > > >>>>>> software supply chain attacks recently. > >> > > >>>>>> > >> > > >>>>>> > >> > > >>>>>> I do share Chesnay's concerns about complexity in tooling, > >> though. > >> > > >>>>>> Both release tooling and test tooling. They are not > >> incompatible > >> > > >>>>>> with that approach, but they are a task we need to tackle > >> during > >> > > >>>>>> this change which will add additional work. > >> > > >>>>>> > >> > > >>>>>> > >> > > >>>>>> > >> > > >>>>>> On Tue, Oct 26, 2021 at 10:31 AM Arvid Heise <ar...@apache.org > >> > > >> > > >>> wrote: > >> > > >>>>>>> Hi folks, > >> > > >>>>>>> > >> > > >>>>>>> I think some questions came up and I'd like to address the > >> > > >>>>>>> question of > >> > > >>>>>> the > >> > > >>>>>>> timing. > >> > > >>>>>>> > >> > > >>>>>>> Could you clarify what release cadence you're thinking of? > >> > > >>>>>>> There's > >> > > >>>>> quite > >> > > >>>>>>>> a big range that fits "more frequent than Flink" (per-commit, > >> > > >>>>>>>> daily, weekly, bi-weekly, monthly, even bi-monthly). > >> > > >>>>>>> The short answer is: as often as needed: > >> > > >>>>>>> - If there is a CVE in a dependency and we need to bump it - > >> > > >>>>>>> release immediately. > >> > > >>>>>>> - If there is a new feature merged, release soonish. We may > >> > > >>>>>>> collect a > >> > > >>>>> few > >> > > >>>>>>> successive features before a release. > >> > > >>>>>>> - If there is a bugfix, release immediately or soonish > >> depending > >> > > >>>>>>> on > >> > > >>>>> the > >> > > >>>>>>> severity and if there are workarounds available. > >> > > >>>>>>> > >> > > >>>>>>> We should not limit ourselves; the whole idea of independent > >> > > >>>>>>> releases > >> > > >>>>> is > >> > > >>>>>>> exactly that you release as needed. There is no release > >> planning > >> > > >>>>>>> or anything needed, you just go with a release as if it was an > >> > > >>>>>>> external artifact. > >> > > >>>>>>> > >> > > >>>>>>> (1) is the connector API already stable? > >> > > >>>>>>>> From another discussion thread [1], connector API is far > >> from > >> > > >>>>> stable. > >> > > >>>>>>>> Currently, it's hard to build connectors against multiple > >> Flink > >> > > >>>>>> versions. > >> > > >>>>>>>> There are breaking API changes both in 1.12 -> 1.13 and 1.13 > >> -> > >> > > >>>>>>>> 1.14 > >> > > >>>>>> and > >> > > >>>>>>>> maybe also in the future versions, because Table related > >> APIs > >> > > >>>>>>>> are > >> > > >>>>>> still > >> > > >>>>>>>> @PublicEvolving and new Sink API is still @Experimental. > >> > > >>>>>>>> > >> > > >>>>>>> The question is: what is stable in an evolving system? We > >> > > >>>>>>> recently discovered that the old SourceFunction needed to be > >> > > >>>>>>> refined such that cancellation works correctly [1]. So that > >> > > >>>>>>> interface is in Flink since > >> > > >>>>> 7 > >> > > >>>>>>> years, heavily used also outside, and we still had to change > >> the > >> > > >>>>> contract > >> > > >>>>>>> in a way that I'd expect any implementer to recheck their > >> > > >>>>> implementation. > >> > > >>>>>>> It might not be necessary to change anything and you can > >> probably > >> > > >>>>> change > >> > > >>>>>>> the the code for all Flink versions but still, the interface > >> was > >> > > >>>>>>> not > >> > > >>>>>> stable > >> > > >>>>>>> in the closest sense. > >> > > >>>>>>> > >> > > >>>>>>> If we focus just on API changes on the unified interfaces, > >> then > >> > > >>>>>>> we > >> > > >>>>> expect > >> > > >>>>>>> one more change to Sink API to support compaction. For Table > >> API, > >> > > >>>>> there > >> > > >>>>>>> will most likely also be some changes in 1.15. So we could > >> wait > >> > > >>>>>>> for > >> > > >>>>> 1.15. > >> > > >>>>>>> But I'm questioning if that's really necessary because we will > >> > > >>>>>>> add > >> > > >>>>> more > >> > > >>>>>>> functionality beyond 1.15 without breaking API. For example, > >> we > >> > > >>>>>>> may > >> > > >>>>> add > >> > > >>>>>>> more unified connector metrics. If you want to use it in your > >> > > >>>>> connector, > >> > > >>>>>>> you have to support multiple Flink versions anyhow. So rather > >> > > >>>>>>> then > >> > > >>>>>> focusing > >> > > >>>>>>> the discussion on "when is stuff stable", I'd rather focus on > >> > > >>>>>>> "how > >> > > >>>>> can we > >> > > >>>>>>> support building connectors against multiple Flink versions" > >> and > >> > > >>>>>>> make > >> > > >>>>> it > >> > > >>>>>> as > >> > > >>>>>>> painless as possible. > >> > > >>>>>>> > >> > > >>>>>>> Chesnay pointed out to use different branches for different > >> Flink > >> > > >>>>>> versions > >> > > >>>>>>> which sounds like a good suggestion. With a mono-repo, we > >> can't > >> > > >>>>>>> use branches differently anyways (there is no way to have > >> release > >> > > >>>>>>> branches > >> > > >>>>>> per > >> > > >>>>>>> connector without chaos). In these branches, we could provide > >> > > >>>>>>> shims to simulate future features in older Flink versions such > >> > > >>>>>>> that code-wise, > >> > > >>>>> the > >> > > >>>>>>> source code of a specific connector may not diverge (much). > >> For > >> > > >>>>> example, > >> > > >>>>>> to > >> > > >>>>>>> register unified connector metrics, we could simulate the > >> current > >> > > >>>>>> approach > >> > > >>>>>>> also in some utility package of the mono-repo. > >> > > >>>>>>> > >> > > >>>>>>> I see the stable core Flink API as a prerequisite for > >> modularity. > >> > > >>>>>>> And > >> > > >>>>>>>> for connectors it is not just the source and sink API (source > >> > > >>>>>>>> being stable as of 1.14), but everything that is required to > >> > > >>>>>>>> build and maintain a connector downstream, such as the test > >> > > >>>>>>>> utilities and infrastructure. > >> > > >>>>>>>> > >> > > >>>>>>> That is a very fair point. I'm actually surprised to see that > >> > > >>>>>>> MiniClusterWithClientResource is not public. I see it being > >> used > >> > > >>>>>>> in > >> > > >>>>> all > >> > > >>>>>>> connectors, especially outside of Flink. I fear that as long > >> as > >> > > >>>>>>> we do > >> > > >>>>> not > >> > > >>>>>>> have connectors outside, we will not properly annotate and > >> > > >>>>>>> maintain > >> > > >>>>> these > >> > > >>>>>>> utilties in a classic hen-and-egg-problem. I will outline an > >> idea > >> > > >>>>>>> at > >> > > >>>>> the > >> > > >>>>>>> end. > >> > > >>>>>>> > >> > > >>>>>>>> the connectors need to be adopted and require at least one > >> > > >>>>>>>> release > >> > > >>>>> per > >> > > >>>>>>>> Flink minor release. > >> > > >>>>>>>> However, this will make the releases of connectors slower, > >> e.g. > >> > > >>>>>> maintain > >> > > >>>>>>>> features for multiple branches and release multiple branches. > >> > > >>>>>>>> I think the main purpose of having an external connector > >> > > >>>>>>>> repository > >> > > >>>>> is > >> > > >>>>>> in > >> > > >>>>>>>> order to have "faster releases of connectors"? > >> > > >>>>>>>> > >> > > >>>>>>>> Imagine a project with a complex set of dependencies. Let's > >> say > >> > > >>>>> Flink > >> > > >>>>>>>> version A plus Flink reliant dependencies released by other > >> > > >>>>>>>> projects (Flink-external connectors, Beam, Iceberg, Hudi, > >> ..). > >> > > >>>>>>>> We don't want > >> > > >>>>> a > >> > > >>>>>>>> situation where we bump the core Flink version to B and > >> things > >> > > >>>>>>>> fall apart (interface changes, utilities that were useful but > >> > > >>>>>>>> not public, transitive dependencies etc.). > >> > > >>>>>>>> > >> > > >>>>>>> Yes, that's why I wanted to automate the processes more which > >> is > >> > > >>>>>>> not > >> > > >>>>> that > >> > > >>>>>>> easy under ASF. Maybe we automate the source provision across > >> > > >>>>> supported > >> > > >>>>>>> versions and have 1 vote thread for all versions of a > >> connector? > >> > > >>>>>>> > >> > > >>>>>>> From the perspective of CDC connector maintainers, the > >> biggest > >> > > >>>>> advantage > >> > > >>>>>> of > >> > > >>>>>>>> maintaining it outside of the Flink project is that: > >> > > >>>>>>>> 1) we can have a more flexible and faster release cycle > >> > > >>>>>>>> 2) we can be more liberal with committership for connector > >> > > >>>>> maintainers > >> > > >>>>>>>> which can also attract more committers to help the release. > >> > > >>>>>>>> > >> > > >>>>>>>> Personally, I think maintaining one connector repository > >> under > >> > > >>>>>>>> the > >> > > >>>>> ASF > >> > > >>>>>>> may > >> > > >>>>>>>> not have the above benefits. > >> > > >>>>>>>> > >> > > >>>>>>> Yes, I also feel that ASF is too restrictive for our needs. > >> But > >> > > >>>>>>> it > >> > > >>>>> feels > >> > > >>>>>>> like there are too many that see it differently and I think we > >> > > >>>>>>> need > >> > > >>>>>>> > >> > > >>>>>>> (2) Flink testability without connectors. > >> > > >>>>>>>> This is a very good question. How can we guarantee the new > >> > > >>>>>>>> Source > >> > > >>>>> and > >> > > >>>>>>> Sink > >> > > >>>>>>>> API are stable with only test implementation? > >> > > >>>>>>>> > >> > > >>>>>>> We can't and shouldn't. Since the connector repo is managed by > >> > > >>>>>>> Flink, > >> > > >>>>> a > >> > > >>>>>>> Flink release manager needs to check if the Flink connectors > >> are > >> > > >>>>> actually > >> > > >>>>>>> working prior to creating an RC. That's similar to how > >> > > >>>>>>> flink-shaded > >> > > >>>>> and > >> > > >>>>>>> flink core are related. > >> > > >>>>>>> > >> > > >>>>>>> > >> > > >>>>>>> So here is one idea that I had to get things rolling. We are > >> > > >>>>>>> going to address the external repo iteratively without > >> > > >>>>>>> compromising what we > >> > > >>>>>> already > >> > > >>>>>>> have: > >> > > >>>>>>> 1.Phase, add new contributions to external repo. We use that > >> time > >> > > >>>>>>> to > >> > > >>>>>> setup > >> > > >>>>>>> infra accordingly and optimize release processes. We will > >> > > >>>>>>> identify > >> > > >>>>> test > >> > > >>>>>>> utilities that are not yet public/stable and fix that. > >> > > >>>>>>> 2.Phase, add ports to the new unified interfaces of existing > >> > > >>>>> connectors. > >> > > >>>>>>> That requires a previous Flink release to make utilities > >> stable. > >> > > >>>>>>> Keep > >> > > >>>>> old > >> > > >>>>>>> interfaces in flink-core. > >> > > >>>>>>> 3.Phase, remove old interfaces in flink-core of some > >> connectors > >> > > >>>>>>> (tbd > >> > > >>>>> at a > >> > > >>>>>>> later point). > >> > > >>>>>>> 4.Phase, optionally move all remaining connectors (tbd at a > >> later > >> > > >>>>> point). > >> > > >>>>>>> I'd envision having ~3 months between the starting the > >> different > >> > > >>>>> phases. > >> > > >>>>>>> WDYT? > >> > > >>>>>>> > >> > > >>>>>>> > >> > > >>>>>>> [1] > >> > > >>>>>>> > >> > https://urldefense.com/v3/__https://issues.apache.org/jira/browse > >> > > >>>>>>> > >> /FLINK-23527__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgd > >> > > >>>>>>> ke_-XjpYgX2sIvAP4$ [issues[.]apache[.]org] > >> > > >>>>>>> > >> > > >>>>>>> On Thu, Oct 21, 2021 at 7:12 AM Kyle Bendickson < > >> k...@tabular.io > >> > > > >> > > >>>>> wrote: > >> > > >>>>>>>> Hi all, > >> > > >>>>>>>> > >> > > >>>>>>>> My name is Kyle and I’m an open source developer primarily > >> > > >>>>>>>> focused > >> > > >>>>> on > >> > > >>>>>>>> Apache Iceberg. > >> > > >>>>>>>> > >> > > >>>>>>>> I’m happy to help clarify or elaborate on any aspect of our > >> > > >>>>> experience > >> > > >>>>>>>> working on a relatively decoupled connector that is > >> downstream > >> > > >>>>>>>> and > >> > > >>>>>> pretty > >> > > >>>>>>>> popular. > >> > > >>>>>>>> > >> > > >>>>>>>> I’d also love to be able to contribute or assist in any way I > >> > > >> can. > >> > > >>>>>>>> I don’t mean to thread jack, but are there any meetings or > >> > > >>>>>>>> community > >> > > >>>>>> sync > >> > > >>>>>>>> ups, specifically around the connector APIs, that I might > >> join > >> > > >>>>>>>> / be > >> > > >>>>>>> invited > >> > > >>>>>>>> to? > >> > > >>>>>>>> > >> > > >>>>>>>> I did want to add that even though I’ve experienced some of > >> the > >> > > >>>>>>>> pain > >> > > >>>>>>> points > >> > > >>>>>>>> of integrating with an evolving system / API (catalog support > >> > > >>>>>>>> is > >> > > >>>>>>> generally > >> > > >>>>>>>> speaking pretty new everywhere really in this space), I also > >> > > >>>>>>>> agree personally that you shouldn’t slow down development > >> > > >>>>>>>> velocity too > >> > > >>>>> much > >> > > >>>>>> for > >> > > >>>>>>>> the sake of external connector. Getting to a performant and > >> > > >>>>>>>> stable > >> > > >>>>>> place > >> > > >>>>>>>> should be the primary goal, and slowing that down to support > >> > > >>>>> stragglers > >> > > >>>>>>>> will (in my personal opinion) always be a losing game. Some > >> > > >>>>>>>> folks > >> > > >>>>> will > >> > > >>>>>>>> simply stay behind on versions regardless until they have to > >> > > >>>>> upgrade. > >> > > >>>>>>>> I am working on ensuring that the Iceberg community stays > >> > > >>>>>>>> within 1-2 versions of Flink, so that we can help provide > >> more > >> > > >>>>>>>> feedback or > >> > > >>>>>>> contribute > >> > > >>>>>>>> things that might make our ability to support multiple Flink > >> > > >>>>> runtimes / > >> > > >>>>>>>> versions with one project / codebase and minimal to no > >> > > >>>>>>>> reflection > >> > > >>>>> (our > >> > > >>>>>>>> desired goal). > >> > > >>>>>>>> > >> > > >>>>>>>> If there’s anything I can do or any way I can be of > >> assistance, > >> > > >>>>> please > >> > > >>>>>>>> don’t hesitate to reach out. Or find me on ASF slack 😀 > >> > > >>>>>>>> > >> > > >>>>>>>> I greatly appreciate your general concern for the needs of > >> > > >>>>> downstream > >> > > >>>>>>>> connector integrators! > >> > > >>>>>>>> > >> > > >>>>>>>> Cheers > >> > > >>>>>>>> Kyle Bendickson (GitHub: kbendick) Open Source Developer kyle > >> > > >>>>>>>> [at] tabular [dot] io > >> > > >>>>>>>> > >> > > >>>>>>>> On Wed, Oct 20, 2021 at 11:35 AM Thomas Weise < > >> t...@apache.org> > >> > > >>>>> wrote: > >> > > >>>>>>>>> Hi, > >> > > >>>>>>>>> > >> > > >>>>>>>>> I see the stable core Flink API as a prerequisite for > >> > > >>> modularity. > >> > > >>>>> And > >> > > >>>>>>>>> for connectors it is not just the source and sink API > >> (source > >> > > >>>>> being > >> > > >>>>>>>>> stable as of 1.14), but everything that is required to build > >> > > >>>>>>>>> and maintain a connector downstream, such as the test > >> > > >>>>>>>>> utilities and infrastructure. > >> > > >>>>>>>>> > >> > > >>>>>>>>> Without the stable surface of core Flink, changes will leak > >> > > >>>>>>>>> into downstream dependencies and force lock step updates. > >> > > >>>>>>>>> Refactoring across N repos is more painful than a single > >> > > >>>>>>>>> repo. Those with experience developing downstream of Flink > >> > > >>>>>>>>> will know the pain, and > >> > > >>>>>> that > >> > > >>>>>>>>> isn't limited to connectors. I don't remember a Flink "minor > >> > > >>>>> version" > >> > > >>>>>>>>> update that was just a dependency version change and did not > >> > > >>>>>>>>> force other downstream changes. > >> > > >>>>>>>>> > >> > > >>>>>>>>> Imagine a project with a complex set of dependencies. Let's > >> > > >>>>>>>>> say > >> > > >>>>> Flink > >> > > >>>>>>>>> version A plus Flink reliant dependencies released by other > >> > > >>>>> projects > >> > > >>>>>>>>> (Flink-external connectors, Beam, Iceberg, Hudi, ..). We > >> > > >>>>>>>>> don't > >> > > >>>>> want a > >> > > >>>>>>>>> situation where we bump the core Flink version to B and > >> > > >>>>>>>>> things > >> > > >>>>> fall > >> > > >>>>>>>>> apart (interface changes, utilities that were useful but not > >> > > >>>>> public, > >> > > >>>>>>>>> transitive dependencies etc.). > >> > > >>>>>>>>> > >> > > >>>>>>>>> The discussion here also highlights the benefits of keeping > >> > > >>>>> certain > >> > > >>>>>>>>> connectors outside Flink. Whether that is due to difference > >> > > >>>>>>>>> in developer community, maturity of the connectors, their > >> > > >>>>>>>>> specialized/limited usage etc. I would like to see that as a > >> > > >>>>>>>>> sign > >> > > >>>>> of > >> > > >>>>>> a > >> > > >>>>>>>>> growing ecosystem and most of the ideas that Arvid has put > >> > > >>>>>>>>> forward would benefit further growth of the connector > >> > > >> ecosystem. > >> > > >>>>>>>>> As for keeping connectors within Apache Flink: I prefer that > >> > > >>>>>>>>> as > >> > > >>>>> the > >> > > >>>>>>>>> path forward for "essential" connectors like FileSource, > >> > > >>>>> KafkaSource, > >> > > >>>>>>>>> ... And we can still achieve a more flexible and faster > >> > > >>>>>>>>> release > >> > > >>>>>> cycle. > >> > > >>>>>>>>> Thanks, > >> > > >>>>>>>>> Thomas > >> > > >>>>>>>>> > >> > > >>>>>>>>> > >> > > >>>>>>>>> > >> > > >>>>>>>>> > >> > > >>>>>>>>> > >> > > >>>>>>>>> On Wed, Oct 20, 2021 at 3:32 AM Jark Wu <imj...@gmail.com> > >> > > >>> wrote: > >> > > >>>>>>>>>> Hi Konstantin, > >> > > >>>>>>>>>> > >> > > >>>>>>>>>>> the connectors need to be adopted and require at least > >> > > >>>>>>>>>>> one > >> > > >>>>>> release > >> > > >>>>>>>> per > >> > > >>>>>>>>>> Flink minor release. > >> > > >>>>>>>>>> However, this will make the releases of connectors slower, > >> > > >>> e.g. > >> > > >>>>>>>> maintain > >> > > >>>>>>>>>> features for multiple branches and release multiple > >> > > >> branches. > >> > > >>>>>>>>>> I think the main purpose of having an external connector > >> > > >>>>> repository > >> > > >>>>>>> is > >> > > >>>>>>>> in > >> > > >>>>>>>>>> order to have "faster releases of connectors"? > >> > > >>>>>>>>>> > >> > > >>>>>>>>>> > >> > > >>>>>>>>>> From the perspective of CDC connector maintainers, the > >> > > >>>>>>>>>> biggest > >> > > >>>>>>>> advantage > >> > > >>>>>>>>> of > >> > > >>>>>>>>>> maintaining it outside of the Flink project is that: > >> > > >>>>>>>>>> 1) we can have a more flexible and faster release cycle > >> > > >>>>>>>>>> 2) we can be more liberal with committership for connector > >> > > >>>>>>> maintainers > >> > > >>>>>>>>>> which can also attract more committers to help the release. > >> > > >>>>>>>>>> > >> > > >>>>>>>>>> Personally, I think maintaining one connector repository > >> > > >>>>>>>>>> under > >> > > >>>>> the > >> > > >>>>>>> ASF > >> > > >>>>>>>>> may > >> > > >>>>>>>>>> not have the above benefits. > >> > > >>>>>>>>>> > >> > > >>>>>>>>>> Best, > >> > > >>>>>>>>>> Jark > >> > > >>>>>>>>>> > >> > > >>>>>>>>>> On Wed, 20 Oct 2021 at 15:14, Konstantin Knauf < > >> > > >>>>> kna...@apache.org> > >> > > >>>>>>>>> wrote: > >> > > >>>>>>>>>>> Hi everyone, > >> > > >>>>>>>>>>> > >> > > >>>>>>>>>>> regarding the stability of the APIs. I think everyone > >> > > >>>>>>>>>>> agrees > >> > > >>>>> that > >> > > >>>>>>>>>>> connector APIs which are stable across minor versions > >> > > >>>>>> (1.13->1.14) > >> > > >>>>>>>> are > >> > > >>>>>>>>> the > >> > > >>>>>>>>>>> mid-term goal. But: > >> > > >>>>>>>>>>> > >> > > >>>>>>>>>>> a) These APIs are still quite young, and we shouldn't > >> > > >>>>>>>>>>> make > >> > > >>>>> them > >> > > >>>>>>>> @Public > >> > > >>>>>>>>>>> prematurely either. > >> > > >>>>>>>>>>> > >> > > >>>>>>>>>>> b) Isn't this *mostly* orthogonal to where the connector > >> > > >>>>>>>>>>> code > >> > > >>>>>>> lives? > >> > > >>>>>>>>> Yes, > >> > > >>>>>>>>>>> as long as there are breaking changes, the connectors > >> > > >>>>>>>>>>> need to > >> > > >>>>> be > >> > > >>>>>>>>> adopted > >> > > >>>>>>>>>>> and require at least one release per Flink minor release. > >> > > >>>>>>>>>>> Documentation-wise this can be addressed via a > >> > > >>>>>>>>>>> compatibility > >> > > >>>>>> matrix > >> > > >>>>>>>> for > >> > > >>>>>>>>>>> each connector as Arvid suggested. IMO we shouldn't block > >> > > >>>>>>>>>>> this > >> > > >>>>>>> effort > >> > > >>>>>>>>> on > >> > > >>>>>>>>>>> the stability of the APIs. > >> > > >>>>>>>>>>> > >> > > >>>>>>>>>>> Cheers, > >> > > >>>>>>>>>>> > >> > > >>>>>>>>>>> Konstantin > >> > > >>>>>>>>>>> > >> > > >>>>>>>>>>> > >> > > >>>>>>>>>>> > >> > > >>>>>>>>>>> On Wed, Oct 20, 2021 at 8:56 AM Jark Wu > >> > > >>>>>>>>>>> <imj...@gmail.com> > >> > > >>>>>> wrote: > >> > > >>>>>>>>>>>> Hi, > >> > > >>>>>>>>>>>> > >> > > >>>>>>>>>>>> I think Thomas raised very good questions and would like > >> > > >>>>>>>>>>>> to > >> > > >>>>> know > >> > > >>>>>>>> your > >> > > >>>>>>>>>>>> opinions if we want to move connectors out of flink in > >> > > >>>>>>>>>>>> this > >> > > >>>>>>> version. > >> > > >>>>>>>>>>>> (1) is the connector API already stable? > >> > > >>>>>>>>>>>>> Separate releases would only make sense if the core > >> > > >>>>>>>>>>>>> Flink > >> > > >>>>>>> surface > >> > > >>>>>>>> is > >> > > >>>>>>>>>>>>> fairly stable though. As evident from Iceberg (and > >> > > >>>>>>>>>>>>> also > >> > > >>>>> Beam), > >> > > >>>>>>>>> that's > >> > > >>>>>>>>>>>>> not the case currently. We should probably focus on > >> > > >>>>> addressing > >> > > >>>>>>> the > >> > > >>>>>>>>>>>>> stability first, before splitting code. A success > >> > > >>>>>>>>>>>>> criteria > >> > > >>>>>> could > >> > > >>>>>>>> be > >> > > >>>>>>>>>>>>> that we are able to build Iceberg and Beam against > >> > > >>>>>>>>>>>>> multiple > >> > > >>>>>>> Flink > >> > > >>>>>>>>>>>>> versions w/o the need to change code. The goal would > >> > > >>>>>>>>>>>>> be > >> > > >>>>> that > >> > > >>>>>> no > >> > > >>>>>>>>>>>>> connector breaks when we make changes to Flink core. > >> > > >>>>>>>>>>>>> Until > >> > > >>>>>>> that's > >> > > >>>>>>>>> the > >> > > >>>>>>>>>>>>> case, code separation creates a setup where 1+1 or N+1 > >> > > >>>>>>>> repositories > >> > > >>>>>>>>>>>>> need to move lock step. > >> > > >>>>>>>>>>>> From another discussion thread [1], connector API is far > >> > > >>>>>>>>>>>> from > >> > > >>>>>>>> stable. > >> > > >>>>>>>>>>>> Currently, it's hard to build connectors against > >> > > >>>>>>>>>>>> multiple > >> > > >>>>> Flink > >> > > >>>>>>>>> versions. > >> > > >>>>>>>>>>>> There are breaking API changes both in 1.12 -> 1.13 and > >> > > >>>>>>>>>>>> 1.13 > >> > > >>>>> -> > >> > > >>>>>>> 1.14 > >> > > >>>>>>>>> and > >> > > >>>>>>>>>>>> maybe also in the future versions, because Table > >> > > >>>>>>>>>>>> related > >> > > >>>>> APIs > >> > > >>>>>>> are > >> > > >>>>>>>>> still > >> > > >>>>>>>>>>>> @PublicEvolving and new Sink API is still @Experimental. > >> > > >>>>>>>>>>>> > >> > > >>>>>>>>>>>> > >> > > >>>>>>>>>>>> (2) Flink testability without connectors. > >> > > >>>>>>>>>>>>> Flink w/o Kafka connector (and few others) isn't > >> > > >>>>>>>>>>>>> viable. Testability of Flink was already brought up, > >> > > >>>>>>>>>>>>> can we > >> > > >>>>>>> really > >> > > >>>>>>>>>>>>> certify a Flink core release without Kafka connector? > >> > > >>>>>>>>>>>>> Maybe > >> > > >>>>>>> those > >> > > >>>>>>>>>>>>> connectors that are used in Flink e2e tests to > >> > > >>>>>>>>>>>>> validate > >> > > >>>>>>>>> functionality > >> > > >>>>>>>>>>>>> of core Flink should not be broken out? > >> > > >>>>>>>>>>>> This is a very good question. How can we guarantee the > >> > > >>>>>>>>>>>> new > >> > > >>>>>> Source > >> > > >>>>>>>> and > >> > > >>>>>>>>> Sink > >> > > >>>>>>>>>>>> API are stable with only test implementation? > >> > > >>>>>>>>>>>> > >> > > >>>>>>>>>>>> > >> > > >>>>>>>>>>>> Best, > >> > > >>>>>>>>>>>> Jark > >> > > >>>>>>>>>>>> > >> > > >>>>>>>>>>>> > >> > > >>>>>>>>>>>> > >> > > >>>>>>>>>>>> > >> > > >>>>>>>>>>>> > >> > > >>>>>>>>>>>> On Tue, 19 Oct 2021 at 23:56, Chesnay Schepler < > >> > > >>>>>>> ches...@apache.org> > >> > > >>>>>>>>>>>> wrote: > >> > > >>>>>>>>>>>> > >> > > >>>>>>>>>>>>> Could you clarify what release cadence you're thinking > >> > > >>> of? > >> > > >>>>>>> There's > >> > > >>>>>>>>> quite > >> > > >>>>>>>>>>>>> a big range that fits "more frequent than Flink" > >> > > >>>>> (per-commit, > >> > > >>>>>>>> daily, > >> > > >>>>>>>>>>>>> weekly, bi-weekly, monthly, even bi-monthly). > >> > > >>>>>>>>>>>>> > >> > > >>>>>>>>>>>>> On 19/10/2021 14:15, Martijn Visser wrote: > >> > > >>>>>>>>>>>>>> Hi all, > >> > > >>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>> I think it would be a huge benefit if we can achieve > >> > > >>>>>>>>>>>>>> more > >> > > >>>>>>>> frequent > >> > > >>>>>>>>>>>>> releases > >> > > >>>>>>>>>>>>>> of connectors, which are not bound to the release > >> > > >>>>>>>>>>>>>> cycle > >> > > >>>>> of > >> > > >>>>>>> Flink > >> > > >>>>>>>>>>>> itself. > >> > > >>>>>>>>>>>>> I > >> > > >>>>>>>>>>>>>> agree that in order to get there, we need to have > >> > > >>>>>>>>>>>>>> stable > >> > > >>>>>>>>> interfaces > >> > > >>>>>>>>>>>> which > >> > > >>>>>>>>>>>>>> are trustworthy and reliable, so they can be safely > >> > > >>>>>>>>>>>>>> used > >> > > >>>>> by > >> > > >>>>>>>> those > >> > > >>>>>>>>>>>>>> connectors. I do think that work still needs to be > >> > > >>>>>>>>>>>>>> done > >> > > >>>>> on > >> > > >>>>>>> those > >> > > >>>>>>>>>>>>>> interfaces, but I am confident that we can get there > >> > > >>>>> from a > >> > > >>>>>>>> Flink > >> > > >>>>>>>>>>>>>> perspective. > >> > > >>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>> I am worried that we would not be able to achieve > >> > > >>>>>>>>>>>>>> those > >> > > >>>>>>> frequent > >> > > >>>>>>>>>>>> releases > >> > > >>>>>>>>>>>>>> of connectors if we are putting these connectors > >> > > >>>>>>>>>>>>>> under > >> > > >>>>> the > >> > > >>>>>>>> Apache > >> > > >>>>>>>>>>>>> umbrella, > >> > > >>>>>>>>>>>>>> because that means that for each connector release > >> > > >>>>>>>>>>>>>> we > >> > > >>>>> have > >> > > >>>>>> to > >> > > >>>>>>>>> follow > >> > > >>>>>>>>>>>> the > >> > > >>>>>>>>>>>>>> Apache release creation process. This requires a lot > >> > > >>>>>>>>>>>>>> of > >> > > >>>>>> manual > >> > > >>>>>>>>> steps > >> > > >>>>>>>>>>>> and > >> > > >>>>>>>>>>>>>> prohibits automation and I think it would be hard to > >> > > >>>>> scale > >> > > >>>>>> out > >> > > >>>>>>>>>>>> frequent > >> > > >>>>>>>>>>>>>> releases of connectors. I'm curious how others think > >> > > >>>>>>>>>>>>>> this > >> > > >>>>>>>>> challenge > >> > > >>>>>>>>>>>> could > >> > > >>>>>>>>>>>>>> be solved. > >> > > >>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>> Best regards, > >> > > >>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>> Martijn > >> > > >>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>> On Mon, 18 Oct 2021 at 22:22, Thomas Weise < > >> > > >>>>> t...@apache.org> > >> > > >>>>>>>>> wrote: > >> > > >>>>>>>>>>>>>>> Thanks for initiating this discussion. > >> > > >>>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>>> There are definitely a few things that are not > >> > > >>>>>>>>>>>>>>> optimal > >> > > >>>>> with > >> > > >>>>>>> our > >> > > >>>>>>>>>>>>>>> current management of connectors. I would not > >> > > >>>>> necessarily > >> > > >>>>>>>>>>>> characterize > >> > > >>>>>>>>>>>>>>> it as a "mess" though. As the points raised so far > >> > > >>>>> show, it > >> > > >>>>>>>> isn't > >> > > >>>>>>>>>>>> easy > >> > > >>>>>>>>>>>>>>> to find a solution that balances competing > >> > > >>>>>>>>>>>>>>> requirements > >> > > >>>>> and > >> > > >>>>>>>>> leads to > >> > > >>>>>>>>>>>> a > >> > > >>>>>>>>>>>>>>> net improvement. > >> > > >>>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>>> It would be great if we can find a setup that > >> > > >>>>>>>>>>>>>>> allows for > >> > > >>>>>>>>> connectors > >> > > >>>>>>>>>>>> to > >> > > >>>>>>>>>>>>>>> be released independently of core Flink and that > >> > > >>>>>>>>>>>>>>> each > >> > > >>>>>>> connector > >> > > >>>>>>>>> can > >> > > >>>>>>>>>>>> be > >> > > >>>>>>>>>>>>>>> released separately. Flink already has separate > >> > > >>>>>>>>>>>>>>> releases (flink-shaded), so that by itself isn't a > >> > > >>> new thing. > >> > > >>>>>>>>> Per-connector > >> > > >>>>>>>>>>>>>>> releases would need to allow for more frequent > >> > > >>>>>>>>>>>>>>> releases > >> > > >>>>>>>> (without > >> > > >>>>>>>>> the > >> > > >>>>>>>>>>>>>>> baggage that a full Flink release comes with). > >> > > >>>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>>> Separate releases would only make sense if the core > >> > > >>>>> Flink > >> > > >>>>>>>>> surface is > >> > > >>>>>>>>>>>>>>> fairly stable though. As evident from Iceberg (and > >> > > >>>>>>>>>>>>>>> also > >> > > >>>>>>> Beam), > >> > > >>>>>>>>> that's > >> > > >>>>>>>>>>>>>>> not the case currently. We should probably focus on > >> > > >>>>>>> addressing > >> > > >>>>>>>>> the > >> > > >>>>>>>>>>>>>>> stability first, before splitting code. A success > >> > > >>>>> criteria > >> > > >>>>>>>> could > >> > > >>>>>>>>> be > >> > > >>>>>>>>>>>>>>> that we are able to build Iceberg and Beam against > >> > > >>>>> multiple > >> > > >>>>>>>> Flink > >> > > >>>>>>>>>>>>>>> versions w/o the need to change code. The goal > >> > > >>>>>>>>>>>>>>> would be > >> > > >>>>>> that > >> > > >>>>>>> no > >> > > >>>>>>>>>>>>>>> connector breaks when we make changes to Flink core. > >> > > >>>>> Until > >> > > >>>>>>>>> that's the > >> > > >>>>>>>>>>>>>>> case, code separation creates a setup where 1+1 or > >> > > >>>>>>>>>>>>>>> N+1 > >> > > >>>>>>>>> repositories > >> > > >>>>>>>>>>>>>>> need to move lock step. > >> > > >>>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>>> Regarding some connectors being more important for > >> > > >>>>>>>>>>>>>>> Flink > >> > > >>>>>> than > >> > > >>>>>>>>> others: > >> > > >>>>>>>>>>>>>>> That's a fact. Flink w/o Kafka connector (and few > >> > > >>>>> others) > >> > > >>>>>>> isn't > >> > > >>>>>>>>>>>>>>> viable. Testability of Flink was already brought > >> > > >>>>>>>>>>>>>>> up, > >> > > >>>>> can we > >> > > >>>>>>>>> really > >> > > >>>>>>>>>>>>>>> certify a Flink core release without Kafka > >> > > >> connector? > >> > > >>>>> Maybe > >> > > >>>>>>>> those > >> > > >>>>>>>>>>>>>>> connectors that are used in Flink e2e tests to > >> > > >>>>>>>>>>>>>>> validate > >> > > >>>>>>>>> functionality > >> > > >>>>>>>>>>>>>>> of core Flink should not be broken out? > >> > > >>>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>>> Finally, I think that the connectors that move into > >> > > >>>>>> separate > >> > > >>>>>>>>> repos > >> > > >>>>>>>>>>>>>>> should remain part of the Apache Flink project. > >> > > >>>>>>>>>>>>>>> Larger > >> > > >>>>>>>>> organizations > >> > > >>>>>>>>>>>>>>> tend to approve the use of and contribution to open > >> > > >>>>> source > >> > > >>>>>> at > >> > > >>>>>>>> the > >> > > >>>>>>>>>>>>>>> project level. Sometimes it is everything ASF. More > >> > > >>>>> often > >> > > >>>>>> it > >> > > >>>>>>> is > >> > > >>>>>>>>>>>>>>> "Apache Foo". It would be fatal to end up with a > >> > > >>>>> patchwork > >> > > >>>>>> of > >> > > >>>>>>>>>>>> projects > >> > > >>>>>>>>>>>>>>> with potentially different licenses and governance > >> > > >>>>>>>>>>>>>>> to > >> > > >>>>>> arrive > >> > > >>>>>>>> at a > >> > > >>>>>>>>>>>>>>> working Flink setup. This may mean we prioritize > >> > > >>>>> usability > >> > > >>>>>>> over > >> > > >>>>>>>>>>>>>>> developer convenience, if that's in the best > >> > > >>>>>>>>>>>>>>> interest of > >> > > >>>>>>> Flink > >> > > >>>>>>>>> as a > >> > > >>>>>>>>>>>>>>> whole. > >> > > >>>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>>> Thanks, > >> > > >>>>>>>>>>>>>>> Thomas > >> > > >>>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>>> On Mon, Oct 18, 2021 at 6:59 AM Chesnay Schepler < > >> > > >>>>>>>>> ches...@apache.org > >> > > >>>>>>>>>>>>>>> wrote: > >> > > >>>>>>>>>>>>>>>> Generally, the issues are reproducibility and > >> > > >>> control. > >> > > >>>>>>>>>>>>>>>> Stuffs completely broken on the Flink side for a > >> > > >>> week? > >> > > >>>>>> Well > >> > > >>>>>>>>> then so > >> > > >>>>>>>>>>>> are > >> > > >>>>>>>>>>>>>>>> the connector repos. > >> > > >>>>>>>>>>>>>>>> (As-is) You can't go back to a previous version of > >> > > >>>>>>>>>>>>>>>> the > >> > > >>>>>>>> snapshot. > >> > > >>>>>>>>>>>> Which > >> > > >>>>>>>>>>>>>>>> also means that checking out older commits can be > >> > > >>>>>>> problematic > >> > > >>>>>>>>>>>> because > >> > > >>>>>>>>>>>>>>>> you'd still work against the latest snapshots, and > >> > > >>>>>>>>>>>>>>>> they > >> > > >>>>>> not > >> > > >>>>>>> be > >> > > >>>>>>>>>>>>>>>> compatible with each other. > >> > > >>>>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>>>> > >> > > >>>>>>>>>>>>>>>> On 18/10/2021 15:22, Arvid Heise wrote: > >> > > >>>>>>>>>>>>>>>>> I was actually betting on snapshots versions. > >> > > >>>>>>>>>>>>>>>>> What are > >> > > >>>>>> the > >> > > >>>>>>>>> limits? > >> > > >>>>>>>>>>>>>>>>> Obviously, we can only do a release of a 1.15 > >> > > >>>>> connector > >> > > >>>>>>> after > >> > > >>>>>>>>> 1.15 > >> > > >>>>>>>>>>>> is > >> > > >>>>>>>>>>>>>>>>> release. > >> > > >>>>>>>>>>>>> > >> > > >>>>>>>>>>> > >> > > >>>>>>>>>>> -- > >> > > >>>>>>>>>>> > >> > > >>>>>>>>>>> Konstantin Knauf > >> > > >>>>>>>>>>> > >> > > >>>>>>>>>>> https://urldefense.com/v3/__https://twitter.com/snntrable > >> > > >>>>>>>>>>> __;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_- > >> > > >>>>>>>>>>> XjpYgX5MUy9M4$ [twitter[.]com] > >> > > >>>>>>>>>>> > >> > > >>>>>>>>>>> https://urldefense.com/v3/__https://github.com/knaufk__;! > >> > > >>>>>>>>>>> !LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpY > >> > > >>>>>>>>>>> gXyX8u50S$ [github[.]com] > >> > > >>>>>>>>>>> > >> > > > >> > > > >> > > >> > >
