I also agree that it feels more natural to go with a repo for each individual connector. Each repository can be made available at flink-packages.org so users can find them, next to referring to them in documentation. +1 from my side.
On Thu, 9 Dec 2021 at 15:38, Arvid Heise <ar...@apache.org> wrote: > Hi all, > > We tried out Chesnay's proposal and went with Option 2. Unfortunately, we > experienced tough nuts to crack and feel like we hit a dead end: > - The main pain point with the outlined Frankensteinian connector repo is > how to handle shared code / infra code. If we have it in some <common> > branch, then we need to merge the common branch in the connector branch on > update. However, it's unclear to me how improvements in the common branch > that naturally appear while working on a specific connector go back into > the common branch. You can't use a pull request from your branch or else > your connector code would poison the connector-less common branch. So you > would probably manually copy the files over to a common branch and create a > PR branch for that. > - A weird solution could be to have the common branch as a submodule in the > repo itself (if that's even possible). I'm sure that this setup would blow > up the minds of all newcomers. > - Similarly, it's mandatory to have safeguards against code from connector > A poisoning connector B, common, or main. I had some similar setup in the > past and code from two "distinct" branch types constantly swept over. > - We could also say that we simply release <common> independently and just > have a maven (SNAPSHOT) dependency on it. But that would create a weird > flow if you need to change in common where you need to constantly switch > branches back and forth. > - In general, Frankensteinian's approach is very switch intensive. If you > maintain 3 connectors and need to fix 1 build stability each at the same > time (quite common nowadays for some reason) and you have 2 review rounds, > you need to switch branches 9 times ignoring changes to common. > > Additionally, we still have the rather user/dev unfriendly main that is > mostly empty. I'm also not sure we can generate an overview README.md to > make it more friendly here because in theory every connector branch should > be based on main and we would get merge conflicts. > > I'd like to propose once again to go with individual repositories. > - The only downside that we discussed so far is that we have more initial > setup to do. Since we organically grow the number of connector/repositories > that load is quite distributed. We can offer templates after finding a good > approach that can even be used by outside organizations. > - Regarding secrets, I think it's actually an advantage that the Kafka > connector has no access to the AWS secrets. If there are secrets to be > shared across connectors, we can and should use Azure's Variable Groups (I > have used it in the past to share Nexus creds across repos). That would > also make rotation easy. > - Working on different connectors would be rather easy as all modern IDE > support multiple repo setups in the same project. You still need to do > multiple releases in case you update common code (either accessed through > Nexus or git submodule) and you want to release your connector. > - There is no difference in respect to how many CI runs there in both > approaches. > - Individual repositories also have the advantage of allowing external > incubation. Let's assume someone builds connector A and hosts it in their > organization (very common setup). If they want to contribute the code to > Flink, we could simply transfer the repository into ASF after ensuring > Flink coding standards. Then we retain git history and Github issues. > > Is there any point that I'm missing? > > On Fri, Nov 26, 2021 at 1:32 PM Chesnay Schepler <ches...@apache.org> > wrote: > > > For sharing workflows we should be able to use composite actions. We'd > > have the main definition files in the flink-connectors repo, that we > > also need to tag/release, which other branches/repos can then import. > > These are also versioned, so we don't have to worry about accidentally > > breaking stuff. > > These could also be used to enforce certain standards / interfaces such > > that we can automate more things (e.g., integration into the Flink > > documentation). > > > > It is true that Option 2) and dedicated repositories share a lot of > > properties. While I did say in an offline conversation that we in that > > case might just as well use separate repositories, I'm not so sure > > anymore. One repo would make administration a bit easier, for example > > secrets wouldn't have to be applied to each repo (we wouldn't want > > certain secrets to be set up organization-wide). > > I overall also like that one repo would present a single access point; > > you can't "miss" a connector repo, and I would hope that having it as > > one repo would nurture more collaboration between the connectors, which > > after all need to solve similar problems. > > > > It is a fair point that the branching model would be quite weird, but I > > think that would subside pretty quickly. > > > > Personally I'd go with Option 2, and if that doesn't work out we can > > still split the repo later on. (Which should then be a trivial matter of > > copying all <connector>/* branches and renaming them). > > > > On 26/11/2021 12:47, Till Rohrmann wrote: > > > Hi Arvid, > > > > > > Thanks for updating this thread with the latest findings. The described > > > limitations for a single connector repo sound suboptimal to me. > > > > > > * Option 2. sounds as if we try to simulate multi connector repos > inside > > of > > > a single repo. I also don't know how we would share code between the > > > different branches (sharing infrastructure would probably be easier > > > though). This seems to have the same limitations as dedicated repos > with > > > the downside of having a not very intuitive branching model. > > > * Isn't option 1. kind of a degenerated version of option 2. where we > > have > > > some unrelated code from other connectors in the individual connector > > > branches? > > > * Option 3. has the downside that someone creating a release has to > > release > > > all connectors. This means that she either has to sync with the > different > > > connector maintainers or has to be able to release all connectors on > her > > > own. We are already seeing in the Flink community that releases require > > > quite good communication/coordination between the different people > > working > > > on different Flink components. Given our goals to make connector > releases > > > easier and more frequent, I think that coupling different connector > > > releases might be counter-productive. > > > > > > To me it sounds not very practical to mainly use a mono repository w/o > > > having some more advanced build infrastructure that, for example, > allows > > to > > > have different git roots in different connector directories. Maybe the > > mono > > > repo can be a catch all repository for connectors that want to be > > released > > > in lock-step (Option 3.) with all other connectors the repo contains. > But > > > for connectors that get changed frequently, having a dedicated > repository > > > that allows independent releases sounds preferable to me. > > > > > > What utilities and infrastructure code do you intend to share? Using > git > > > submodules can definitely be one option to share code. However, it > might > > > also be ok to depend on flink-connector-common artifacts which could > make > > > things easier. Where I am unsure is whether git submodules can be used > to > > > share infrastructure code (e.g. the .github/workflows) because you need > > > these files in the repo to trigger the CI infrastructure. > > > > > > Cheers, > > > Till > > > > > > On Thu, Nov 25, 2021 at 1:59 PM Arvid Heise <ar...@apache.org> wrote: > > > > > >> Hi Brian, > > >> > > >> Thank you for sharing. I think your approach is very valid and is in > > line > > >> with what I had in mind. > > >> > > >> Basically Pravega community aligns the connector releases with the > > Pravega > > >>> mainline release > > >>> > > >> This certainly would mean that there is little value in coupling > > connector > > >> versions. So it's making a good case for having separate connector > > repos. > > >> > > >> > > >>> and maintains the connector with the latest 3 Flink versions(CI will > > >>> publish snapshots for all these 3 branches) > > >>> > > >> I'd like to give connector devs a simple way to express to which Flink > > >> versions the current branch is compatible. From there we can generate > > the > > >> compatibility matrix automatically and optionally also create > different > > >> releases per supported Flink version. Not sure if the latter is indeed > > >> better than having just one artifact that happens to run with multiple > > >> Flink versions. I guess it depends on what dependencies we are > > exposing. If > > >> the connector uses flink-connector-base, then we probably need > separate > > >> artifacts with poms anyways. > > >> > > >> Best, > > >> > > >> Arvid > > >> > > >> On Fri, Nov 19, 2021 at 10:55 AM Zhou, Brian <b.z...@dell.com> wrote: > > >> > > >>> Hi Arvid, > > >>> > > >>> For branching model, the Pravega Flink connector has some experience > > what > > >>> I would like to share. Here[1][2] is the compatibility matrix and > wiki > > >>> explaining the branching model and releases. Basically Pravega > > community > > >>> aligns the connector releases with the Pravega mainline release, and > > >>> maintains the connector with the latest 3 Flink versions(CI will > > publish > > >>> snapshots for all these 3 branches). > > >>> For example, recently we have 0.10.1 release[3], and in maven central > > we > > >>> need to upload three artifacts(For Flink 1.13, 1.12, 1.11) for 0.10.1 > > >>> version[4]. > > >>> > > >>> There are some alternatives. Another solution that we once discussed > > but > > >>> finally got abandoned is to have a independent version just like the > > >>> current CDC connector, and then give a big compatibility matrix to > > users. > > >>> We think it would be too confusing when the connector develops. On > the > > >>> contrary, we can also do the opposite way to align with Flink version > > and > > >>> maintain several branches for different system version. > > >>> > > >>> I would say this is only a fairly-OK solution because it is a bit > > painful > > >>> for maintainers as cherry-picks are very common and releases would > > >> require > > >>> much work. However, if neither systems do not have a nice backward > > >>> compatibility, there seems to be no comfortable solution to the their > > >>> connector. > > >>> > > >>> [1] https://github.com/pravega/flink-connectors#compatibility-matrix > > >>> [2] > > >>> > > >> > > > https://github.com/pravega/flink-connectors/wiki/Versioning-strategy-for-Flink-connector > > >>> [3] https://github.com/pravega/flink-connectors/releases/tag/v0.10.1 > > >>> [4] https://search.maven.org/search?q=pravega-connectors-flink > > >>> > > >>> Best Regards, > > >>> Brian > > >>> > > >>> > > >>> Internal Use - Confidential > > >>> > > >>> -----Original Message----- > > >>> From: Arvid Heise <ar...@apache.org> > > >>> Sent: Friday, November 19, 2021 4:12 PM > > >>> To: dev > > >>> Subject: Re: [DISCUSS] Creating an external connector repository > > >>> > > >>> > > >>> [EXTERNAL EMAIL] > > >>> > > >>> Hi everyone, > > >>> > > >>> we are currently in the process of setting up the flink-connectors > repo > > >>> [1] for new connectors but we hit a wall that we currently cannot > take: > > >>> branching model. > > >>> To reiterate the original motivation of the external connector repo: > We > > >>> want to decouple the release cycle of a connector with Flink. > However, > > if > > >>> we want to support semantic versioning in the connectors with the > > ability > > >>> to introduce breaking changes through major version bumps and support > > >>> bugfixes on old versions, then we need release branches similar to > how > > >>> Flink core operates. > > >>> Consider two connectors, let's call them kafka and hbase. We have > kafka > > >> in > > >>> version 1.0.X, 1.1.Y (small improvement), 2.0.Z (config option) > change > > >> and > > >>> hbase only on 1.0.A. > > >>> > > >>> Now our current assumption was that we can work with a mono-repo > under > > >> ASF > > >>> (flink-connectors). Then, for release-branches, we found 3 options: > > >>> 1. We would need to create some ugly mess with the cross product of > > >>> connector and version: so you have kafka-release-1.0, > > kafka-release-1.1, > > >>> kafka-release-2.0, hbase-release-1.0. The main issue is not the > amount > > of > > >>> branches (that's something that git can handle) but there the state > of > > >>> kafka is undefined in hbase-release-1.0. That's a call for desaster > and > > >>> makes releasing connectors very cumbersome (CI would only execute and > > >>> publish hbase SNAPSHOTS on hbase-release-1.0). > > >>> 2. We could avoid the undefined state by having an empty master and > > each > > >>> release branch really only holds the code of the connector. But > that's > > >> also > > >>> not great: any user that looks at the repo and sees no connector > would > > >>> assume that it's dead. > > >>> 3. We could have synced releases similar to the CDC connectors [2]. > > That > > >>> means that if any connector introduces a breaking change, all > > connectors > > >>> get a new major. I find that quite confusing to a user if hbase gets > a > > >> new > > >>> release without any change because kafka introduced a breaking > change. > > >>> > > >>> To fully decouple release cycles and CI of connectors, we could add > > >>> individual repositories under ASF (flink-connector-kafka, > > >>> flink-connector-hbase). Then we can apply the same branching model as > > >>> before. I quickly checked if there are precedences in the apache > > >> community > > >>> for that approach and just by scanning alphabetically I found cordova > > >> with > > >>> 70 and couchdb with 77 apache repos respectively. So it certainly > seems > > >>> like other projects approached our problem in that way and the apache > > >>> organization is okay with that. I currently expect max 20 additional > > >> repos > > >>> for connectors and in the future 10 max each for formats and > > filesystems > > >> if > > >>> we would also move them out at some point in time. So we would be at > a > > >>> total of 50 repos. > > >>> > > >>> Note for all options, we need to provide a compability matrix that we > > aim > > >>> to autogenerate. > > >>> > > >>> Now for the potential downsides that we internally discussed: > > >>> - How can we ensure common infra structure code, utilties, and > quality? > > >>> I propose to add a flink-connector-common that contains all these > > things > > >>> and is added as a git submodule/subtree to the repos. > > >>> - Do we implicitly discourage connector developers to maintain more > > than > > >>> one connector with a fragmented code base? > > >>> That is certainly a risk. However, I currently also see few devs > > working > > >>> on more than one connector. However, it may actually help keeping the > > >> devs > > >>> that maintain a specific connector on the hook. We could use github > > >> issues > > >>> to track bugs and feature requests and a dev can focus his limited > time > > >> on > > >>> getting that one connector right. > > >>> > > >>> So WDYT? Compared to some intermediate suggestions with split repos, > > the > > >>> big difference is that everything remains under Apache umbrella and > the > > >>> Flink community. > > >>> > > >>> [1] > > >>> > > >> > > > https://urldefense.com/v3/__https://github.com/apache/flink-connectors__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzxxweh4$ > > >>> [github[.]com] [2] > > >>> > > >> > > > https://urldefense.com/v3/__https://github.com/ververica/flink-cdc-connectors/__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzgoPGA8$ > > >>> [github[.]com] > > >>> > > >>> On Fri, Nov 12, 2021 at 3:39 PM Arvid Heise <ar...@apache.org> > wrote: > > >>> > > >>>> Hi everyone, > > >>>> > > >>>> I created the flink-connectors repo [1] to advance the topic. We > would > > >>>> create a proof-of-concept in the next few weeks as a special branch > > >>>> that I'd then use for discussions. If the community agrees with the > > >>>> approach, that special branch will become the master. If not, we can > > >>>> reiterate over it or create competing POCs. > > >>>> > > >>>> If someone wants to try things out in parallel, just make sure that > > >>>> you are not accidentally pushing POCs to the master. > > >>>> > > >>>> As a reminder: We will not move out any current connector from Flink > > >>>> at this point in time, so everything in Flink will remain as is and > be > > >>>> maintained there. > > >>>> > > >>>> Best, > > >>>> > > >>>> Arvid > > >>>> > > >>>> [1] > > >>>> > > https://urldefense.com/v3/__https://github.com/apache/flink-connectors > > >>>> > __;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzxxweh4 > > >>>> $ [github[.]com] > > >>>> > > >>>> On Fri, Oct 29, 2021 at 6:57 PM Till Rohrmann <trohrm...@apache.org > > > > >>>> wrote: > > >>>> > > >>>>> Hi everyone, > > >>>>> > > >>>>> From the discussion, it seems to me that we have different > opinions > > >>>>> whether to have an ASF umbrella repository or to host them outside > of > > >>>>> the ASF. It also seems that this is not really the problem to > solve. > > >>>>> Since there are many good arguments for either approach, we could > > >>>>> simply start with an ASF umbrella repository and see how people > adopt > > >>>>> it. If the individual connectors cannot move fast enough or if > people > > >>>>> prefer to not buy into the more heavy-weight ASF processes, then > they > > >>>>> can host the code also somewhere else. We simply need to make sure > > >>>>> that these connectors are discoverable (e.g. via flink-packages). > > >>>>> > > >>>>> The more important problem seems to be to provide common tooling > > >>>>> (testing, infrastructure, documentation) that can easily be reused. > > >>>>> Similarly, it has become clear that the Flink community needs to > > >>>>> improve on providing stable APIs. I think it is not realistic to > > >>>>> first complete these tasks before starting to move connectors to > > >>>>> dedicated repositories. As Stephan said, creating a connector > > >>>>> repository will force us to pay more attention to API stability and > > >>>>> also to think about which testing tools are required. Hence, I > > >>>>> believe that starting to add connectors to a different repository > > >>>>> than apache/flink will help improve our connector tooling > (declaring > > >>>>> testing classes as public, creating a common test utility repo, > > >>>>> creating a repo > > >>>>> template) and vice versa. Hence, I like Arvid's proposed process as > > >>>>> it will start kicking things off w/o letting this effort fizzle > out. > > >>>>> > > >>>>> Cheers, > > >>>>> Till > > >>>>> > > >>>>> On Thu, Oct 28, 2021 at 11:44 AM Stephan Ewen <se...@apache.org> > > >> wrote: > > >>>>>> Thank you all, for the nice discussion! > > >>>>>> > > >>>>>> From my point of view, I very much like the idea of putting > > >>>>>> connectors > > >>>>> in a > > >>>>>> separate repository. But I would argue it should be part of Apache > > >>>>> Flink, > > >>>>>> similar to flink-statefun, flink-ml, etc. > > >>>>>> > > >>>>>> I share many of the reasons for that: > > >>>>>> - As argued many times, reduces complexity of the Flink repo, > > >>>>> increases > > >>>>>> response times of CI, etc. > > >>>>>> - Much lower barrier of contribution, because an unstable > > >>>>>> connector > > >>>>> would > > >>>>>> not de-stabilize the whole build. Of course, we would need to make > > >>>>>> sure > > >>>>> we > > >>>>>> set this up the right way, with connectors having individual CI > > >>>>>> runs, > > >>>>> build > > >>>>>> status, etc. But it certainly seems possible. > > >>>>>> > > >>>>>> > > >>>>>> I would argue some points a bit different than some cases made > > >> before: > > >>>>>> (a) I believe the separation would increase connector stability. > > >>>>> Because it > > >>>>>> really forces us to work with the connectors against the APIs like > > >>>>>> any external developer. A mono repo is somehow the wrong thing if > > >>>>>> you in practice want to actually guarantee stable internal APIs at > > >>> some layer. > > >>>>>> Because the mono repo makes it easy to just change something on > > >>>>>> both > > >>>>> sides > > >>>>>> of the API (provider and consumer) seamlessly. > > >>>>>> > > >>>>>> Major refactorings in Flink need to keep all connector API > > >>>>>> contracts intact, or we need to have a new version of the > connector > > >>> API. > > >>>>>> (b) We may even be able to go towards more lightweight and > > >>>>>> automated releases over time, even if we stay in Apache Flink with > > >>> that repo. > > >>>>>> This isn't yet fully aligned with the Apache release policies, > yet, > > >>>>>> but there are board discussions about whether there can be > > >>>>>> bot-triggered releases (by dependabot) and how that could fit into > > >>> the Apache process. > > >>>>>> This doesn't seem to be quite there just yet, but seeing that > those > > >>>>> start > > >>>>>> is a good sign, and there is a good chance we can do some things > > >>> there. > > >>>>>> I am not sure whether we should let bots trigger releases, because > > >>>>>> a > > >>>>> final > > >>>>>> human look at things isn't a bad thing, especially given the > > >>>>>> popularity > > >>>>> of > > >>>>>> software supply chain attacks recently. > > >>>>>> > > >>>>>> > > >>>>>> I do share Chesnay's concerns about complexity in tooling, though. > > >>>>>> Both release tooling and test tooling. They are not incompatible > > >>>>>> with that approach, but they are a task we need to tackle during > > >>>>>> this change which will add additional work. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> On Tue, Oct 26, 2021 at 10:31 AM Arvid Heise <ar...@apache.org> > > >>> wrote: > > >>>>>>> Hi folks, > > >>>>>>> > > >>>>>>> I think some questions came up and I'd like to address the > > >>>>>>> question of > > >>>>>> the > > >>>>>>> timing. > > >>>>>>> > > >>>>>>> Could you clarify what release cadence you're thinking of? > > >>>>>>> There's > > >>>>> quite > > >>>>>>>> a big range that fits "more frequent than Flink" (per-commit, > > >>>>>>>> daily, weekly, bi-weekly, monthly, even bi-monthly). > > >>>>>>> The short answer is: as often as needed: > > >>>>>>> - If there is a CVE in a dependency and we need to bump it - > > >>>>>>> release immediately. > > >>>>>>> - If there is a new feature merged, release soonish. We may > > >>>>>>> collect a > > >>>>> few > > >>>>>>> successive features before a release. > > >>>>>>> - If there is a bugfix, release immediately or soonish depending > > >>>>>>> on > > >>>>> the > > >>>>>>> severity and if there are workarounds available. > > >>>>>>> > > >>>>>>> We should not limit ourselves; the whole idea of independent > > >>>>>>> releases > > >>>>> is > > >>>>>>> exactly that you release as needed. There is no release planning > > >>>>>>> or anything needed, you just go with a release as if it was an > > >>>>>>> external artifact. > > >>>>>>> > > >>>>>>> (1) is the connector API already stable? > > >>>>>>>> From another discussion thread [1], connector API is far from > > >>>>> stable. > > >>>>>>>> Currently, it's hard to build connectors against multiple Flink > > >>>>>> versions. > > >>>>>>>> There are breaking API changes both in 1.12 -> 1.13 and 1.13 -> > > >>>>>>>> 1.14 > > >>>>>> and > > >>>>>>>> maybe also in the future versions, because Table related APIs > > >>>>>>>> are > > >>>>>> still > > >>>>>>>> @PublicEvolving and new Sink API is still @Experimental. > > >>>>>>>> > > >>>>>>> The question is: what is stable in an evolving system? We > > >>>>>>> recently discovered that the old SourceFunction needed to be > > >>>>>>> refined such that cancellation works correctly [1]. So that > > >>>>>>> interface is in Flink since > > >>>>> 7 > > >>>>>>> years, heavily used also outside, and we still had to change the > > >>>>> contract > > >>>>>>> in a way that I'd expect any implementer to recheck their > > >>>>> implementation. > > >>>>>>> It might not be necessary to change anything and you can probably > > >>>>> change > > >>>>>>> the the code for all Flink versions but still, the interface was > > >>>>>>> not > > >>>>>> stable > > >>>>>>> in the closest sense. > > >>>>>>> > > >>>>>>> If we focus just on API changes on the unified interfaces, then > > >>>>>>> we > > >>>>> expect > > >>>>>>> one more change to Sink API to support compaction. For Table API, > > >>>>> there > > >>>>>>> will most likely also be some changes in 1.15. So we could wait > > >>>>>>> for > > >>>>> 1.15. > > >>>>>>> But I'm questioning if that's really necessary because we will > > >>>>>>> add > > >>>>> more > > >>>>>>> functionality beyond 1.15 without breaking API. For example, we > > >>>>>>> may > > >>>>> add > > >>>>>>> more unified connector metrics. If you want to use it in your > > >>>>> connector, > > >>>>>>> you have to support multiple Flink versions anyhow. So rather > > >>>>>>> then > > >>>>>> focusing > > >>>>>>> the discussion on "when is stuff stable", I'd rather focus on > > >>>>>>> "how > > >>>>> can we > > >>>>>>> support building connectors against multiple Flink versions" and > > >>>>>>> make > > >>>>> it > > >>>>>> as > > >>>>>>> painless as possible. > > >>>>>>> > > >>>>>>> Chesnay pointed out to use different branches for different Flink > > >>>>>> versions > > >>>>>>> which sounds like a good suggestion. With a mono-repo, we can't > > >>>>>>> use branches differently anyways (there is no way to have release > > >>>>>>> branches > > >>>>>> per > > >>>>>>> connector without chaos). In these branches, we could provide > > >>>>>>> shims to simulate future features in older Flink versions such > > >>>>>>> that code-wise, > > >>>>> the > > >>>>>>> source code of a specific connector may not diverge (much). For > > >>>>> example, > > >>>>>> to > > >>>>>>> register unified connector metrics, we could simulate the current > > >>>>>> approach > > >>>>>>> also in some utility package of the mono-repo. > > >>>>>>> > > >>>>>>> I see the stable core Flink API as a prerequisite for modularity. > > >>>>>>> And > > >>>>>>>> for connectors it is not just the source and sink API (source > > >>>>>>>> being stable as of 1.14), but everything that is required to > > >>>>>>>> build and maintain a connector downstream, such as the test > > >>>>>>>> utilities and infrastructure. > > >>>>>>>> > > >>>>>>> That is a very fair point. I'm actually surprised to see that > > >>>>>>> MiniClusterWithClientResource is not public. I see it being used > > >>>>>>> in > > >>>>> all > > >>>>>>> connectors, especially outside of Flink. I fear that as long as > > >>>>>>> we do > > >>>>> not > > >>>>>>> have connectors outside, we will not properly annotate and > > >>>>>>> maintain > > >>>>> these > > >>>>>>> utilties in a classic hen-and-egg-problem. I will outline an idea > > >>>>>>> at > > >>>>> the > > >>>>>>> end. > > >>>>>>> > > >>>>>>>> the connectors need to be adopted and require at least one > > >>>>>>>> release > > >>>>> per > > >>>>>>>> Flink minor release. > > >>>>>>>> However, this will make the releases of connectors slower, e.g. > > >>>>>> maintain > > >>>>>>>> features for multiple branches and release multiple branches. > > >>>>>>>> I think the main purpose of having an external connector > > >>>>>>>> repository > > >>>>> is > > >>>>>> in > > >>>>>>>> order to have "faster releases of connectors"? > > >>>>>>>> > > >>>>>>>> Imagine a project with a complex set of dependencies. Let's say > > >>>>> Flink > > >>>>>>>> version A plus Flink reliant dependencies released by other > > >>>>>>>> projects (Flink-external connectors, Beam, Iceberg, Hudi, ..). > > >>>>>>>> We don't want > > >>>>> a > > >>>>>>>> situation where we bump the core Flink version to B and things > > >>>>>>>> fall apart (interface changes, utilities that were useful but > > >>>>>>>> not public, transitive dependencies etc.). > > >>>>>>>> > > >>>>>>> Yes, that's why I wanted to automate the processes more which is > > >>>>>>> not > > >>>>> that > > >>>>>>> easy under ASF. Maybe we automate the source provision across > > >>>>> supported > > >>>>>>> versions and have 1 vote thread for all versions of a connector? > > >>>>>>> > > >>>>>>> From the perspective of CDC connector maintainers, the biggest > > >>>>> advantage > > >>>>>> of > > >>>>>>>> maintaining it outside of the Flink project is that: > > >>>>>>>> 1) we can have a more flexible and faster release cycle > > >>>>>>>> 2) we can be more liberal with committership for connector > > >>>>> maintainers > > >>>>>>>> which can also attract more committers to help the release. > > >>>>>>>> > > >>>>>>>> Personally, I think maintaining one connector repository under > > >>>>>>>> the > > >>>>> ASF > > >>>>>>> may > > >>>>>>>> not have the above benefits. > > >>>>>>>> > > >>>>>>> Yes, I also feel that ASF is too restrictive for our needs. But > > >>>>>>> it > > >>>>> feels > > >>>>>>> like there are too many that see it differently and I think we > > >>>>>>> need > > >>>>>>> > > >>>>>>> (2) Flink testability without connectors. > > >>>>>>>> This is a very good question. How can we guarantee the new > > >>>>>>>> Source > > >>>>> and > > >>>>>>> Sink > > >>>>>>>> API are stable with only test implementation? > > >>>>>>>> > > >>>>>>> We can't and shouldn't. Since the connector repo is managed by > > >>>>>>> Flink, > > >>>>> a > > >>>>>>> Flink release manager needs to check if the Flink connectors are > > >>>>> actually > > >>>>>>> working prior to creating an RC. That's similar to how > > >>>>>>> flink-shaded > > >>>>> and > > >>>>>>> flink core are related. > > >>>>>>> > > >>>>>>> > > >>>>>>> So here is one idea that I had to get things rolling. We are > > >>>>>>> going to address the external repo iteratively without > > >>>>>>> compromising what we > > >>>>>> already > > >>>>>>> have: > > >>>>>>> 1.Phase, add new contributions to external repo. We use that time > > >>>>>>> to > > >>>>>> setup > > >>>>>>> infra accordingly and optimize release processes. We will > > >>>>>>> identify > > >>>>> test > > >>>>>>> utilities that are not yet public/stable and fix that. > > >>>>>>> 2.Phase, add ports to the new unified interfaces of existing > > >>>>> connectors. > > >>>>>>> That requires a previous Flink release to make utilities stable. > > >>>>>>> Keep > > >>>>> old > > >>>>>>> interfaces in flink-core. > > >>>>>>> 3.Phase, remove old interfaces in flink-core of some connectors > > >>>>>>> (tbd > > >>>>> at a > > >>>>>>> later point). > > >>>>>>> 4.Phase, optionally move all remaining connectors (tbd at a later > > >>>>> point). > > >>>>>>> I'd envision having ~3 months between the starting the different > > >>>>> phases. > > >>>>>>> WDYT? > > >>>>>>> > > >>>>>>> > > >>>>>>> [1] > > >>>>>>> > https://urldefense.com/v3/__https://issues.apache.org/jira/browse > > >>>>>>> /FLINK-23527__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgd > > >>>>>>> ke_-XjpYgX2sIvAP4$ [issues[.]apache[.]org] > > >>>>>>> > > >>>>>>> On Thu, Oct 21, 2021 at 7:12 AM Kyle Bendickson <k...@tabular.io > > > > >>>>> wrote: > > >>>>>>>> Hi all, > > >>>>>>>> > > >>>>>>>> My name is Kyle and I’m an open source developer primarily > > >>>>>>>> focused > > >>>>> on > > >>>>>>>> Apache Iceberg. > > >>>>>>>> > > >>>>>>>> I’m happy to help clarify or elaborate on any aspect of our > > >>>>> experience > > >>>>>>>> working on a relatively decoupled connector that is downstream > > >>>>>>>> and > > >>>>>> pretty > > >>>>>>>> popular. > > >>>>>>>> > > >>>>>>>> I’d also love to be able to contribute or assist in any way I > > >> can. > > >>>>>>>> I don’t mean to thread jack, but are there any meetings or > > >>>>>>>> community > > >>>>>> sync > > >>>>>>>> ups, specifically around the connector APIs, that I might join > > >>>>>>>> / be > > >>>>>>> invited > > >>>>>>>> to? > > >>>>>>>> > > >>>>>>>> I did want to add that even though I’ve experienced some of the > > >>>>>>>> pain > > >>>>>>> points > > >>>>>>>> of integrating with an evolving system / API (catalog support > > >>>>>>>> is > > >>>>>>> generally > > >>>>>>>> speaking pretty new everywhere really in this space), I also > > >>>>>>>> agree personally that you shouldn’t slow down development > > >>>>>>>> velocity too > > >>>>> much > > >>>>>> for > > >>>>>>>> the sake of external connector. Getting to a performant and > > >>>>>>>> stable > > >>>>>> place > > >>>>>>>> should be the primary goal, and slowing that down to support > > >>>>> stragglers > > >>>>>>>> will (in my personal opinion) always be a losing game. Some > > >>>>>>>> folks > > >>>>> will > > >>>>>>>> simply stay behind on versions regardless until they have to > > >>>>> upgrade. > > >>>>>>>> I am working on ensuring that the Iceberg community stays > > >>>>>>>> within 1-2 versions of Flink, so that we can help provide more > > >>>>>>>> feedback or > > >>>>>>> contribute > > >>>>>>>> things that might make our ability to support multiple Flink > > >>>>> runtimes / > > >>>>>>>> versions with one project / codebase and minimal to no > > >>>>>>>> reflection > > >>>>> (our > > >>>>>>>> desired goal). > > >>>>>>>> > > >>>>>>>> If there’s anything I can do or any way I can be of assistance, > > >>>>> please > > >>>>>>>> don’t hesitate to reach out. Or find me on ASF slack 😀 > > >>>>>>>> > > >>>>>>>> I greatly appreciate your general concern for the needs of > > >>>>> downstream > > >>>>>>>> connector integrators! > > >>>>>>>> > > >>>>>>>> Cheers > > >>>>>>>> Kyle Bendickson (GitHub: kbendick) Open Source Developer kyle > > >>>>>>>> [at] tabular [dot] io > > >>>>>>>> > > >>>>>>>> On Wed, Oct 20, 2021 at 11:35 AM Thomas Weise <t...@apache.org> > > >>>>> wrote: > > >>>>>>>>> Hi, > > >>>>>>>>> > > >>>>>>>>> I see the stable core Flink API as a prerequisite for > > >>> modularity. > > >>>>> And > > >>>>>>>>> for connectors it is not just the source and sink API (source > > >>>>> being > > >>>>>>>>> stable as of 1.14), but everything that is required to build > > >>>>>>>>> and maintain a connector downstream, such as the test > > >>>>>>>>> utilities and infrastructure. > > >>>>>>>>> > > >>>>>>>>> Without the stable surface of core Flink, changes will leak > > >>>>>>>>> into downstream dependencies and force lock step updates. > > >>>>>>>>> Refactoring across N repos is more painful than a single > > >>>>>>>>> repo. Those with experience developing downstream of Flink > > >>>>>>>>> will know the pain, and > > >>>>>> that > > >>>>>>>>> isn't limited to connectors. I don't remember a Flink "minor > > >>>>> version" > > >>>>>>>>> update that was just a dependency version change and did not > > >>>>>>>>> force other downstream changes. > > >>>>>>>>> > > >>>>>>>>> Imagine a project with a complex set of dependencies. Let's > > >>>>>>>>> say > > >>>>> Flink > > >>>>>>>>> version A plus Flink reliant dependencies released by other > > >>>>> projects > > >>>>>>>>> (Flink-external connectors, Beam, Iceberg, Hudi, ..). We > > >>>>>>>>> don't > > >>>>> want a > > >>>>>>>>> situation where we bump the core Flink version to B and > > >>>>>>>>> things > > >>>>> fall > > >>>>>>>>> apart (interface changes, utilities that were useful but not > > >>>>> public, > > >>>>>>>>> transitive dependencies etc.). > > >>>>>>>>> > > >>>>>>>>> The discussion here also highlights the benefits of keeping > > >>>>> certain > > >>>>>>>>> connectors outside Flink. Whether that is due to difference > > >>>>>>>>> in developer community, maturity of the connectors, their > > >>>>>>>>> specialized/limited usage etc. I would like to see that as a > > >>>>>>>>> sign > > >>>>> of > > >>>>>> a > > >>>>>>>>> growing ecosystem and most of the ideas that Arvid has put > > >>>>>>>>> forward would benefit further growth of the connector > > >> ecosystem. > > >>>>>>>>> As for keeping connectors within Apache Flink: I prefer that > > >>>>>>>>> as > > >>>>> the > > >>>>>>>>> path forward for "essential" connectors like FileSource, > > >>>>> KafkaSource, > > >>>>>>>>> ... And we can still achieve a more flexible and faster > > >>>>>>>>> release > > >>>>>> cycle. > > >>>>>>>>> Thanks, > > >>>>>>>>> Thomas > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> On Wed, Oct 20, 2021 at 3:32 AM Jark Wu <imj...@gmail.com> > > >>> wrote: > > >>>>>>>>>> Hi Konstantin, > > >>>>>>>>>> > > >>>>>>>>>>> the connectors need to be adopted and require at least > > >>>>>>>>>>> one > > >>>>>> release > > >>>>>>>> per > > >>>>>>>>>> Flink minor release. > > >>>>>>>>>> However, this will make the releases of connectors slower, > > >>> e.g. > > >>>>>>>> maintain > > >>>>>>>>>> features for multiple branches and release multiple > > >> branches. > > >>>>>>>>>> I think the main purpose of having an external connector > > >>>>> repository > > >>>>>>> is > > >>>>>>>> in > > >>>>>>>>>> order to have "faster releases of connectors"? > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> From the perspective of CDC connector maintainers, the > > >>>>>>>>>> biggest > > >>>>>>>> advantage > > >>>>>>>>> of > > >>>>>>>>>> maintaining it outside of the Flink project is that: > > >>>>>>>>>> 1) we can have a more flexible and faster release cycle > > >>>>>>>>>> 2) we can be more liberal with committership for connector > > >>>>>>> maintainers > > >>>>>>>>>> which can also attract more committers to help the release. > > >>>>>>>>>> > > >>>>>>>>>> Personally, I think maintaining one connector repository > > >>>>>>>>>> under > > >>>>> the > > >>>>>>> ASF > > >>>>>>>>> may > > >>>>>>>>>> not have the above benefits. > > >>>>>>>>>> > > >>>>>>>>>> Best, > > >>>>>>>>>> Jark > > >>>>>>>>>> > > >>>>>>>>>> On Wed, 20 Oct 2021 at 15:14, Konstantin Knauf < > > >>>>> kna...@apache.org> > > >>>>>>>>> wrote: > > >>>>>>>>>>> Hi everyone, > > >>>>>>>>>>> > > >>>>>>>>>>> regarding the stability of the APIs. I think everyone > > >>>>>>>>>>> agrees > > >>>>> that > > >>>>>>>>>>> connector APIs which are stable across minor versions > > >>>>>> (1.13->1.14) > > >>>>>>>> are > > >>>>>>>>> the > > >>>>>>>>>>> mid-term goal. But: > > >>>>>>>>>>> > > >>>>>>>>>>> a) These APIs are still quite young, and we shouldn't > > >>>>>>>>>>> make > > >>>>> them > > >>>>>>>> @Public > > >>>>>>>>>>> prematurely either. > > >>>>>>>>>>> > > >>>>>>>>>>> b) Isn't this *mostly* orthogonal to where the connector > > >>>>>>>>>>> code > > >>>>>>> lives? > > >>>>>>>>> Yes, > > >>>>>>>>>>> as long as there are breaking changes, the connectors > > >>>>>>>>>>> need to > > >>>>> be > > >>>>>>>>> adopted > > >>>>>>>>>>> and require at least one release per Flink minor release. > > >>>>>>>>>>> Documentation-wise this can be addressed via a > > >>>>>>>>>>> compatibility > > >>>>>> matrix > > >>>>>>>> for > > >>>>>>>>>>> each connector as Arvid suggested. IMO we shouldn't block > > >>>>>>>>>>> this > > >>>>>>> effort > > >>>>>>>>> on > > >>>>>>>>>>> the stability of the APIs. > > >>>>>>>>>>> > > >>>>>>>>>>> Cheers, > > >>>>>>>>>>> > > >>>>>>>>>>> Konstantin > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> On Wed, Oct 20, 2021 at 8:56 AM Jark Wu > > >>>>>>>>>>> <imj...@gmail.com> > > >>>>>> wrote: > > >>>>>>>>>>>> Hi, > > >>>>>>>>>>>> > > >>>>>>>>>>>> I think Thomas raised very good questions and would like > > >>>>>>>>>>>> to > > >>>>> know > > >>>>>>>> your > > >>>>>>>>>>>> opinions if we want to move connectors out of flink in > > >>>>>>>>>>>> this > > >>>>>>> version. > > >>>>>>>>>>>> (1) is the connector API already stable? > > >>>>>>>>>>>>> Separate releases would only make sense if the core > > >>>>>>>>>>>>> Flink > > >>>>>>> surface > > >>>>>>>> is > > >>>>>>>>>>>>> fairly stable though. As evident from Iceberg (and > > >>>>>>>>>>>>> also > > >>>>> Beam), > > >>>>>>>>> that's > > >>>>>>>>>>>>> not the case currently. We should probably focus on > > >>>>> addressing > > >>>>>>> the > > >>>>>>>>>>>>> stability first, before splitting code. A success > > >>>>>>>>>>>>> criteria > > >>>>>> could > > >>>>>>>> be > > >>>>>>>>>>>>> that we are able to build Iceberg and Beam against > > >>>>>>>>>>>>> multiple > > >>>>>>> Flink > > >>>>>>>>>>>>> versions w/o the need to change code. The goal would > > >>>>>>>>>>>>> be > > >>>>> that > > >>>>>> no > > >>>>>>>>>>>>> connector breaks when we make changes to Flink core. > > >>>>>>>>>>>>> Until > > >>>>>>> that's > > >>>>>>>>> the > > >>>>>>>>>>>>> case, code separation creates a setup where 1+1 or N+1 > > >>>>>>>> repositories > > >>>>>>>>>>>>> need to move lock step. > > >>>>>>>>>>>> From another discussion thread [1], connector API is far > > >>>>>>>>>>>> from > > >>>>>>>> stable. > > >>>>>>>>>>>> Currently, it's hard to build connectors against > > >>>>>>>>>>>> multiple > > >>>>> Flink > > >>>>>>>>> versions. > > >>>>>>>>>>>> There are breaking API changes both in 1.12 -> 1.13 and > > >>>>>>>>>>>> 1.13 > > >>>>> -> > > >>>>>>> 1.14 > > >>>>>>>>> and > > >>>>>>>>>>>> maybe also in the future versions, because Table > > >>>>>>>>>>>> related > > >>>>> APIs > > >>>>>>> are > > >>>>>>>>> still > > >>>>>>>>>>>> @PublicEvolving and new Sink API is still @Experimental. > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> (2) Flink testability without connectors. > > >>>>>>>>>>>>> Flink w/o Kafka connector (and few others) isn't > > >>>>>>>>>>>>> viable. Testability of Flink was already brought up, > > >>>>>>>>>>>>> can we > > >>>>>>> really > > >>>>>>>>>>>>> certify a Flink core release without Kafka connector? > > >>>>>>>>>>>>> Maybe > > >>>>>>> those > > >>>>>>>>>>>>> connectors that are used in Flink e2e tests to > > >>>>>>>>>>>>> validate > > >>>>>>>>> functionality > > >>>>>>>>>>>>> of core Flink should not be broken out? > > >>>>>>>>>>>> This is a very good question. How can we guarantee the > > >>>>>>>>>>>> new > > >>>>>> Source > > >>>>>>>> and > > >>>>>>>>> Sink > > >>>>>>>>>>>> API are stable with only test implementation? > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> Best, > > >>>>>>>>>>>> Jark > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Tue, 19 Oct 2021 at 23:56, Chesnay Schepler < > > >>>>>>> ches...@apache.org> > > >>>>>>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>>> Could you clarify what release cadence you're thinking > > >>> of? > > >>>>>>> There's > > >>>>>>>>> quite > > >>>>>>>>>>>>> a big range that fits "more frequent than Flink" > > >>>>> (per-commit, > > >>>>>>>> daily, > > >>>>>>>>>>>>> weekly, bi-weekly, monthly, even bi-monthly). > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On 19/10/2021 14:15, Martijn Visser wrote: > > >>>>>>>>>>>>>> Hi all, > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> I think it would be a huge benefit if we can achieve > > >>>>>>>>>>>>>> more > > >>>>>>>> frequent > > >>>>>>>>>>>>> releases > > >>>>>>>>>>>>>> of connectors, which are not bound to the release > > >>>>>>>>>>>>>> cycle > > >>>>> of > > >>>>>>> Flink > > >>>>>>>>>>>> itself. > > >>>>>>>>>>>>> I > > >>>>>>>>>>>>>> agree that in order to get there, we need to have > > >>>>>>>>>>>>>> stable > > >>>>>>>>> interfaces > > >>>>>>>>>>>> which > > >>>>>>>>>>>>>> are trustworthy and reliable, so they can be safely > > >>>>>>>>>>>>>> used > > >>>>> by > > >>>>>>>> those > > >>>>>>>>>>>>>> connectors. I do think that work still needs to be > > >>>>>>>>>>>>>> done > > >>>>> on > > >>>>>>> those > > >>>>>>>>>>>>>> interfaces, but I am confident that we can get there > > >>>>> from a > > >>>>>>>> Flink > > >>>>>>>>>>>>>> perspective. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> I am worried that we would not be able to achieve > > >>>>>>>>>>>>>> those > > >>>>>>> frequent > > >>>>>>>>>>>> releases > > >>>>>>>>>>>>>> of connectors if we are putting these connectors > > >>>>>>>>>>>>>> under > > >>>>> the > > >>>>>>>> Apache > > >>>>>>>>>>>>> umbrella, > > >>>>>>>>>>>>>> because that means that for each connector release > > >>>>>>>>>>>>>> we > > >>>>> have > > >>>>>> to > > >>>>>>>>> follow > > >>>>>>>>>>>> the > > >>>>>>>>>>>>>> Apache release creation process. This requires a lot > > >>>>>>>>>>>>>> of > > >>>>>> manual > > >>>>>>>>> steps > > >>>>>>>>>>>> and > > >>>>>>>>>>>>>> prohibits automation and I think it would be hard to > > >>>>> scale > > >>>>>> out > > >>>>>>>>>>>> frequent > > >>>>>>>>>>>>>> releases of connectors. I'm curious how others think > > >>>>>>>>>>>>>> this > > >>>>>>>>> challenge > > >>>>>>>>>>>> could > > >>>>>>>>>>>>>> be solved. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Best regards, > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Martijn > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> On Mon, 18 Oct 2021 at 22:22, Thomas Weise < > > >>>>> t...@apache.org> > > >>>>>>>>> wrote: > > >>>>>>>>>>>>>>> Thanks for initiating this discussion. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> There are definitely a few things that are not > > >>>>>>>>>>>>>>> optimal > > >>>>> with > > >>>>>>> our > > >>>>>>>>>>>>>>> current management of connectors. I would not > > >>>>> necessarily > > >>>>>>>>>>>> characterize > > >>>>>>>>>>>>>>> it as a "mess" though. As the points raised so far > > >>>>> show, it > > >>>>>>>> isn't > > >>>>>>>>>>>> easy > > >>>>>>>>>>>>>>> to find a solution that balances competing > > >>>>>>>>>>>>>>> requirements > > >>>>> and > > >>>>>>>>> leads to > > >>>>>>>>>>>> a > > >>>>>>>>>>>>>>> net improvement. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> It would be great if we can find a setup that > > >>>>>>>>>>>>>>> allows for > > >>>>>>>>> connectors > > >>>>>>>>>>>> to > > >>>>>>>>>>>>>>> be released independently of core Flink and that > > >>>>>>>>>>>>>>> each > > >>>>>>> connector > > >>>>>>>>> can > > >>>>>>>>>>>> be > > >>>>>>>>>>>>>>> released separately. Flink already has separate > > >>>>>>>>>>>>>>> releases (flink-shaded), so that by itself isn't a > > >>> new thing. > > >>>>>>>>> Per-connector > > >>>>>>>>>>>>>>> releases would need to allow for more frequent > > >>>>>>>>>>>>>>> releases > > >>>>>>>> (without > > >>>>>>>>> the > > >>>>>>>>>>>>>>> baggage that a full Flink release comes with). > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Separate releases would only make sense if the core > > >>>>> Flink > > >>>>>>>>> surface is > > >>>>>>>>>>>>>>> fairly stable though. As evident from Iceberg (and > > >>>>>>>>>>>>>>> also > > >>>>>>> Beam), > > >>>>>>>>> that's > > >>>>>>>>>>>>>>> not the case currently. We should probably focus on > > >>>>>>> addressing > > >>>>>>>>> the > > >>>>>>>>>>>>>>> stability first, before splitting code. A success > > >>>>> criteria > > >>>>>>>> could > > >>>>>>>>> be > > >>>>>>>>>>>>>>> that we are able to build Iceberg and Beam against > > >>>>> multiple > > >>>>>>>> Flink > > >>>>>>>>>>>>>>> versions w/o the need to change code. The goal > > >>>>>>>>>>>>>>> would be > > >>>>>> that > > >>>>>>> no > > >>>>>>>>>>>>>>> connector breaks when we make changes to Flink core. > > >>>>> Until > > >>>>>>>>> that's the > > >>>>>>>>>>>>>>> case, code separation creates a setup where 1+1 or > > >>>>>>>>>>>>>>> N+1 > > >>>>>>>>> repositories > > >>>>>>>>>>>>>>> need to move lock step. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Regarding some connectors being more important for > > >>>>>>>>>>>>>>> Flink > > >>>>>> than > > >>>>>>>>> others: > > >>>>>>>>>>>>>>> That's a fact. Flink w/o Kafka connector (and few > > >>>>> others) > > >>>>>>> isn't > > >>>>>>>>>>>>>>> viable. Testability of Flink was already brought > > >>>>>>>>>>>>>>> up, > > >>>>> can we > > >>>>>>>>> really > > >>>>>>>>>>>>>>> certify a Flink core release without Kafka > > >> connector? > > >>>>> Maybe > > >>>>>>>> those > > >>>>>>>>>>>>>>> connectors that are used in Flink e2e tests to > > >>>>>>>>>>>>>>> validate > > >>>>>>>>> functionality > > >>>>>>>>>>>>>>> of core Flink should not be broken out? > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Finally, I think that the connectors that move into > > >>>>>> separate > > >>>>>>>>> repos > > >>>>>>>>>>>>>>> should remain part of the Apache Flink project. > > >>>>>>>>>>>>>>> Larger > > >>>>>>>>> organizations > > >>>>>>>>>>>>>>> tend to approve the use of and contribution to open > > >>>>> source > > >>>>>> at > > >>>>>>>> the > > >>>>>>>>>>>>>>> project level. Sometimes it is everything ASF. More > > >>>>> often > > >>>>>> it > > >>>>>>> is > > >>>>>>>>>>>>>>> "Apache Foo". It would be fatal to end up with a > > >>>>> patchwork > > >>>>>> of > > >>>>>>>>>>>> projects > > >>>>>>>>>>>>>>> with potentially different licenses and governance > > >>>>>>>>>>>>>>> to > > >>>>>> arrive > > >>>>>>>> at a > > >>>>>>>>>>>>>>> working Flink setup. This may mean we prioritize > > >>>>> usability > > >>>>>>> over > > >>>>>>>>>>>>>>> developer convenience, if that's in the best > > >>>>>>>>>>>>>>> interest of > > >>>>>>> Flink > > >>>>>>>>> as a > > >>>>>>>>>>>>>>> whole. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>>>> Thomas > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Mon, Oct 18, 2021 at 6:59 AM Chesnay Schepler < > > >>>>>>>>> ches...@apache.org > > >>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>> Generally, the issues are reproducibility and > > >>> control. > > >>>>>>>>>>>>>>>> Stuffs completely broken on the Flink side for a > > >>> week? > > >>>>>> Well > > >>>>>>>>> then so > > >>>>>>>>>>>> are > > >>>>>>>>>>>>>>>> the connector repos. > > >>>>>>>>>>>>>>>> (As-is) You can't go back to a previous version of > > >>>>>>>>>>>>>>>> the > > >>>>>>>> snapshot. > > >>>>>>>>>>>> Which > > >>>>>>>>>>>>>>>> also means that checking out older commits can be > > >>>>>>> problematic > > >>>>>>>>>>>> because > > >>>>>>>>>>>>>>>> you'd still work against the latest snapshots, and > > >>>>>>>>>>>>>>>> they > > >>>>>> not > > >>>>>>> be > > >>>>>>>>>>>>>>>> compatible with each other. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> On 18/10/2021 15:22, Arvid Heise wrote: > > >>>>>>>>>>>>>>>>> I was actually betting on snapshots versions. > > >>>>>>>>>>>>>>>>> What are > > >>>>>> the > > >>>>>>>>> limits? > > >>>>>>>>>>>>>>>>> Obviously, we can only do a release of a 1.15 > > >>>>> connector > > >>>>>>> after > > >>>>>>>>> 1.15 > > >>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>> release. > > >>>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> -- > > >>>>>>>>>>> > > >>>>>>>>>>> Konstantin Knauf > > >>>>>>>>>>> > > >>>>>>>>>>> https://urldefense.com/v3/__https://twitter.com/snntrable > > >>>>>>>>>>> __;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_- > > >>>>>>>>>>> XjpYgX5MUy9M4$ [twitter[.]com] > > >>>>>>>>>>> > > >>>>>>>>>>> https://urldefense.com/v3/__https://github.com/knaufk__;! > > >>>>>>>>>>> !LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpY > > >>>>>>>>>>> gXyX8u50S$ [github[.]com] > > >>>>>>>>>>> > > > > >
