Good question: we want to use the same setup as we currently have for Flink, so using the existing CI infrastructure.
On Mon, 10 Jan 2022 at 11:19, Chesnay Schepler <ches...@apache.org> wrote: > What CI resources do you actually intend use? Asking since the ASF GHA > resources are afaik quite overloaded. > > On 05/01/2022 11:48, Martijn Visser wrote: > > Hi everyone, > > > > I wanted to summarise the email thread and see if there are any open > items > > that still need to be discussed, before we can finalise the discussion in > > this email thread: > > > > 1. About having multi connectors in one repo or each connector in its own > > repository > > > > As explained by @Arvid Heise <ar...@apache.org> we ultimately propose to > > have a single repository per connector, which seems to be favoured in the > > community. > > > > 2. About having the connector repositories under ASF or not. > > > > The consensus is that all connectors would remain under the ASF. > > > > I think we can categorise the questions or concerns that are brought > > forward as the following one: > > > > 3. How would we set up the testing? > > > > We need to make sure that we provide a proper testing framework, which > > means that we provide a public Source- and Sink testing framework. As > > mentioned extensively in the thread, we need to make sure that the > > necessary interfaces are properly annotated and at least @PublicEvolving. > > This also includes the test infrastructure, like MiniCluster. For the > > latter, we don't know exactly yet how to balance having publicly > available > > test infrastructure vs being able to iterate inside of Flink, but we can > > all agree this has to be solved. > > > > For testing infrastructure, we would like to use Github Actions. In the > > current state, it probably makes sense for a connector repo to follow the > > branching strategy of Flink. That will ensure a match between the > released > > connector and Flink version. This should change when all the Flink > > interfaces have stabilised so you can use a connector with multiple Flink > > versions. That means that we should have a nightly build test for: > > > > - The `main` branch of the connector (which would be the unreleased > > version) against the `master` branch of Flink (the unreleased version of > > Flink). > > - Any supported `release-X.YY` branch of the connector against the > > `release-X.YY` branch of Flink. > > > > We should also have a smoke test E2E tests in Flink (one for DataStream, > > one for Table, one for SQL, one for Python) which loads all the > connectors > > and does an arbitrary test (post data on source, load into Flink, sink > > output and compare that output is as expected. > > > > 4. How would we integrate documentation? > > > > Documentation for a connector should probably end up in the connector > > repository. The Flink website should contain one entrance to all > connectors > > (so not the current approach where we have connectors per DataStream API, > > Table API etc). Each connector documentation should end up as one menu > item > > in connectors, containing all necessary information for all DataStream, > > Table, SQL and Python implementations. > > > > 5. Which connectors should end up in the external connector repo? > > > > I'll open up a separate thread on this topic to have a parallel > discussion > > on that. We should reach consensus on both threads before we can move > > forward on this topic as a whole. > > > > Best regards, > > > > Martijn > > > > On Fri, 10 Dec 2021 at 04:47, Thomas Weise <t...@apache.org> wrote: > > > >> +1 for repo per connector from my side also > >> > >> Thanks for trying out the different approaches. > >> > >> Where would the common/infra pieces live? In a separate repository > >> with its own release? > >> > >> Thomas > >> > >> On Thu, Dec 9, 2021 at 12:42 PM Till Rohrmann <trohrm...@apache.org> > >> wrote: > >>> Sorry if I was a bit unclear. +1 for the single repo per connector > >> approach. > >>> Cheers, > >>> Till > >>> > >>> On Thu, Dec 9, 2021 at 5:41 PM Till Rohrmann <trohrm...@apache.org> > >> wrote: > >>>> +1 for the single repo approach. > >>>> > >>>> Cheers, > >>>> Till > >>>> > >>>> On Thu, Dec 9, 2021 at 3:54 PM Martijn Visser <mart...@ververica.com> > >>>> wrote: > >>>> > >>>>> I also agree that it feels more natural to go with a repo for each > >>>>> individual connector. Each repository can be made available at > >>>>> flink-packages.org so users can find them, next to referring to them > >> in > >>>>> documentation. +1 from my side. > >>>>> > >>>>> On Thu, 9 Dec 2021 at 15:38, Arvid Heise <ar...@apache.org> wrote: > >>>>> > >>>>>> Hi all, > >>>>>> > >>>>>> We tried out Chesnay's proposal and went with Option 2. > >> Unfortunately, > >>>>> we > >>>>>> experienced tough nuts to crack and feel like we hit a dead end: > >>>>>> - The main pain point with the outlined Frankensteinian connector > >> repo > >>>>> is > >>>>>> how to handle shared code / infra code. If we have it in some > >> <common> > >>>>>> branch, then we need to merge the common branch in the connector > >> branch > >>>>> on > >>>>>> update. However, it's unclear to me how improvements in the common > >>>>> branch > >>>>>> that naturally appear while working on a specific connector go back > >> into > >>>>>> the common branch. You can't use a pull request from your branch or > >> else > >>>>>> your connector code would poison the connector-less common branch. > >> So > >>>>> you > >>>>>> would probably manually copy the files over to a common branch and > >>>>> create a > >>>>>> PR branch for that. > >>>>>> - A weird solution could be to have the common branch as a > >> submodule in > >>>>> the > >>>>>> repo itself (if that's even possible). I'm sure that this setup > >> would > >>>>> blow > >>>>>> up the minds of all newcomers. > >>>>>> - Similarly, it's mandatory to have safeguards against code from > >>>>> connector > >>>>>> A poisoning connector B, common, or main. I had some similar setup > >> in > >>>>> the > >>>>>> past and code from two "distinct" branch types constantly swept > >> over. > >>>>>> - We could also say that we simply release <common> independently > >> and > >>>>> just > >>>>>> have a maven (SNAPSHOT) dependency on it. But that would create a > >> weird > >>>>>> flow if you need to change in common where you need to constantly > >> switch > >>>>>> branches back and forth. > >>>>>> - In general, Frankensteinian's approach is very switch intensive. > >> If > >>>>> you > >>>>>> maintain 3 connectors and need to fix 1 build stability each at the > >> same > >>>>>> time (quite common nowadays for some reason) and you have 2 review > >>>>> rounds, > >>>>>> you need to switch branches 9 times ignoring changes to common. > >>>>>> > >>>>>> Additionally, we still have the rather user/dev unfriendly main > >> that is > >>>>>> mostly empty. I'm also not sure we can generate an overview > >> README.md to > >>>>>> make it more friendly here because in theory every connector branch > >>>>> should > >>>>>> be based on main and we would get merge conflicts. > >>>>>> > >>>>>> I'd like to propose once again to go with individual repositories. > >>>>>> - The only downside that we discussed so far is that we have more > >>>>> initial > >>>>>> setup to do. Since we organically grow the number of > >>>>> connector/repositories > >>>>>> that load is quite distributed. We can offer templates after > >> finding a > >>>>> good > >>>>>> approach that can even be used by outside organizations. > >>>>>> - Regarding secrets, I think it's actually an advantage that the > >> Kafka > >>>>>> connector has no access to the AWS secrets. If there are secrets to > >> be > >>>>>> shared across connectors, we can and should use Azure's Variable > >> Groups > >>>>> (I > >>>>>> have used it in the past to share Nexus creds across repos). That > >> would > >>>>>> also make rotation easy. > >>>>>> - Working on different connectors would be rather easy as all > >> modern IDE > >>>>>> support multiple repo setups in the same project. You still need to > >> do > >>>>>> multiple releases in case you update common code (either accessed > >>>>> through > >>>>>> Nexus or git submodule) and you want to release your connector. > >>>>>> - There is no difference in respect to how many CI runs there in > >> both > >>>>>> approaches. > >>>>>> - Individual repositories also have the advantage of allowing > >> external > >>>>>> incubation. Let's assume someone builds connector A and hosts it in > >>>>> their > >>>>>> organization (very common setup). If they want to contribute the > >> code to > >>>>>> Flink, we could simply transfer the repository into ASF after > >> ensuring > >>>>>> Flink coding standards. Then we retain git history and Github > >> issues. > >>>>>> Is there any point that I'm missing? > >>>>>> > >>>>>> On Fri, Nov 26, 2021 at 1:32 PM Chesnay Schepler < > >> ches...@apache.org> > >>>>>> wrote: > >>>>>> > >>>>>>> For sharing workflows we should be able to use composite actions. > >> We'd > >>>>>>> have the main definition files in the flink-connectors repo, that > >> we > >>>>>>> also need to tag/release, which other branches/repos can then > >> import. > >>>>>>> These are also versioned, so we don't have to worry about > >> accidentally > >>>>>>> breaking stuff. > >>>>>>> These could also be used to enforce certain standards / interfaces > >>>>> such > >>>>>>> that we can automate more things (e.g., integration into the Flink > >>>>>>> documentation). > >>>>>>> > >>>>>>> It is true that Option 2) and dedicated repositories share a lot > >> of > >>>>>>> properties. While I did say in an offline conversation that we in > >> that > >>>>>>> case might just as well use separate repositories, I'm not so sure > >>>>>>> anymore. One repo would make administration a bit easier, for > >> example > >>>>>>> secrets wouldn't have to be applied to each repo (we wouldn't want > >>>>>>> certain secrets to be set up organization-wide). > >>>>>>> I overall also like that one repo would present a single access > >> point; > >>>>>>> you can't "miss" a connector repo, and I would hope that having > >> it as > >>>>>>> one repo would nurture more collaboration between the connectors, > >>>>> which > >>>>>>> after all need to solve similar problems. > >>>>>>> > >>>>>>> It is a fair point that the branching model would be quite weird, > >> but > >>>>> I > >>>>>>> think that would subside pretty quickly. > >>>>>>> > >>>>>>> Personally I'd go with Option 2, and if that doesn't work out we > >> can > >>>>>>> still split the repo later on. (Which should then be a trivial > >> matter > >>>>> of > >>>>>>> copying all <connector>/* branches and renaming them). > >>>>>>> > >>>>>>> On 26/11/2021 12:47, Till Rohrmann wrote: > >>>>>>>> Hi Arvid, > >>>>>>>> > >>>>>>>> Thanks for updating this thread with the latest findings. The > >>>>> described > >>>>>>>> limitations for a single connector repo sound suboptimal to me. > >>>>>>>> > >>>>>>>> * Option 2. sounds as if we try to simulate multi connector > >> repos > >>>>>> inside > >>>>>>> of > >>>>>>>> a single repo. I also don't know how we would share code > >> between the > >>>>>>>> different branches (sharing infrastructure would probably be > >> easier > >>>>>>>> though). This seems to have the same limitations as dedicated > >> repos > >>>>>> with > >>>>>>>> the downside of having a not very intuitive branching model. > >>>>>>>> * Isn't option 1. kind of a degenerated version of option 2. > >> where > >>>>> we > >>>>>>> have > >>>>>>>> some unrelated code from other connectors in the individual > >>>>> connector > >>>>>>>> branches? > >>>>>>>> * Option 3. has the downside that someone creating a release > >> has to > >>>>>>> release > >>>>>>>> all connectors. This means that she either has to sync with the > >>>>>> different > >>>>>>>> connector maintainers or has to be able to release all > >> connectors on > >>>>>> her > >>>>>>>> own. We are already seeing in the Flink community that releases > >>>>> require > >>>>>>>> quite good communication/coordination between the different > >> people > >>>>>>> working > >>>>>>>> on different Flink components. Given our goals to make connector > >>>>>> releases > >>>>>>>> easier and more frequent, I think that coupling different > >> connector > >>>>>>>> releases might be counter-productive. > >>>>>>>> > >>>>>>>> To me it sounds not very practical to mainly use a mono > >> repository > >>>>> w/o > >>>>>>>> having some more advanced build infrastructure that, for > >> example, > >>>>>> allows > >>>>>>> to > >>>>>>>> have different git roots in different connector directories. > >> Maybe > >>>>> the > >>>>>>> mono > >>>>>>>> repo can be a catch all repository for connectors that want to > >> be > >>>>>>> released > >>>>>>>> in lock-step (Option 3.) with all other connectors the repo > >>>>> contains. > >>>>>> But > >>>>>>>> for connectors that get changed frequently, having a dedicated > >>>>>> repository > >>>>>>>> that allows independent releases sounds preferable to me. > >>>>>>>> > >>>>>>>> What utilities and infrastructure code do you intend to share? > >> Using > >>>>>> git > >>>>>>>> submodules can definitely be one option to share code. However, > >> it > >>>>>> might > >>>>>>>> also be ok to depend on flink-connector-common artifacts which > >> could > >>>>>> make > >>>>>>>> things easier. Where I am unsure is whether git submodules can > >> be > >>>>> used > >>>>>> to > >>>>>>>> share infrastructure code (e.g. the .github/workflows) because > >> you > >>>>> need > >>>>>>>> these files in the repo to trigger the CI infrastructure. > >>>>>>>> > >>>>>>>> Cheers, > >>>>>>>> Till > >>>>>>>> > >>>>>>>> On Thu, Nov 25, 2021 at 1:59 PM Arvid Heise <ar...@apache.org> > >>>>> wrote: > >>>>>>>>> Hi Brian, > >>>>>>>>> > >>>>>>>>> Thank you for sharing. I think your approach is very valid and > >> is > >>>>> in > >>>>>>> line > >>>>>>>>> with what I had in mind. > >>>>>>>>> > >>>>>>>>> Basically Pravega community aligns the connector releases with > >> the > >>>>>>> Pravega > >>>>>>>>>> mainline release > >>>>>>>>>> > >>>>>>>>> This certainly would mean that there is little value in > >> coupling > >>>>>>> connector > >>>>>>>>> versions. So it's making a good case for having separate > >> connector > >>>>>>> repos. > >>>>>>>>> > >>>>>>>>>> and maintains the connector with the latest 3 Flink > >> versions(CI > >>>>> will > >>>>>>>>>> publish snapshots for all these 3 branches) > >>>>>>>>>> > >>>>>>>>> I'd like to give connector devs a simple way to express to > >> which > >>>>> Flink > >>>>>>>>> versions the current branch is compatible. From there we can > >>>>> generate > >>>>>>> the > >>>>>>>>> compatibility matrix automatically and optionally also create > >>>>>> different > >>>>>>>>> releases per supported Flink version. Not sure if the latter is > >>>>> indeed > >>>>>>>>> better than having just one artifact that happens to run with > >>>>> multiple > >>>>>>>>> Flink versions. I guess it depends on what dependencies we are > >>>>>>> exposing. If > >>>>>>>>> the connector uses flink-connector-base, then we probably need > >>>>>> separate > >>>>>>>>> artifacts with poms anyways. > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> > >>>>>>>>> Arvid > >>>>>>>>> > >>>>>>>>> On Fri, Nov 19, 2021 at 10:55 AM Zhou, Brian <b.z...@dell.com> > >>>>> wrote: > >>>>>>>>>> Hi Arvid, > >>>>>>>>>> > >>>>>>>>>> For branching model, the Pravega Flink connector has some > >>>>> experience > >>>>>>> what > >>>>>>>>>> I would like to share. Here[1][2] is the compatibility matrix > >> and > >>>>>> wiki > >>>>>>>>>> explaining the branching model and releases. Basically Pravega > >>>>>>> community > >>>>>>>>>> aligns the connector releases with the Pravega mainline > >> release, > >>>>> and > >>>>>>>>>> maintains the connector with the latest 3 Flink versions(CI > >> will > >>>>>>> publish > >>>>>>>>>> snapshots for all these 3 branches). > >>>>>>>>>> For example, recently we have 0.10.1 release[3], and in maven > >>>>> central > >>>>>>> we > >>>>>>>>>> need to upload three artifacts(For Flink 1.13, 1.12, 1.11) for > >>>>> 0.10.1 > >>>>>>>>>> version[4]. > >>>>>>>>>> > >>>>>>>>>> There are some alternatives. Another solution that we once > >>>>> discussed > >>>>>>> but > >>>>>>>>>> finally got abandoned is to have a independent version just > >> like > >>>>> the > >>>>>>>>>> current CDC connector, and then give a big compatibility > >> matrix to > >>>>>>> users. > >>>>>>>>>> We think it would be too confusing when the connector > >> develops. On > >>>>>> the > >>>>>>>>>> contrary, we can also do the opposite way to align with Flink > >>>>> version > >>>>>>> and > >>>>>>>>>> maintain several branches for different system version. > >>>>>>>>>> > >>>>>>>>>> I would say this is only a fairly-OK solution because it is a > >> bit > >>>>>>> painful > >>>>>>>>>> for maintainers as cherry-picks are very common and releases > >> would > >>>>>>>>> require > >>>>>>>>>> much work. However, if neither systems do not have a nice > >> backward > >>>>>>>>>> compatibility, there seems to be no comfortable solution to > >> the > >>>>> their > >>>>>>>>>> connector. > >>>>>>>>>> > >>>>>>>>>> [1] > >>>>> https://github.com/pravega/flink-connectors#compatibility-matrix > >>>>>>>>>> [2] > >>>>>>>>>> > >> > https://github.com/pravega/flink-connectors/wiki/Versioning-strategy-for-Flink-connector > >>>>>>>>>> [3] > >>>>> https://github.com/pravega/flink-connectors/releases/tag/v0.10.1 > >>>>>>>>>> [4] > >> https://search.maven.org/search?q=pravega-connectors-flink > >>>>>>>>>> Best Regards, > >>>>>>>>>> Brian > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Internal Use - Confidential > >>>>>>>>>> > >>>>>>>>>> -----Original Message----- > >>>>>>>>>> From: Arvid Heise <ar...@apache.org> > >>>>>>>>>> Sent: Friday, November 19, 2021 4:12 PM > >>>>>>>>>> To: dev > >>>>>>>>>> Subject: Re: [DISCUSS] Creating an external connector > >> repository > >>>>>>>>>> > >>>>>>>>>> [EXTERNAL EMAIL] > >>>>>>>>>> > >>>>>>>>>> Hi everyone, > >>>>>>>>>> > >>>>>>>>>> we are currently in the process of setting up the > >> flink-connectors > >>>>>> repo > >>>>>>>>>> [1] for new connectors but we hit a wall that we currently > >> cannot > >>>>>> take: > >>>>>>>>>> branching model. > >>>>>>>>>> To reiterate the original motivation of the external connector > >>>>> repo: > >>>>>> We > >>>>>>>>>> want to decouple the release cycle of a connector with Flink. > >>>>>> However, > >>>>>>> if > >>>>>>>>>> we want to support semantic versioning in the connectors with > >> the > >>>>>>> ability > >>>>>>>>>> to introduce breaking changes through major version bumps and > >>>>> support > >>>>>>>>>> bugfixes on old versions, then we need release branches > >> similar to > >>>>>> how > >>>>>>>>>> Flink core operates. > >>>>>>>>>> Consider two connectors, let's call them kafka and hbase. We > >> have > >>>>>> kafka > >>>>>>>>> in > >>>>>>>>>> version 1.0.X, 1.1.Y (small improvement), 2.0.Z (config > >> option) > >>>>>> change > >>>>>>>>> and > >>>>>>>>>> hbase only on 1.0.A. > >>>>>>>>>> > >>>>>>>>>> Now our current assumption was that we can work with a > >> mono-repo > >>>>>> under > >>>>>>>>> ASF > >>>>>>>>>> (flink-connectors). Then, for release-branches, we found 3 > >>>>> options: > >>>>>>>>>> 1. We would need to create some ugly mess with the cross > >> product > >>>>> of > >>>>>>>>>> connector and version: so you have kafka-release-1.0, > >>>>>>> kafka-release-1.1, > >>>>>>>>>> kafka-release-2.0, hbase-release-1.0. The main issue is not > >> the > >>>>>> amount > >>>>>>> of > >>>>>>>>>> branches (that's something that git can handle) but there the > >>>>> state > >>>>>> of > >>>>>>>>>> kafka is undefined in hbase-release-1.0. That's a call for > >>>>> desaster > >>>>>> and > >>>>>>>>>> makes releasing connectors very cumbersome (CI would only > >> execute > >>>>> and > >>>>>>>>>> publish hbase SNAPSHOTS on hbase-release-1.0). > >>>>>>>>>> 2. We could avoid the undefined state by having an empty > >> master > >>>>> and > >>>>>>> each > >>>>>>>>>> release branch really only holds the code of the connector. > >> But > >>>>>> that's > >>>>>>>>> also > >>>>>>>>>> not great: any user that looks at the repo and sees no > >> connector > >>>>>> would > >>>>>>>>>> assume that it's dead. > >>>>>>>>>> 3. We could have synced releases similar to the CDC connectors > >>>>> [2]. > >>>>>>> That > >>>>>>>>>> means that if any connector introduces a breaking change, all > >>>>>>> connectors > >>>>>>>>>> get a new major. I find that quite confusing to a user if > >> hbase > >>>>> gets > >>>>>> a > >>>>>>>>> new > >>>>>>>>>> release without any change because kafka introduced a breaking > >>>>>> change. > >>>>>>>>>> To fully decouple release cycles and CI of connectors, we > >> could > >>>>> add > >>>>>>>>>> individual repositories under ASF (flink-connector-kafka, > >>>>>>>>>> flink-connector-hbase). Then we can apply the same branching > >>>>> model as > >>>>>>>>>> before. I quickly checked if there are precedences in the > >> apache > >>>>>>>>> community > >>>>>>>>>> for that approach and just by scanning alphabetically I found > >>>>> cordova > >>>>>>>>> with > >>>>>>>>>> 70 and couchdb with 77 apache repos respectively. So it > >> certainly > >>>>>> seems > >>>>>>>>>> like other projects approached our problem in that way and the > >>>>> apache > >>>>>>>>>> organization is okay with that. I currently expect max 20 > >>>>> additional > >>>>>>>>> repos > >>>>>>>>>> for connectors and in the future 10 max each for formats and > >>>>>>> filesystems > >>>>>>>>> if > >>>>>>>>>> we would also move them out at some point in time. So we > >> would be > >>>>> at > >>>>>> a > >>>>>>>>>> total of 50 repos. > >>>>>>>>>> > >>>>>>>>>> Note for all options, we need to provide a compability matrix > >>>>> that we > >>>>>>> aim > >>>>>>>>>> to autogenerate. > >>>>>>>>>> > >>>>>>>>>> Now for the potential downsides that we internally discussed: > >>>>>>>>>> - How can we ensure common infra structure code, utilties, and > >>>>>> quality? > >>>>>>>>>> I propose to add a flink-connector-common that contains all > >> these > >>>>>>> things > >>>>>>>>>> and is added as a git submodule/subtree to the repos. > >>>>>>>>>> - Do we implicitly discourage connector developers to maintain > >>>>> more > >>>>>>> than > >>>>>>>>>> one connector with a fragmented code base? > >>>>>>>>>> That is certainly a risk. However, I currently also see few > >> devs > >>>>>>> working > >>>>>>>>>> on more than one connector. However, it may actually help > >> keeping > >>>>> the > >>>>>>>>> devs > >>>>>>>>>> that maintain a specific connector on the hook. We could use > >>>>> github > >>>>>>>>> issues > >>>>>>>>>> to track bugs and feature requests and a dev can focus his > >> limited > >>>>>> time > >>>>>>>>> on > >>>>>>>>>> getting that one connector right. > >>>>>>>>>> > >>>>>>>>>> So WDYT? Compared to some intermediate suggestions with split > >>>>> repos, > >>>>>>> the > >>>>>>>>>> big difference is that everything remains under Apache > >> umbrella > >>>>> and > >>>>>> the > >>>>>>>>>> Flink community. > >>>>>>>>>> > >>>>>>>>>> [1] > >>>>>>>>>> > >> > https://urldefense.com/v3/__https://github.com/apache/flink-connectors__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzxxweh4$ > >>>>>>>>>> [github[.]com] [2] > >>>>>>>>>> > >> > https://urldefense.com/v3/__https://github.com/ververica/flink-cdc-connectors/__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzgoPGA8$ > >>>>>>>>>> [github[.]com] > >>>>>>>>>> > >>>>>>>>>> On Fri, Nov 12, 2021 at 3:39 PM Arvid Heise <ar...@apache.org > >>>>>> wrote: > >>>>>>>>>>> Hi everyone, > >>>>>>>>>>> > >>>>>>>>>>> I created the flink-connectors repo [1] to advance the > >> topic. We > >>>>>> would > >>>>>>>>>>> create a proof-of-concept in the next few weeks as a special > >>>>> branch > >>>>>>>>>>> that I'd then use for discussions. If the community agrees > >> with > >>>>> the > >>>>>>>>>>> approach, that special branch will become the master. If > >> not, we > >>>>> can > >>>>>>>>>>> reiterate over it or create competing POCs. > >>>>>>>>>>> > >>>>>>>>>>> If someone wants to try things out in parallel, just make > >> sure > >>>>> that > >>>>>>>>>>> you are not accidentally pushing POCs to the master. > >>>>>>>>>>> > >>>>>>>>>>> As a reminder: We will not move out any current connector > >> from > >>>>> Flink > >>>>>>>>>>> at this point in time, so everything in Flink will remain as > >> is > >>>>> and > >>>>>> be > >>>>>>>>>>> maintained there. > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> > >>>>>>>>>>> Arvid > >>>>>>>>>>> > >>>>>>>>>>> [1] > >>>>>>>>>>> > >> https://urldefense.com/v3/__https://github.com/apache/flink-connectors > >> __;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzxxweh4 > >>>>>>>>>>> $ [github[.]com] > >>>>>>>>>>> > >>>>>>>>>>> On Fri, Oct 29, 2021 at 6:57 PM Till Rohrmann < > >>>>> trohrm...@apache.org > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi everyone, > >>>>>>>>>>>> > >>>>>>>>>>>> From the discussion, it seems to me that we have different > >>>>>> opinions > >>>>>>>>>>>> whether to have an ASF umbrella repository or to host them > >>>>> outside > >>>>>> of > >>>>>>>>>>>> the ASF. It also seems that this is not really the problem > >> to > >>>>>> solve. > >>>>>>>>>>>> Since there are many good arguments for either approach, we > >>>>> could > >>>>>>>>>>>> simply start with an ASF umbrella repository and see how > >> people > >>>>>> adopt > >>>>>>>>>>>> it. If the individual connectors cannot move fast enough or > >> if > >>>>>> people > >>>>>>>>>>>> prefer to not buy into the more heavy-weight ASF processes, > >> then > >>>>>> they > >>>>>>>>>>>> can host the code also somewhere else. We simply need to > >> make > >>>>> sure > >>>>>>>>>>>> that these connectors are discoverable (e.g. via > >>>>> flink-packages). > >>>>>>>>>>>> The more important problem seems to be to provide common > >> tooling > >>>>>>>>>>>> (testing, infrastructure, documentation) that can easily be > >>>>> reused. > >>>>>>>>>>>> Similarly, it has become clear that the Flink community > >> needs to > >>>>>>>>>>>> improve on providing stable APIs. I think it is not > >> realistic to > >>>>>>>>>>>> first complete these tasks before starting to move > >> connectors to > >>>>>>>>>>>> dedicated repositories. As Stephan said, creating a > >> connector > >>>>>>>>>>>> repository will force us to pay more attention to API > >> stability > >>>>> and > >>>>>>>>>>>> also to think about which testing tools are required. > >> Hence, I > >>>>>>>>>>>> believe that starting to add connectors to a different > >>>>> repository > >>>>>>>>>>>> than apache/flink will help improve our connector tooling > >>>>>> (declaring > >>>>>>>>>>>> testing classes as public, creating a common test utility > >> repo, > >>>>>>>>>>>> creating a repo > >>>>>>>>>>>> template) and vice versa. Hence, I like Arvid's proposed > >>>>> process as > >>>>>>>>>>>> it will start kicking things off w/o letting this effort > >> fizzle > >>>>>> out. > >>>>>>>>>>>> Cheers, > >>>>>>>>>>>> Till > >>>>>>>>>>>> > >>>>>>>>>>>> On Thu, Oct 28, 2021 at 11:44 AM Stephan Ewen < > >> se...@apache.org > >>>>>>>>> wrote: > >>>>>>>>>>>>> Thank you all, for the nice discussion! > >>>>>>>>>>>>> > >>>>>>>>>>>>> From my point of view, I very much like the idea of > >> putting > >>>>>>>>>>>>> connectors > >>>>>>>>>>>> in a > >>>>>>>>>>>>> separate repository. But I would argue it should be part of > >>>>> Apache > >>>>>>>>>>>> Flink, > >>>>>>>>>>>>> similar to flink-statefun, flink-ml, etc. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I share many of the reasons for that: > >>>>>>>>>>>>> - As argued many times, reduces complexity of the Flink > >>>>> repo, > >>>>>>>>>>>> increases > >>>>>>>>>>>>> response times of CI, etc. > >>>>>>>>>>>>> - Much lower barrier of contribution, because an > >> unstable > >>>>>>>>>>>>> connector > >>>>>>>>>>>> would > >>>>>>>>>>>>> not de-stabilize the whole build. Of course, we would need > >> to > >>>>> make > >>>>>>>>>>>>> sure > >>>>>>>>>>>> we > >>>>>>>>>>>>> set this up the right way, with connectors having > >> individual CI > >>>>>>>>>>>>> runs, > >>>>>>>>>>>> build > >>>>>>>>>>>>> status, etc. But it certainly seems possible. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> I would argue some points a bit different than some cases > >> made > >>>>>>>>> before: > >>>>>>>>>>>>> (a) I believe the separation would increase connector > >>>>> stability. > >>>>>>>>>>>> Because it > >>>>>>>>>>>>> really forces us to work with the connectors against the > >> APIs > >>>>> like > >>>>>>>>>>>>> any external developer. A mono repo is somehow the wrong > >> thing > >>>>> if > >>>>>>>>>>>>> you in practice want to actually guarantee stable internal > >>>>> APIs at > >>>>>>>>>> some layer. > >>>>>>>>>>>>> Because the mono repo makes it easy to just change > >> something on > >>>>>>>>>>>>> both > >>>>>>>>>>>> sides > >>>>>>>>>>>>> of the API (provider and consumer) seamlessly. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Major refactorings in Flink need to keep all connector API > >>>>>>>>>>>>> contracts intact, or we need to have a new version of the > >>>>>> connector > >>>>>>>>>> API. > >>>>>>>>>>>>> (b) We may even be able to go towards more lightweight and > >>>>>>>>>>>>> automated releases over time, even if we stay in Apache > >> Flink > >>>>> with > >>>>>>>>>> that repo. > >>>>>>>>>>>>> This isn't yet fully aligned with the Apache release > >> policies, > >>>>>> yet, > >>>>>>>>>>>>> but there are board discussions about whether there can be > >>>>>>>>>>>>> bot-triggered releases (by dependabot) and how that could > >> fit > >>>>> into > >>>>>>>>>> the Apache process. > >>>>>>>>>>>>> This doesn't seem to be quite there just yet, but seeing > >> that > >>>>>> those > >>>>>>>>>>>> start > >>>>>>>>>>>>> is a good sign, and there is a good chance we can do some > >>>>> things > >>>>>>>>>> there. > >>>>>>>>>>>>> I am not sure whether we should let bots trigger releases, > >>>>> because > >>>>>>>>>>>>> a > >>>>>>>>>>>> final > >>>>>>>>>>>>> human look at things isn't a bad thing, especially given > >> the > >>>>>>>>>>>>> popularity > >>>>>>>>>>>> of > >>>>>>>>>>>>> software supply chain attacks recently. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> I do share Chesnay's concerns about complexity in tooling, > >>>>> though. > >>>>>>>>>>>>> Both release tooling and test tooling. They are not > >>>>> incompatible > >>>>>>>>>>>>> with that approach, but they are a task we need to tackle > >>>>> during > >>>>>>>>>>>>> this change which will add additional work. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Tue, Oct 26, 2021 at 10:31 AM Arvid Heise < > >> ar...@apache.org > >>>>>>>>>> wrote: > >>>>>>>>>>>>>> Hi folks, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I think some questions came up and I'd like to address the > >>>>>>>>>>>>>> question of > >>>>>>>>>>>>> the > >>>>>>>>>>>>>> timing. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Could you clarify what release cadence you're thinking of? > >>>>>>>>>>>>>> There's > >>>>>>>>>>>> quite > >>>>>>>>>>>>>>> a big range that fits "more frequent than Flink" > >> (per-commit, > >>>>>>>>>>>>>>> daily, weekly, bi-weekly, monthly, even bi-monthly). > >>>>>>>>>>>>>> The short answer is: as often as needed: > >>>>>>>>>>>>>> - If there is a CVE in a dependency and we need to bump > >> it - > >>>>>>>>>>>>>> release immediately. > >>>>>>>>>>>>>> - If there is a new feature merged, release soonish. We > >> may > >>>>>>>>>>>>>> collect a > >>>>>>>>>>>> few > >>>>>>>>>>>>>> successive features before a release. > >>>>>>>>>>>>>> - If there is a bugfix, release immediately or soonish > >>>>> depending > >>>>>>>>>>>>>> on > >>>>>>>>>>>> the > >>>>>>>>>>>>>> severity and if there are workarounds available. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> We should not limit ourselves; the whole idea of > >> independent > >>>>>>>>>>>>>> releases > >>>>>>>>>>>> is > >>>>>>>>>>>>>> exactly that you release as needed. There is no release > >>>>> planning > >>>>>>>>>>>>>> or anything needed, you just go with a release as if it > >> was an > >>>>>>>>>>>>>> external artifact. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> (1) is the connector API already stable? > >>>>>>>>>>>>>>> From another discussion thread [1], connector API is far > >>>>> from > >>>>>>>>>>>> stable. > >>>>>>>>>>>>>>> Currently, it's hard to build connectors against multiple > >>>>> Flink > >>>>>>>>>>>>> versions. > >>>>>>>>>>>>>>> There are breaking API changes both in 1.12 -> 1.13 and > >> 1.13 > >>>>> -> > >>>>>>>>>>>>>>> 1.14 > >>>>>>>>>>>>> and > >>>>>>>>>>>>>>> maybe also in the future versions, because Table > >> related > >>>>> APIs > >>>>>>>>>>>>>>> are > >>>>>>>>>>>>> still > >>>>>>>>>>>>>>> @PublicEvolving and new Sink API is still @Experimental. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> The question is: what is stable in an evolving system? We > >>>>>>>>>>>>>> recently discovered that the old SourceFunction needed to > >> be > >>>>>>>>>>>>>> refined such that cancellation works correctly [1]. So > >> that > >>>>>>>>>>>>>> interface is in Flink since > >>>>>>>>>>>> 7 > >>>>>>>>>>>>>> years, heavily used also outside, and we still had to > >> change > >>>>> the > >>>>>>>>>>>> contract > >>>>>>>>>>>>>> in a way that I'd expect any implementer to recheck their > >>>>>>>>>>>> implementation. > >>>>>>>>>>>>>> It might not be necessary to change anything and you can > >>>>> probably > >>>>>>>>>>>> change > >>>>>>>>>>>>>> the the code for all Flink versions but still, the > >> interface > >>>>> was > >>>>>>>>>>>>>> not > >>>>>>>>>>>>> stable > >>>>>>>>>>>>>> in the closest sense. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> If we focus just on API changes on the unified interfaces, > >>>>> then > >>>>>>>>>>>>>> we > >>>>>>>>>>>> expect > >>>>>>>>>>>>>> one more change to Sink API to support compaction. For > >> Table > >>>>> API, > >>>>>>>>>>>> there > >>>>>>>>>>>>>> will most likely also be some changes in 1.15. So we could > >>>>> wait > >>>>>>>>>>>>>> for > >>>>>>>>>>>> 1.15. > >>>>>>>>>>>>>> But I'm questioning if that's really necessary because we > >> will > >>>>>>>>>>>>>> add > >>>>>>>>>>>> more > >>>>>>>>>>>>>> functionality beyond 1.15 without breaking API. For > >> example, > >>>>> we > >>>>>>>>>>>>>> may > >>>>>>>>>>>> add > >>>>>>>>>>>>>> more unified connector metrics. If you want to use it in > >> your > >>>>>>>>>>>> connector, > >>>>>>>>>>>>>> you have to support multiple Flink versions anyhow. So > >> rather > >>>>>>>>>>>>>> then > >>>>>>>>>>>>> focusing > >>>>>>>>>>>>>> the discussion on "when is stuff stable", I'd rather > >> focus on > >>>>>>>>>>>>>> "how > >>>>>>>>>>>> can we > >>>>>>>>>>>>>> support building connectors against multiple Flink > >> versions" > >>>>> and > >>>>>>>>>>>>>> make > >>>>>>>>>>>> it > >>>>>>>>>>>>> as > >>>>>>>>>>>>>> painless as possible. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Chesnay pointed out to use different branches for > >> different > >>>>> Flink > >>>>>>>>>>>>> versions > >>>>>>>>>>>>>> which sounds like a good suggestion. With a mono-repo, we > >>>>> can't > >>>>>>>>>>>>>> use branches differently anyways (there is no way to have > >>>>> release > >>>>>>>>>>>>>> branches > >>>>>>>>>>>>> per > >>>>>>>>>>>>>> connector without chaos). In these branches, we could > >> provide > >>>>>>>>>>>>>> shims to simulate future features in older Flink versions > >> such > >>>>>>>>>>>>>> that code-wise, > >>>>>>>>>>>> the > >>>>>>>>>>>>>> source code of a specific connector may not diverge > >> (much). > >>>>> For > >>>>>>>>>>>> example, > >>>>>>>>>>>>> to > >>>>>>>>>>>>>> register unified connector metrics, we could simulate the > >>>>> current > >>>>>>>>>>>>> approach > >>>>>>>>>>>>>> also in some utility package of the mono-repo. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I see the stable core Flink API as a prerequisite for > >>>>> modularity. > >>>>>>>>>>>>>> And > >>>>>>>>>>>>>>> for connectors it is not just the source and sink API > >> (source > >>>>>>>>>>>>>>> being stable as of 1.14), but everything that is > >> required to > >>>>>>>>>>>>>>> build and maintain a connector downstream, such as the > >> test > >>>>>>>>>>>>>>> utilities and infrastructure. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> That is a very fair point. I'm actually surprised to see > >> that > >>>>>>>>>>>>>> MiniClusterWithClientResource is not public. I see it > >> being > >>>>> used > >>>>>>>>>>>>>> in > >>>>>>>>>>>> all > >>>>>>>>>>>>>> connectors, especially outside of Flink. I fear that as > >> long > >>>>> as > >>>>>>>>>>>>>> we do > >>>>>>>>>>>> not > >>>>>>>>>>>>>> have connectors outside, we will not properly annotate and > >>>>>>>>>>>>>> maintain > >>>>>>>>>>>> these > >>>>>>>>>>>>>> utilties in a classic hen-and-egg-problem. I will outline > >> an > >>>>> idea > >>>>>>>>>>>>>> at > >>>>>>>>>>>> the > >>>>>>>>>>>>>> end. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> the connectors need to be adopted and require at least > >> one > >>>>>>>>>>>>>>> release > >>>>>>>>>>>> per > >>>>>>>>>>>>>>> Flink minor release. > >>>>>>>>>>>>>>> However, this will make the releases of connectors > >> slower, > >>>>> e.g. > >>>>>>>>>>>>> maintain > >>>>>>>>>>>>>>> features for multiple branches and release multiple > >> branches. > >>>>>>>>>>>>>>> I think the main purpose of having an external connector > >>>>>>>>>>>>>>> repository > >>>>>>>>>>>> is > >>>>>>>>>>>>> in > >>>>>>>>>>>>>>> order to have "faster releases of connectors"? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Imagine a project with a complex set of dependencies. > >> Let's > >>>>> say > >>>>>>>>>>>> Flink > >>>>>>>>>>>>>>> version A plus Flink reliant dependencies released by > >> other > >>>>>>>>>>>>>>> projects (Flink-external connectors, Beam, Iceberg, Hudi, > >>>>> ..). > >>>>>>>>>>>>>>> We don't want > >>>>>>>>>>>> a > >>>>>>>>>>>>>>> situation where we bump the core Flink version to B and > >>>>> things > >>>>>>>>>>>>>>> fall apart (interface changes, utilities that were > >> useful but > >>>>>>>>>>>>>>> not public, transitive dependencies etc.). > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> Yes, that's why I wanted to automate the processes more > >> which > >>>>> is > >>>>>>>>>>>>>> not > >>>>>>>>>>>> that > >>>>>>>>>>>>>> easy under ASF. Maybe we automate the source provision > >> across > >>>>>>>>>>>> supported > >>>>>>>>>>>>>> versions and have 1 vote thread for all versions of a > >>>>> connector? > >>>>>>>>>>>>>> From the perspective of CDC connector maintainers, the > >>>>> biggest > >>>>>>>>>>>> advantage > >>>>>>>>>>>>> of > >>>>>>>>>>>>>>> maintaining it outside of the Flink project is that: > >>>>>>>>>>>>>>> 1) we can have a more flexible and faster release cycle > >>>>>>>>>>>>>>> 2) we can be more liberal with committership for > >> connector > >>>>>>>>>>>> maintainers > >>>>>>>>>>>>>>> which can also attract more committers to help the > >> release. > >>>>>>>>>>>>>>> Personally, I think maintaining one connector repository > >>>>> under > >>>>>>>>>>>>>>> the > >>>>>>>>>>>> ASF > >>>>>>>>>>>>>> may > >>>>>>>>>>>>>>> not have the above benefits. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> Yes, I also feel that ASF is too restrictive for our > >> needs. > >>>>> But > >>>>>>>>>>>>>> it > >>>>>>>>>>>> feels > >>>>>>>>>>>>>> like there are too many that see it differently and I > >> think we > >>>>>>>>>>>>>> need > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> (2) Flink testability without connectors. > >>>>>>>>>>>>>>> This is a very good question. How can we guarantee the > >> new > >>>>>>>>>>>>>>> Source > >>>>>>>>>>>> and > >>>>>>>>>>>>>> Sink > >>>>>>>>>>>>>>> API are stable with only test implementation? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> We can't and shouldn't. Since the connector repo is > >> managed by > >>>>>>>>>>>>>> Flink, > >>>>>>>>>>>> a > >>>>>>>>>>>>>> Flink release manager needs to check if the Flink > >> connectors > >>>>> are > >>>>>>>>>>>> actually > >>>>>>>>>>>>>> working prior to creating an RC. That's similar to how > >>>>>>>>>>>>>> flink-shaded > >>>>>>>>>>>> and > >>>>>>>>>>>>>> flink core are related. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> So here is one idea that I had to get things rolling. We > >> are > >>>>>>>>>>>>>> going to address the external repo iteratively without > >>>>>>>>>>>>>> compromising what we > >>>>>>>>>>>>> already > >>>>>>>>>>>>>> have: > >>>>>>>>>>>>>> 1.Phase, add new contributions to external repo. We use > >> that > >>>>> time > >>>>>>>>>>>>>> to > >>>>>>>>>>>>> setup > >>>>>>>>>>>>>> infra accordingly and optimize release processes. We will > >>>>>>>>>>>>>> identify > >>>>>>>>>>>> test > >>>>>>>>>>>>>> utilities that are not yet public/stable and fix that. > >>>>>>>>>>>>>> 2.Phase, add ports to the new unified interfaces of > >> existing > >>>>>>>>>>>> connectors. > >>>>>>>>>>>>>> That requires a previous Flink release to make utilities > >>>>> stable. > >>>>>>>>>>>>>> Keep > >>>>>>>>>>>> old > >>>>>>>>>>>>>> interfaces in flink-core. > >>>>>>>>>>>>>> 3.Phase, remove old interfaces in flink-core of some > >>>>> connectors > >>>>>>>>>>>>>> (tbd > >>>>>>>>>>>> at a > >>>>>>>>>>>>>> later point). > >>>>>>>>>>>>>> 4.Phase, optionally move all remaining connectors (tbd at > >> a > >>>>> later > >>>>>>>>>>>> point). > >>>>>>>>>>>>>> I'd envision having ~3 months between the starting the > >>>>> different > >>>>>>>>>>>> phases. > >>>>>>>>>>>>>> WDYT? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>> > >>>>>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse > >>>>> /FLINK-23527__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgd > >>>>>>>>>>>>>> ke_-XjpYgX2sIvAP4$ [issues[.]apache[.]org] > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Thu, Oct 21, 2021 at 7:12 AM Kyle Bendickson < > >>>>> k...@tabular.io > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> My name is Kyle and I’m an open source developer > >> primarily > >>>>>>>>>>>>>>> focused > >>>>>>>>>>>> on > >>>>>>>>>>>>>>> Apache Iceberg. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I’m happy to help clarify or elaborate on any aspect of > >> our > >>>>>>>>>>>> experience > >>>>>>>>>>>>>>> working on a relatively decoupled connector that is > >>>>> downstream > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>> pretty > >>>>>>>>>>>>>>> popular. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I’d also love to be able to contribute or assist in any > >> way I > >>>>>>>>> can. > >>>>>>>>>>>>>>> I don’t mean to thread jack, but are there any meetings > >> or > >>>>>>>>>>>>>>> community > >>>>>>>>>>>>> sync > >>>>>>>>>>>>>>> ups, specifically around the connector APIs, that I might > >>>>> join > >>>>>>>>>>>>>>> / be > >>>>>>>>>>>>>> invited > >>>>>>>>>>>>>>> to? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I did want to add that even though I’ve experienced some > >> of > >>>>> the > >>>>>>>>>>>>>>> pain > >>>>>>>>>>>>>> points > >>>>>>>>>>>>>>> of integrating with an evolving system / API (catalog > >> support > >>>>>>>>>>>>>>> is > >>>>>>>>>>>>>> generally > >>>>>>>>>>>>>>> speaking pretty new everywhere really in this space), I > >> also > >>>>>>>>>>>>>>> agree personally that you shouldn’t slow down development > >>>>>>>>>>>>>>> velocity too > >>>>>>>>>>>> much > >>>>>>>>>>>>> for > >>>>>>>>>>>>>>> the sake of external connector. Getting to a performant > >> and > >>>>>>>>>>>>>>> stable > >>>>>>>>>>>>> place > >>>>>>>>>>>>>>> should be the primary goal, and slowing that down to > >> support > >>>>>>>>>>>> stragglers > >>>>>>>>>>>>>>> will (in my personal opinion) always be a losing game. > >> Some > >>>>>>>>>>>>>>> folks > >>>>>>>>>>>> will > >>>>>>>>>>>>>>> simply stay behind on versions regardless until they > >> have to > >>>>>>>>>>>> upgrade. > >>>>>>>>>>>>>>> I am working on ensuring that the Iceberg community stays > >>>>>>>>>>>>>>> within 1-2 versions of Flink, so that we can help provide > >>>>> more > >>>>>>>>>>>>>>> feedback or > >>>>>>>>>>>>>> contribute > >>>>>>>>>>>>>>> things that might make our ability to support multiple > >> Flink > >>>>>>>>>>>> runtimes / > >>>>>>>>>>>>>>> versions with one project / codebase and minimal to no > >>>>>>>>>>>>>>> reflection > >>>>>>>>>>>> (our > >>>>>>>>>>>>>>> desired goal). > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> If there’s anything I can do or any way I can be of > >>>>> assistance, > >>>>>>>>>>>> please > >>>>>>>>>>>>>>> don’t hesitate to reach out. Or find me on ASF slack 😀 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I greatly appreciate your general concern for the needs > >> of > >>>>>>>>>>>> downstream > >>>>>>>>>>>>>>> connector integrators! > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Cheers > >>>>>>>>>>>>>>> Kyle Bendickson (GitHub: kbendick) Open Source Developer > >> kyle > >>>>>>>>>>>>>>> [at] tabular [dot] io > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Wed, Oct 20, 2021 at 11:35 AM Thomas Weise < > >>>>> t...@apache.org> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I see the stable core Flink API as a prerequisite for > >>>>>>>>>> modularity. > >>>>>>>>>>>> And > >>>>>>>>>>>>>>>> for connectors it is not just the source and sink API > >>>>> (source > >>>>>>>>>>>> being > >>>>>>>>>>>>>>>> stable as of 1.14), but everything that is required to > >> build > >>>>>>>>>>>>>>>> and maintain a connector downstream, such as the test > >>>>>>>>>>>>>>>> utilities and infrastructure. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Without the stable surface of core Flink, changes will > >> leak > >>>>>>>>>>>>>>>> into downstream dependencies and force lock step > >> updates. > >>>>>>>>>>>>>>>> Refactoring across N repos is more painful than a single > >>>>>>>>>>>>>>>> repo. Those with experience developing downstream of > >> Flink > >>>>>>>>>>>>>>>> will know the pain, and > >>>>>>>>>>>>> that > >>>>>>>>>>>>>>>> isn't limited to connectors. I don't remember a Flink > >> "minor > >>>>>>>>>>>> version" > >>>>>>>>>>>>>>>> update that was just a dependency version change and > >> did not > >>>>>>>>>>>>>>>> force other downstream changes. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Imagine a project with a complex set of dependencies. > >> Let's > >>>>>>>>>>>>>>>> say > >>>>>>>>>>>> Flink > >>>>>>>>>>>>>>>> version A plus Flink reliant dependencies released by > >> other > >>>>>>>>>>>> projects > >>>>>>>>>>>>>>>> (Flink-external connectors, Beam, Iceberg, Hudi, ..). We > >>>>>>>>>>>>>>>> don't > >>>>>>>>>>>> want a > >>>>>>>>>>>>>>>> situation where we bump the core Flink version to B and > >>>>>>>>>>>>>>>> things > >>>>>>>>>>>> fall > >>>>>>>>>>>>>>>> apart (interface changes, utilities that were useful > >> but not > >>>>>>>>>>>> public, > >>>>>>>>>>>>>>>> transitive dependencies etc.). > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> The discussion here also highlights the benefits of > >> keeping > >>>>>>>>>>>> certain > >>>>>>>>>>>>>>>> connectors outside Flink. Whether that is due to > >> difference > >>>>>>>>>>>>>>>> in developer community, maturity of the connectors, > >> their > >>>>>>>>>>>>>>>> specialized/limited usage etc. I would like to see that > >> as a > >>>>>>>>>>>>>>>> sign > >>>>>>>>>>>> of > >>>>>>>>>>>>> a > >>>>>>>>>>>>>>>> growing ecosystem and most of the ideas that Arvid has > >> put > >>>>>>>>>>>>>>>> forward would benefit further growth of the connector > >>>>>>>>> ecosystem. > >>>>>>>>>>>>>>>> As for keeping connectors within Apache Flink: I prefer > >> that > >>>>>>>>>>>>>>>> as > >>>>>>>>>>>> the > >>>>>>>>>>>>>>>> path forward for "essential" connectors like FileSource, > >>>>>>>>>>>> KafkaSource, > >>>>>>>>>>>>>>>> ... And we can still achieve a more flexible and faster > >>>>>>>>>>>>>>>> release > >>>>>>>>>>>>> cycle. > >>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>> Thomas > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Wed, Oct 20, 2021 at 3:32 AM Jark Wu < > >> imj...@gmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> Hi Konstantin, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> the connectors need to be adopted and require at least > >>>>>>>>>>>>>>>>>> one > >>>>>>>>>>>>> release > >>>>>>>>>>>>>>> per > >>>>>>>>>>>>>>>>> Flink minor release. > >>>>>>>>>>>>>>>>> However, this will make the releases of connectors > >> slower, > >>>>>>>>>> e.g. > >>>>>>>>>>>>>>> maintain > >>>>>>>>>>>>>>>>> features for multiple branches and release multiple > >>>>>>>>> branches. > >>>>>>>>>>>>>>>>> I think the main purpose of having an external > >> connector > >>>>>>>>>>>> repository > >>>>>>>>>>>>>> is > >>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>> order to have "faster releases of connectors"? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> From the perspective of CDC connector maintainers, the > >>>>>>>>>>>>>>>>> biggest > >>>>>>>>>>>>>>> advantage > >>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>> maintaining it outside of the Flink project is that: > >>>>>>>>>>>>>>>>> 1) we can have a more flexible and faster release cycle > >>>>>>>>>>>>>>>>> 2) we can be more liberal with committership for > >> connector > >>>>>>>>>>>>>> maintainers > >>>>>>>>>>>>>>>>> which can also attract more committers to help the > >> release. > >>>>>>>>>>>>>>>>> Personally, I think maintaining one connector > >> repository > >>>>>>>>>>>>>>>>> under > >>>>>>>>>>>> the > >>>>>>>>>>>>>> ASF > >>>>>>>>>>>>>>>> may > >>>>>>>>>>>>>>>>> not have the above benefits. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>> Jark > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Wed, 20 Oct 2021 at 15:14, Konstantin Knauf < > >>>>>>>>>>>> kna...@apache.org> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> Hi everyone, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> regarding the stability of the APIs. I think everyone > >>>>>>>>>>>>>>>>>> agrees > >>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>> connector APIs which are stable across minor versions > >>>>>>>>>>>>> (1.13->1.14) > >>>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> mid-term goal. But: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> a) These APIs are still quite young, and we shouldn't > >>>>>>>>>>>>>>>>>> make > >>>>>>>>>>>> them > >>>>>>>>>>>>>>> @Public > >>>>>>>>>>>>>>>>>> prematurely either. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> b) Isn't this *mostly* orthogonal to where the > >> connector > >>>>>>>>>>>>>>>>>> code > >>>>>>>>>>>>>> lives? > >>>>>>>>>>>>>>>> Yes, > >>>>>>>>>>>>>>>>>> as long as there are breaking changes, the connectors > >>>>>>>>>>>>>>>>>> need to > >>>>>>>>>>>> be > >>>>>>>>>>>>>>>> adopted > >>>>>>>>>>>>>>>>>> and require at least one release per Flink minor > >> release. > >>>>>>>>>>>>>>>>>> Documentation-wise this can be addressed via a > >>>>>>>>>>>>>>>>>> compatibility > >>>>>>>>>>>>> matrix > >>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>> each connector as Arvid suggested. IMO we shouldn't > >> block > >>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>> effort > >>>>>>>>>>>>>>>> on > >>>>>>>>>>>>>>>>>> the stability of the APIs. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Konstantin > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Wed, Oct 20, 2021 at 8:56 AM Jark Wu > >>>>>>>>>>>>>>>>>> <imj...@gmail.com> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I think Thomas raised very good questions and would > >> like > >>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>> know > >>>>>>>>>>>>>>> your > >>>>>>>>>>>>>>>>>>> opinions if we want to move connectors out of flink > >> in > >>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>> version. > >>>>>>>>>>>>>>>>>>> (1) is the connector API already stable? > >>>>>>>>>>>>>>>>>>>> Separate releases would only make sense if the core > >>>>>>>>>>>>>>>>>>>> Flink > >>>>>>>>>>>>>> surface > >>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>> fairly stable though. As evident from Iceberg (and > >>>>>>>>>>>>>>>>>>>> also > >>>>>>>>>>>> Beam), > >>>>>>>>>>>>>>>> that's > >>>>>>>>>>>>>>>>>>>> not the case currently. We should probably focus on > >>>>>>>>>>>> addressing > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>> stability first, before splitting code. A success > >>>>>>>>>>>>>>>>>>>> criteria > >>>>>>>>>>>>> could > >>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>> that we are able to build Iceberg and Beam against > >>>>>>>>>>>>>>>>>>>> multiple > >>>>>>>>>>>>>> Flink > >>>>>>>>>>>>>>>>>>>> versions w/o the need to change code. The goal would > >>>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>> that > >>>>>>>>>>>>> no > >>>>>>>>>>>>>>>>>>>> connector breaks when we make changes to Flink core. > >>>>>>>>>>>>>>>>>>>> Until > >>>>>>>>>>>>>> that's > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>> case, code separation creates a setup where 1+1 or > >> N+1 > >>>>>>>>>>>>>>> repositories > >>>>>>>>>>>>>>>>>>>> need to move lock step. > >>>>>>>>>>>>>>>>>>> From another discussion thread [1], connector API > >> is far > >>>>>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>> stable. > >>>>>>>>>>>>>>>>>>> Currently, it's hard to build connectors against > >>>>>>>>>>>>>>>>>>> multiple > >>>>>>>>>>>> Flink > >>>>>>>>>>>>>>>> versions. > >>>>>>>>>>>>>>>>>>> There are breaking API changes both in 1.12 -> 1.13 > >> and > >>>>>>>>>>>>>>>>>>> 1.13 > >>>>>>>>>>>> -> > >>>>>>>>>>>>>> 1.14 > >>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>> maybe also in the future versions, because Table > >>>>>>>>>>>>>>>>>>> related > >>>>>>>>>>>> APIs > >>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>> still > >>>>>>>>>>>>>>>>>>> @PublicEvolving and new Sink API is still > >> @Experimental. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> (2) Flink testability without connectors. > >>>>>>>>>>>>>>>>>>>> Flink w/o Kafka connector (and few others) isn't > >>>>>>>>>>>>>>>>>>>> viable. Testability of Flink was already brought up, > >>>>>>>>>>>>>>>>>>>> can we > >>>>>>>>>>>>>> really > >>>>>>>>>>>>>>>>>>>> certify a Flink core release without Kafka > >> connector? > >>>>>>>>>>>>>>>>>>>> Maybe > >>>>>>>>>>>>>> those > >>>>>>>>>>>>>>>>>>>> connectors that are used in Flink e2e tests to > >>>>>>>>>>>>>>>>>>>> validate > >>>>>>>>>>>>>>>> functionality > >>>>>>>>>>>>>>>>>>>> of core Flink should not be broken out? > >>>>>>>>>>>>>>>>>>> This is a very good question. How can we guarantee > >> the > >>>>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>> Source > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>> Sink > >>>>>>>>>>>>>>>>>>> API are stable with only test implementation? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>> Jark > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Tue, 19 Oct 2021 at 23:56, Chesnay Schepler < > >>>>>>>>>>>>>> ches...@apache.org> > >>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Could you clarify what release cadence you're > >> thinking > >>>>>>>>>> of? > >>>>>>>>>>>>>> There's > >>>>>>>>>>>>>>>> quite > >>>>>>>>>>>>>>>>>>>> a big range that fits "more frequent than Flink" > >>>>>>>>>>>> (per-commit, > >>>>>>>>>>>>>>> daily, > >>>>>>>>>>>>>>>>>>>> weekly, bi-weekly, monthly, even bi-monthly). > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On 19/10/2021 14:15, Martijn Visser wrote: > >>>>>>>>>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> I think it would be a huge benefit if we can > >> achieve > >>>>>>>>>>>>>>>>>>>>> more > >>>>>>>>>>>>>>> frequent > >>>>>>>>>>>>>>>>>>>> releases > >>>>>>>>>>>>>>>>>>>>> of connectors, which are not bound to the release > >>>>>>>>>>>>>>>>>>>>> cycle > >>>>>>>>>>>> of > >>>>>>>>>>>>>> Flink > >>>>>>>>>>>>>>>>>>> itself. > >>>>>>>>>>>>>>>>>>>> I > >>>>>>>>>>>>>>>>>>>>> agree that in order to get there, we need to have > >>>>>>>>>>>>>>>>>>>>> stable > >>>>>>>>>>>>>>>> interfaces > >>>>>>>>>>>>>>>>>>> which > >>>>>>>>>>>>>>>>>>>>> are trustworthy and reliable, so they can be safely > >>>>>>>>>>>>>>>>>>>>> used > >>>>>>>>>>>> by > >>>>>>>>>>>>>>> those > >>>>>>>>>>>>>>>>>>>>> connectors. I do think that work still needs to be > >>>>>>>>>>>>>>>>>>>>> done > >>>>>>>>>>>> on > >>>>>>>>>>>>>> those > >>>>>>>>>>>>>>>>>>>>> interfaces, but I am confident that we can get > >> there > >>>>>>>>>>>> from a > >>>>>>>>>>>>>>> Flink > >>>>>>>>>>>>>>>>>>>>> perspective. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> I am worried that we would not be able to achieve > >>>>>>>>>>>>>>>>>>>>> those > >>>>>>>>>>>>>> frequent > >>>>>>>>>>>>>>>>>>> releases > >>>>>>>>>>>>>>>>>>>>> of connectors if we are putting these connectors > >>>>>>>>>>>>>>>>>>>>> under > >>>>>>>>>>>> the > >>>>>>>>>>>>>>> Apache > >>>>>>>>>>>>>>>>>>>> umbrella, > >>>>>>>>>>>>>>>>>>>>> because that means that for each connector release > >>>>>>>>>>>>>>>>>>>>> we > >>>>>>>>>>>> have > >>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> follow > >>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>> Apache release creation process. This requires a > >> lot > >>>>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>> manual > >>>>>>>>>>>>>>>> steps > >>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>> prohibits automation and I think it would be hard > >> to > >>>>>>>>>>>> scale > >>>>>>>>>>>>> out > >>>>>>>>>>>>>>>>>>> frequent > >>>>>>>>>>>>>>>>>>>>> releases of connectors. I'm curious how others > >> think > >>>>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>> challenge > >>>>>>>>>>>>>>>>>>> could > >>>>>>>>>>>>>>>>>>>>> be solved. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Best regards, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Martijn > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> On Mon, 18 Oct 2021 at 22:22, Thomas Weise < > >>>>>>>>>>>> t...@apache.org> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>> Thanks for initiating this discussion. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> There are definitely a few things that are not > >>>>>>>>>>>>>>>>>>>>>> optimal > >>>>>>>>>>>> with > >>>>>>>>>>>>>> our > >>>>>>>>>>>>>>>>>>>>>> current management of connectors. I would not > >>>>>>>>>>>> necessarily > >>>>>>>>>>>>>>>>>>> characterize > >>>>>>>>>>>>>>>>>>>>>> it as a "mess" though. As the points raised so far > >>>>>>>>>>>> show, it > >>>>>>>>>>>>>>> isn't > >>>>>>>>>>>>>>>>>>> easy > >>>>>>>>>>>>>>>>>>>>>> to find a solution that balances competing > >>>>>>>>>>>>>>>>>>>>>> requirements > >>>>>>>>>>>> and > >>>>>>>>>>>>>>>> leads to > >>>>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>> net improvement. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> It would be great if we can find a setup that > >>>>>>>>>>>>>>>>>>>>>> allows for > >>>>>>>>>>>>>>>> connectors > >>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>> be released independently of core Flink and that > >>>>>>>>>>>>>>>>>>>>>> each > >>>>>>>>>>>>>> connector > >>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>> released separately. Flink already has separate > >>>>>>>>>>>>>>>>>>>>> releases (flink-shaded), so that by itself isn't a > >>>>>>>>>> new thing. > >>>>>>>>>>>>>>>> Per-connector > >>>>>>>>>>>>>>>>>>>>>> releases would need to allow for more frequent > >>>>>>>>>>>>>>>>>>>>>> releases > >>>>>>>>>>>>>>> (without > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> baggage that a full Flink release comes with). > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Separate releases would only make sense if the core > >>>>>>>>>>>> Flink > >>>>>>>>>>>>>>>> surface is > >>>>>>>>>>>>>>>>>>>>>> fairly stable though. As evident from Iceberg (and > >>>>>>>>>>>>>>>>>>>>>> also > >>>>>>>>>>>>>> Beam), > >>>>>>>>>>>>>>>> that's > >>>>>>>>>>>>>>>>>>>>>> not the case currently. We should probably focus > >> on > >>>>>>>>>>>>>> addressing > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> stability first, before splitting code. A success > >>>>>>>>>>>> criteria > >>>>>>>>>>>>>>> could > >>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>> that we are able to build Iceberg and Beam against > >>>>>>>>>>>> multiple > >>>>>>>>>>>>>>> Flink > >>>>>>>>>>>>>>>>>>>>>> versions w/o the need to change code. The goal > >>>>>>>>>>>>>>>>>>>>>> would be > >>>>>>>>>>>>> that > >>>>>>>>>>>>>> no > >>>>>>>>>>>>>>>>>>>>>> connector breaks when we make changes to Flink > >> core. > >>>>>>>>>>>> Until > >>>>>>>>>>>>>>>> that's the > >>>>>>>>>>>>>>>>>>>>>> case, code separation creates a setup where 1+1 or > >>>>>>>>>>>>>>>>>>>>>> N+1 > >>>>>>>>>>>>>>>> repositories > >>>>>>>>>>>>>>>>>>>>>> need to move lock step. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Regarding some connectors being more important for > >>>>>>>>>>>>>>>>>>>>>> Flink > >>>>>>>>>>>>> than > >>>>>>>>>>>>>>>> others: > >>>>>>>>>>>>>>>>>>>>> That's a fact. Flink w/o Kafka connector (and few > >>>>>>>>>>>> others) > >>>>>>>>>>>>>> isn't > >>>>>>>>>>>>>>>>>>>>>> viable. Testability of Flink was already brought > >>>>>>>>>>>>>>>>>>>>>> up, > >>>>>>>>>>>> can we > >>>>>>>>>>>>>>>> really > >>>>>>>>>>>>>>>>>>>>>> certify a Flink core release without Kafka > >>>>>>>>> connector? > >>>>>>>>>>>> Maybe > >>>>>>>>>>>>>>> those > >>>>>>>>>>>>>>>>>>>>>> connectors that are used in Flink e2e tests to > >>>>>>>>>>>>>>>>>>>>>> validate > >>>>>>>>>>>>>>>> functionality > >>>>>>>>>>>>>>>>>>>>>> of core Flink should not be broken out? > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Finally, I think that the connectors that move > >> into > >>>>>>>>>>>>> separate > >>>>>>>>>>>>>>>> repos > >>>>>>>>>>>>>>>>>>>>>> should remain part of the Apache Flink project. > >>>>>>>>>>>>>>>>>>>>>> Larger > >>>>>>>>>>>>>>>> organizations > >>>>>>>>>>>>>>>>>>>>>> tend to approve the use of and contribution to > >> open > >>>>>>>>>>>> source > >>>>>>>>>>>>> at > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> project level. Sometimes it is everything ASF. > >> More > >>>>>>>>>>>> often > >>>>>>>>>>>>> it > >>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>> "Apache Foo". It would be fatal to end up with a > >>>>>>>>>>>> patchwork > >>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>> projects > >>>>>>>>>>>>>>>>>>>>>> with potentially different licenses and governance > >>>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>> arrive > >>>>>>>>>>>>>>> at a > >>>>>>>>>>>>>>>>>>>>>> working Flink setup. This may mean we prioritize > >>>>>>>>>>>> usability > >>>>>>>>>>>>>> over > >>>>>>>>>>>>>>>>>>>>>> developer convenience, if that's in the best > >>>>>>>>>>>>>>>>>>>>>> interest of > >>>>>>>>>>>>>> Flink > >>>>>>>>>>>>>>>> as a > >>>>>>>>>>>>>>>>>>>>>> whole. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>> Thomas > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 18, 2021 at 6:59 AM Chesnay Schepler < > >>>>>>>>>>>>>>>> ches...@apache.org > >>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>> Generally, the issues are reproducibility and > >>>>>>>>>> control. > >>>>>>>>>>>>>>>>>>>>>>> Stuffs completely broken on the Flink side for a > >>>>>>>>>> week? > >>>>>>>>>>>>> Well > >>>>>>>>>>>>>>>> then so > >>>>>>>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>>>>>>>>> the connector repos. > >>>>>>>>>>>>>>>>>>>>>>> (As-is) You can't go back to a previous version > >> of > >>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>> snapshot. > >>>>>>>>>>>>>>>>>>> Which > >>>>>>>>>>>>>>>>>>>>>>> also means that checking out older commits can be > >>>>>>>>>>>>>> problematic > >>>>>>>>>>>>>>>>>>> because > >>>>>>>>>>>>>>>>>>>>>>> you'd still work against the latest snapshots, > >> and > >>>>>>>>>>>>>>>>>>>>>>> they > >>>>>>>>>>>>> not > >>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>> compatible with each other. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> On 18/10/2021 15:22, Arvid Heise wrote: > >>>>>>>>>>>>>>>>>>>>>>>> I was actually betting on snapshots versions. > >>>>>>>>>>>>>>>>>>>>>>>> What are > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> limits? > >>>>>>>>>>>>>>>>>>>>>>>> Obviously, we can only do a release of a 1.15 > >>>>>>>>>>>> connector > >>>>>>>>>>>>>> after > >>>>>>>>>>>>>>>> 1.15 > >>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>>> release. > >>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Konstantin Knauf > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >> https://urldefense.com/v3/__https://twitter.com/snntrable > >> __;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_- > >>>>>>>>>>>>>>>>>> XjpYgX5MUy9M4$ [twitter[.]com] > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >> https://urldefense.com/v3/__https://github.com/knaufk__;! > >> !LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpY > >>>>>>>>>>>>>>>>>> gXyX8u50S$ [github[.]com] > >>>>>>>>>>>>>>>>>> > >>>>>>> > >
