Thanks for clarifying my story Konstantin, this was indeed as we discussed. Thanks Chesnay and Thomas for your input too!
On Thu, 24 Feb 2022 at 02:43, Thomas Weise <t...@apache.org> wrote: > This plan LGTM. > > Thanks, > Thomas > > On Wed, Feb 23, 2022 at 4:28 AM Chesnay Schepler <ches...@apache.org> > wrote: > > > > That sounds fine to me. > > > > On 23/02/2022 10:49, Konstantin Knauf wrote: > > > Hi Chesnay, Hi everyone, > > > > > > I think the idea for the migration is the following (with the example > of > > > ElasticSearch). I talked to Martijn offline. > > > > > > 1. ElasticSearch Connector is released from the core repository with > the > > > Flink 1.15.0 release. No changes. > > > > > > 2. At the beginning of the Flink 1.16 release cycle the connector is > > > removed from `master` of the core repository. It remains on the > > > `release-1.15` branch and earlier release branches. > > > > > > 3. The connector code is "copied" over to the `master` branch of a > > > `flink-connector-elastic-search` repository. Bugfixes to the connector > need > > > to go to both `release-1.15` and before in the core repository and > `master` > > > of the external repository. > > > > > > 4. Once all the processes required to do a release in the > > > `flink-connector-elastic-search` are in place (docs integration, > release > > > automation,...), we release flink-connector-elastic-search:3.0.0, which > > > will be compatible with Flink 1.15. At this point, users can choose > whether > > > they use flink-connector-elastic-search:1.15.x (released from the core > > > repository) or flink-connector-elastic-search:3.0.0 already released > from > > > the external repository with Flink 1.15. The documentation will already > > > advertise the one released from the external repository. This is the > > > "overlap" that Martijn mentioned. > > > > > > 5. From here onwards, the release cycle of the ElasticSearch Connector > is > > > independent. There could be 3.1.0 and 3.0.1 etc. The compatibility > matrix > > > will be part of the connector documentation. > > > > > > 6. If there is a patch release for Flink 1.15-, this will of course > also > > > include flink-connector-elastic-search release from the core > repository. > > > > > > 7. For Flink 1.16, there might or might not be a release of the > > > elastic-search-connector from the external repository. Depends on > > > compatibility. > > > > > > I hope this clarifies it a bit and it makes sense to me. > > > > > > It also makes sense to me to do this as soon as possible (probably > once the > > > release-1.15 branch is cut) with the example of ElasticSearch. > Afterwards > > > (hopefully still in the Flink 1.16 release cycle) we do the same for > other > > > connectors like Kafka, Pulsar, Kinesis. I don't think it's feasible or > > > helpful to make it a condition that this happens for all connectors at > the > > > same time. > > > > > > Cheers and thanks, > > > > > > Konstantin > > > > > > > > > On Sun, Feb 20, 2022 at 7:57 PM Chesnay Schepler <ches...@apache.org> > wrote: > > > > > >> If we don't make a release, I think it would appear as partially > > >> externalized (since the binaries are still only created with Flink > core, > > >> not from the external repository). > > >> > > >> I'm wondering you are referring to when you say "it appear[s]". Users > > >> don't know about it, and contributors can be easily informed about the > > >> current state. Who's opinion are you worried about? > > >> > > >> doing a release [means] that we have then completely externalized the > > >> connector > > >> > > >> In my mind the connector is completely externalized once the connector > > >> project is complete and can act independently from the core repo. That > > >> includes having all the code, working CI and the documentation being > > >> integrated into the Flink website. And /then/ we can do a release. I > > >> don't see how this could work any other way; how could we possibly > argue > > >> that the connector is externalized when development on the connector > > >> isn't even possible in that repository? > > >> > > >> There are also other connectors (like Opensearch and I believe > RabbitMQ) > > >> that will end up straight in their own repositories > > >> > > >> > > >> Which is a bit of a different situation because here the only source > of > > >> this connector will have been that repository. > > >> > > >> Would you prefer to remove a connector in a Flink patch release? > > >> > > >> > > >> No. I think I misread your statement; when you said that there "is 1 > > >> release cycle where the connector both exists in Flink core and the > > >> external repo", you are referring to 1.15, correct? (although this > > >> should also apply to 1.14 so isn't it 2 releases...?) > > >> How I understood it was that we'd keep the connector around until > 1.16, > > >> which would obviously be terrible. > > >> > > >> On 19/02/2022 13:30, Martijn Visser wrote: > > >>> Hi Chesnay, > > >>> > > >>> I think the advantage of also doing a release is that we have then > > >>> completely externalized the connector. If we don't make a release, I > > >> think > > >>> it would appear as partially externalized (since the binaries are > still > > >>> only created with Flink core, not from the external repository). It > would > > >>> also mean that in our documentation we would still point to the > binary > > >>> created with the Flink core release. > > >>> > > >>> There are also other connectors (like Opensearch and I believe > RabbitMQ) > > >>> that will end up straight in their own repositories. Since we also > would > > >>> like to document those, I don't think the situation will be messy. > We can > > >>> also solve it with an information hint in the documentation. > > >>> > > >>> With regards to point 6, do you have an alternative? Would you > prefer to > > >>> remove a connector in a Flink patch release? > > >>> > > >>> Best regards, > > >>> > > >>> Martijn > > >>> > > >>> On Fri, 18 Feb 2022 at 16:17, Chesnay Schepler<ches...@apache.org> > > >> wrote: > > >>>> Why do you want to immediately do a release for the elasticsearch > > >>>> connector? What does that provide us? > > >>>> > > >>>> I'd rather first have a fully working setup and integrated > documentation > > >>>> before thinking about releasing anything. > > >>>> Once we have that we may be able to externalize all connectors > within 1 > > >>>> release cycle and do a clean switch; otherwise we end up with a bit > of a > > >>>> messy situation for users where some connectors use version scheme > A and > > >>>> others B. > > >>>> > > >>>> I also doubt the value of 6). They'll have to update the version > anyway > > >>>> (and discover at some point that the version scheme has changed), > so I > > >>>> don't see what this makes easier. > > >>>> > > >>>> On 18/02/2022 14:54, Martijn Visser wrote: > > >>>>> Hi everyone, > > >>>>> > > >>>>> As a follow-up to earlier discussions [1] [2] to externalize the > > >>>> connectors > > >>>>> from the Flink repository, I would like to propose a plan to > > >> externalize > > >>>>> these connectors. The goal of this plan is to start with moving > > >>>> connectors > > >>>>> to its own repositories without introducing regressions for > connector > > >>>>> developers. > > >>>>> > > >>>>> The plan is as follows: > > >>>>> > > >>>>> 1. A new repository is requested for a connector. > > >>>>> 2. The code for that connector is moved to its individual > repository, > > >>>>> including the commit history > > >>>>> 3. Any first release made for a connector in an external connector > > >>>>> repository starts with major version 3, so 3.0.0. The reason for > that > > >> is > > >>>>> that we want to decouple the Flink releases from a connector > release. > > >> If > > >>>> we > > >>>>> would start with major version 2, it could cause some confusion > because > > >>>>> people could think a Flink 2.0 has been released. This does mean > that > > >>>> each > > >>>>> connector needs to have a compatibility matrix generated, stating > which > > >>>>> version number of the connector is compatible with the correct > Flink > > >>>>> version. > > >>>>> 4. The group id and artifact id for the connector will remain the > same, > > >>>>> only the version is different. > > >>>>> 5. The connector dependencies on the Flink website are updated to > point > > >>>> to > > >>>>> the newly released connector artifact. > > >>>>> 6. If a connector is moved, there is one release cycle where there > will > > >>>> be > > >>>>> binary releases for that connector in both Flink core and from the > > >>>>> connector repository. This is to make Flink users who are upgrading > > >>>>> slightly easier. We will have to make a note in the release notes > that > > >> a > > >>>>> connector has been moved and that a user should update any > references > > >>>> from > > >>>>> the original connector artifact (from the Flink connector) to the > new > > >>>>> connector artifact (from the external conenctor version) > > >>>>> > > >>>>> We propose to first try to execute this plan for the Elasticsearch > > >>>>> connector as follows: > > >>>>> > > >>>>> 1. We wait until the Flink 1.15 release branch is cut > > >>>>> 2. When that's done, the Elasticsearch code (including commit > history) > > >>>> from > > >>>>> Flink's 1.15 release branch will be moved to the > > >>>>> flink-connector-elasticsearch main branch. > > >>>>> 3. When Flink 1.15 is released, we will also release an > Elasticsearch > > >>>>> connector for the external connector repository with version 3.0.0. > > >>>>> 4. Bugfixes or improvements will be made first pointing to the > external > > >>>>> connector repository and will be cherry-picked back to the > release-1.15 > > >>>>> branch in the Flink core repository. > > >>>>> 5. The Elasticsearch code, test etc will be removed from the master > > >>>> branch > > >>>>> in the Flink core repository and dropped with Flink 1.16 > > >>>>> > > >>>>> Looking forward to your thoughts on this! > > >>>>> > > >>>>> Best regards, > > >>>>> > > >>>>> Martijn Visser > > >>>>> https://twitter.com/MartijnVisser82 > > >>>>> > > >>>>> [1] > https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm > > >>>>> [2] > https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr > > >>>>> > > > > > >
