+1 to the approach. I expect that we will encounter more questions and challenges as we go, but these are best discussed and addressed in the context of a specific connector like ElasticSearch.
On Fri, Feb 18, 2022 at 2:54 PM Martijn Visser <mart...@ververica.com> wrote: > Hi everyone, > > As a follow-up to earlier discussions [1] [2] to externalize the connectors > from the Flink repository, I would like to propose a plan to externalize > these connectors. The goal of this plan is to start with moving connectors > to its own repositories without introducing regressions for connector > developers. > > The plan is as follows: > > 1. A new repository is requested for a connector. > 2. The code for that connector is moved to its individual repository, > including the commit history > 3. Any first release made for a connector in an external connector > repository starts with major version 3, so 3.0.0. The reason for that is > that we want to decouple the Flink releases from a connector release. If we > would start with major version 2, it could cause some confusion because > people could think a Flink 2.0 has been released. This does mean that each > connector needs to have a compatibility matrix generated, stating which > version number of the connector is compatible with the correct Flink > version. > 4. The group id and artifact id for the connector will remain the same, > only the version is different. > 5. The connector dependencies on the Flink website are updated to point to > the newly released connector artifact. > 6. If a connector is moved, there is one release cycle where there will be > binary releases for that connector in both Flink core and from the > connector repository. This is to make Flink users who are upgrading > slightly easier. We will have to make a note in the release notes that a > connector has been moved and that a user should update any references from > the original connector artifact (from the Flink connector) to the new > connector artifact (from the external conenctor version) > > We propose to first try to execute this plan for the Elasticsearch > connector as follows: > > 1. We wait until the Flink 1.15 release branch is cut > 2. When that's done, the Elasticsearch code (including commit history) from > Flink's 1.15 release branch will be moved to the > flink-connector-elasticsearch main branch. > 3. When Flink 1.15 is released, we will also release an Elasticsearch > connector for the external connector repository with version 3.0.0. > 4. Bugfixes or improvements will be made first pointing to the external > connector repository and will be cherry-picked back to the release-1.15 > branch in the Flink core repository. > 5. The Elasticsearch code, test etc will be removed from the master branch > in the Flink core repository and dropped with Flink 1.16 > > Looking forward to your thoughts on this! > > Best regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > > [1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm > [2] https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
