Hi Danny, I think your proposal is a good one. This is the approach that we took with the Egeria project, firstly taking the connectors out of the main repo, then connectors having their own versions that incremented organically rather then tied to the core release.
Blue sky thinking - I wonder if we could : - have a wizard / utility so the user inputs which Flink level they want and which connectors; the utility knows the compatibility matrix and downloads the appropriate bundles. - have the docs interrogate the core and connector repos to check the poms for the Flink levels and the pr builds to have ?live? docs showing the supported Flink levels. PyTorch does something like this for it?s docs. Kind regards, David. From: Danny Cranmer <dannycran...@apache.org> Date: Monday, 10 June 2024 at 17:26 To: dev <dev@flink.apache.org> Subject: [EXTERNAL] [DISCUSS] Connector Externalization Retrospective Hello Flink community, It has been over 2 years [1] since we started externalizing the Flink connectors to dedicated repositories from the main Flink code base. The past discussions can be found here [2]. The community decided to externalize the connectors to primarily 1/ improve stability and speed of the CI, and 2/ decouple version and release lifecycle to allow the projects to evolve independently. The outcome of this has resulted in each connector requiring a dedicated release per Flink minor version, which is a burden on the community. Flink 1.19.0 was released on 2024-03-18 [3], the first supported connector followed roughly 2.5 months later on 2024-06-06 [4] (MongoDB). There are still 5 connectors that do not support Flink 1.19 [5]. Two decisions contribute to the high lag between releases. 1/ creating one repository per connector instead of a single flink-connector mono-repo and 2/ coupling the Flink version to the connector version [6]. A single connector repository would reduce the number of connector releases from N to 1, but would couple the connector CI and reduce release flexibility. Decoupling the connector versions from Flink would eliminate the need to release each connector for each new Flink minor version, but we would need a new compatibility mechanism. I propose that from each next connector release we drop the coupling on the Flink version. For example, instead of 3.4.0-1.20 (<connector>.<flink>) we would release 3.4.0 (<connector>). We can model a compatibility matrix within the Flink docs to help users pick the correct versions. This would mean we would usually not need to release a new connector version per Flink version, assuming there are no breaking changes. Worst case, if breaking changes impact all connectors we would still need to release all connectors. However, for Flink 1.17 and 1.18 there were only a handful of issues (breaking changes), and mostly impacting tests. We could decide to align this with Flink 2.0, however I see no compelling reason to do so. This was discussed previously [2] as a long term goal once the connector APIs are stable. But I think the current compatibility rules support this change now. I would prefer to not create a connector mono-repo. Separate repos gives each connector more flexibility to evolve independently, and removing unnecessary releases will significantly reduce the release effort. I would like to hear opinions and ideas from the community. In particular, are there any other issues you have observed that we should consider addressing? Thanks, Danny. [1] https://github.com/apache/flink-connector-elasticsearch/commit/3ca2e625e3149e8864a4ad478773ab4a82720241 [2] https://lists.apache.org/thread/8k1xonqt7hn0xldbky1cxfx3fzh6sj7h [3] https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ [4] https://flink.apache.org/downloads/#apache-flink-connectors-1 [5] https://issues.apache.org/jira/browse/FLINK-35131 [6] https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development#ExternalizedConnectordevelopment-Examples Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
