Hello Flink community, It has been over 2 years [1] since we started externalizing the Flink connectors to dedicated repositories from the main Flink code base. The past discussions can be found here [2]. The community decided to externalize the connectors to primarily 1/ improve stability and speed of the CI, and 2/ decouple version and release lifecycle to allow the projects to evolve independently. The outcome of this has resulted in each connector requiring a dedicated release per Flink minor version, which is a burden on the community. Flink 1.19.0 was released on 2024-03-18 [3], the first supported connector followed roughly 2.5 months later on 2024-06-06 [4] (MongoDB). There are still 5 connectors that do not support Flink 1.19 [5].
Two decisions contribute to the high lag between releases. 1/ creating one repository per connector instead of a single flink-connector mono-repo and 2/ coupling the Flink version to the connector version [6]. A single connector repository would reduce the number of connector releases from N to 1, but would couple the connector CI and reduce release flexibility. Decoupling the connector versions from Flink would eliminate the need to release each connector for each new Flink minor version, but we would need a new compatibility mechanism. I propose that from each next connector release we drop the coupling on the Flink version. For example, instead of 3.4.0-1.20 (<connector>.<flink>) we would release 3.4.0 (<connector>). We can model a compatibility matrix within the Flink docs to help users pick the correct versions. This would mean we would usually not need to release a new connector version per Flink version, assuming there are no breaking changes. Worst case, if breaking changes impact all connectors we would still need to release all connectors. However, for Flink 1.17 and 1.18 there were only a handful of issues (breaking changes), and mostly impacting tests. We could decide to align this with Flink 2.0, however I see no compelling reason to do so. This was discussed previously [2] as a long term goal once the connector APIs are stable. But I think the current compatibility rules support this change now. I would prefer to not create a connector mono-repo. Separate repos gives each connector more flexibility to evolve independently, and removing unnecessary releases will significantly reduce the release effort. I would like to hear opinions and ideas from the community. In particular, are there any other issues you have observed that we should consider addressing? Thanks, Danny. [1] https://github.com/apache/flink-connector-elasticsearch/commit/3ca2e625e3149e8864a4ad478773ab4a82720241 [2] https://lists.apache.org/thread/8k1xonqt7hn0xldbky1cxfx3fzh6sj7h [3] https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ [4] https://flink.apache.org/downloads/#apache-flink-connectors-1 [5] https://issues.apache.org/jira/browse/FLINK-35131 [6] https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development#ExternalizedConnectordevelopment-Examples
