Hi Tom, I had assumed that for the existing Kafka connectors for Flink 1.20 and Flink 2, we would not move up the Kafka client level, so there were no migration issues. So we would want a new Version of the Kafka client to adopt Kafka 4.1.0. When you say “ I think we should just release a new version of the connector for every Flink and Kafka release “ - could we just do this for Kafka connector 5 supporting Flink 2. I am not sure we want a new Kafka connector version for Flink 1.20 - unless there is a big demand for it.
Kind regards, David. From: Tom Cooper <[email protected]> Date: Wednesday, 10 September 2025 at 15:00 To: [email protected] <[email protected]> Subject: [EXTERNAL] RE: Flink Connector Kafka Releases Hi David, The Kafka connector with Flink 2.0 should work with Flink 2.1, but as the PR [1] to do the update to 2.1 is already merged its a moot point now. Personally, I don't think we need a 4.0.1 release of the connector. That would only really be useful for users with very old Kafka clusters who want to use new versions of Flink. Frankly, users should be on a newer Kafka version than 2.0 by now. The main branch of the connector is on Flink 2.1 and Kafka 4.0.0. I think we should proceed with a 5.0 release with those. I do have a PR [2] open for Kafka 4.1 support and at this point I don't see why we shouldn't ship the most up to date client library version available, but that is "a nice to have". I am interested in your "supported concurrent releases" comment. I know in Flink we support multiple previous versions, but I couldn't find any docs/reference to what we should do with connectors? Connectors are an interesting case as there is a Flink library version and one or more significant external library (e.g. Kafka) versions, so the support matrix becomes complex. We could say we support a connector for current-N Flink versions and will provide updates to the latest available Kafka versions for each one? The question then becomes what value N should be. If it was 2 then, with the release of Kafka 4.1.0, we would have something like this: - Kafka Connector 3.5.0 / Flink 1.20 / Kafka 4.1.0 - Kafka Connector 4.1.0 / Flink 2.0 / Kafka 4.1.0 - Kafka Connector 5.0.0 / Flink 2.1 / Kafka 4.1.0 Essentially the major version is for flink updates and the minor for Kafka updates within each Flink version. When Flink 2.2 comes out we would stop updating the 3.x (Flink 1.20) branch. This is just an example and would actually be a lot of work, with N+1 release processes for every Flink or Kafka release. Personally, I think we should just release a new version of the connector for every Flink and Kafka release (maybe using the major/minor versions approach above) and do other releases on an as-needed basis (which actually sounds like the current situation). I am interested what you (and the community) think? Cheers, Tom Cooper @tomcooper.dev | https://tomcooper.dev [1] https://github.com/apache/flink-connector-kafka/pull/187 [2] https://github.com/apache/flink-connector-kafka/pull/190 On Friday, 29 August 2025 at 14:28, David Radley <[email protected]> wrote: > Hi Tom, > Thinking about releases. I see: > > Kafka connector 3.3 and 3.4, Flink 1.20 > Kafka connector 4.0, Kafka client 3.9.0 Flink 2.0 > Kafka connector 4.0.1, Kafka client 3.9.1 Flink 2.0. (does this version work > with Flink 2.1 or is [1] a show stopper?) > > I assume we might want: > Kafka connector 4.0.2, Kafka client 3.9.1 Flink 2.1 > Kafka connector 5 Kafka client 4.0 Flink 2.1 > > This would be a lot of concurrent releases that are supported, > Kind regards, David. > > [1] https://github.com/apache/flink-connector-kafka/pull/187 > > From: Tom Cooper [email protected] > > Date: Friday, 29 August 2025 at 09:57 > To: [email protected] [email protected] > > Subject: [EXTERNAL] Re: Flink Connector Kafka Releases > > FYI, I have posted a PR [1] for the Flink 2.1 update for the Kafka connector. > > The main change with this update is that Flink has removed [2] (without > deprecating first) the stripRowPrefix method from the DataTypeUtilsTest > class. I couldn't see any evidence that this logic was moved to any other > public API so this PR adds that logic into a util class within the connector. > > Also note that with the update to Flink 2.1.0 the Kafka connector drops > support for Python 3.8, I added testing for Python 3.11 but 3.12-cython > (Flink 2.1 added support for Python 3.12) is not yet available. > > If we can get his PR merged and the Kafka 4.0.0 update PR [3], we will be in > a good position for the 5.0.0 release? > > Thanks, > > Tom Cooper > @tomcooper.dev | https://tomcooper.dev > > [1] https://github.com/apache/flink-connector-kafka/pull/187 > [2] https://github.com/apache/flink/pull/26784 > > On Thursday, 14 August 2025 at 21:59, Weiqing Yang [email protected] > wrote: > > > Thanks, Fabian. This is helpful to know. > > > > On Thu, Aug 14, 2025 at 1:50 AM Fabian Paul [email protected] wrote: > > > > > Hi Weiqing, > > > > > > So far, there is no concrete timeline to publish the Kafka 5.0 > > > release. We are still releasing 4.0.1 at the moment. Regarding the > > > Flink 2.1 support, you should be able to use the flink kafka connector > > > that supports 2.0 also with 2.1. The underlying connector library is > > > stable across minor versions. This also means that a new Flink minor > > > release doesn't necessarily warrant a new connector release. > > > > > > Best, > > > Fabian > > > > > > On Wed, Aug 13, 2025 at 6:57 PM Weiqing Yang [email protected] > > > wrote: > > > > > > > Hi Tom, Fabian, > > > > > > > > Thanks for the updates on the v4.0.1 release. I noticed that v4.0.1 is > > > > going out with Flink 2.0 (link > > > > < > > > > https://github.com/apache/flink-connector-kafka/blob/v4.0.1-rc2/pom.xml#L56 > > > > ). > > > > Do you have a timeline or target window in mind for the Flink Connector > > > > Kafka 5.0 release that will include Flink 2.1 support? > > > > > > > > Thanks, > > > > Weiqing > > > > > > > > On Mon, Aug 11, 2025 at 7:41 AM Tom Cooper [email protected] wrote: > > > > > > > > > Hi Fabian, > > > > > > > > > > Sorry, for the late reply, this message somehow ended up in my spam > > > > > filter!? > > > > > > > > > > I think having the Flink 2.1 upgrade included in the move to Flink > > > > > Connector Kafka 5.0 makes sense. > > > > > I am hoping to find the time to work on the upgrade to Flink 2.1 at > > > > > end of > > > > > this week or next. > > > > > Unless, of course, you are plan to work on that? > > > > > > > > > > Regards, > > > > > > > > > > Tom Cooper > > > > > @tomcooper.dev | https://tomcooper.dev > > > > > > > > > > On Tuesday, 29 July 2025 at 09:08, Fabian Paul > > > > > [email protected] > > > > > wrote: > > > > > > > > > > > Hi Tom, > > > > > > > > > > > > Sounds good to me, I can start with the 4.0.1 release. > > > > > > Regarding the 5.0 release, I am not super sure yet what to include. > > > > > > Since releasing always takes some effort, I would also be okay with > > > > > > doing the 5.0 release with incorporating Flink 2.1. The connector > > > > > > already offers a release that is compatible with Flink 2.0, and in > > > > > > theory, 2.1 should not introduce breaking changes that affect the > > > > > > connector. > > > > > > > > > > > > Best, > > > > > > Fabian > > > > > > > > > > > > On Mon, Jul 28, 2025 at 11:03 AM Tom Cooper [email protected] > > > > > > wrote: > > > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > > > You make a good point, as there are only dependency updates, a > > > > > > > 4.0.1 > > > > > > > release makes more sense. > > > > > > > > > > > > > > At this point the 5.0 connector release could include the soon to > > > > > > > be > > > > > > > released Kafka 4.0.1 client libraries (the RC for that is out > > > > > > > already). > > > > > > > I assume we would want to leave the flink 2.1 upgrade to a future > > > > > > > 5.1 > > > > > > > release? > > > > > > > > > > > > > > Thanks for looking at this. > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Tom Cooper > > > > > > > @tomcooper.dev | https://tomcooper.dev > > > > > > > > > > > > > > On Monday, 28 July 2025 at 09:51, Fabian Paul [email protected] > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Tom, > > > > > > > > > > > > > > > > Thanks for starting this discussion. I think it's a good idea to > > > > > > > > do > > > > > > > > another 4.1.0 release before proceeding with 5.0 to offer a > > > > > > > > release > > > > > > > > with the vulnerability fixed without requiring users to upgrade > > > > > > > > to > > > > > > > > Kafka 4.0. Is there a reason you prefer to do the 4.1.0 release > > > > > > > > instead of the 4.0.1 release? I reviewed the changes between the > > > > > > > > current main and the release 4.0.0 [1], and they are mostly > > > > > > > > dependency > > > > > > > > upgrades and some fixes, but without any new features. What do > > > > > > > > you > > > > > > > > think about doing a 4.0.1 release and then kicking off 5.0.0 > > > > > > > > with the > > > > > > > > Kafka client upgrade? > > > > > > > > > > > > > > > > Best, > > > > > > > > Fabian > > > > > > > > > > > > > > > > [1] > > > > > > > > https://github.com/apache/flink-connector-kafka/compare/v4.0...main > > > > > > > > > > > > > > > > On Fri, Jul 25, 2025 at 11:58 AM Tom Cooper [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Bumping this thread as we are now ready to merge the Kafka > > > > > > > > > 4.0.0 > > > > > > > > > client update PR [1]. This will bump the major version of the > > > > > > > > > connector to > > > > > > > > > 5.0, as we are dropping support for Kafka brokers running > > > > > > > > > version > > > > > > > > > 2.0.0 or > > > > > > > > > earlier. > > > > > > > > > > > > > > > > > > However, I still think it would be worth doing a 4.1.0 release > > > > > > > > > of > > > > > > > > > the connector (with the Kafka 3.9.1 client), before the Kafka > > > > > > > > > 4.0.0 > > > > > > > > > client > > > > > > > > > update is merged. > > > > > > > > > > > > > > > > > > The current Flink Kafka Connector (4.0) has a critical CVE > > > > > > > > > [2], > > > > > > > > > which is patched in the 3.9.1 Kafka client library (which the > > > > > > > > > current > > > > > > > > > main > > > > > > > > > branch of the Flink connector is using). Doing a 4.1 release > > > > > > > > > of the > > > > > > > > > connector would cover any users of older Kafka versions that > > > > > > > > > want this > > > > > > > > > CVE > > > > > > > > > patched and also give a stable release of the connector using > > > > > > > > > a point > > > > > > > > > release of the Kafka client (with all the bug fixes that > > > > > > > > > entails). This > > > > > > > > > would be a good option for users who don't want to jump > > > > > > > > > straight onto > > > > > > > > > the > > > > > > > > > new major Kafka client version. > > > > > > > > > > > > > > > > > > What do people think? > > > > > > > > > > > > > > > > > > Tom Cooper > > > > > > > > > @tomcooper.dev | https://tomcooper.dev > > > > > > > > > > > > > > > > > > [1] https://github.com/apache/flink-connector-kafka/pull/161 > > > > > > > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2025-27817 > > > > > > > > > > > > > > > > > > On Wednesday, 9 July 2025 at 09:35, Tom Cooper > > > > > > > > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > I would like to start a conversation about releases for the > > > > > > > > > > Flink Connector Kafka project. > > > > > > > > > > > > > > > > > > > > We have recently updated [0] to version 3.9.1 of the Kafka > > > > > > > > > > client library, which fixes a critical CVE [1]. With that > > > > > > > > > > in mind, I > > > > > > > > > > think > > > > > > > > > > it would be prudent to have a 4.1.0 release as soon as > > > > > > > > > > possible that > > > > > > > > > > includes this. It would also be good to include the > > > > > > > > > > dependency bumps > > > > > > > > > > from > > > > > > > > > > this PR [2] in that release. > > > > > > > > > > > > > > > > > > > > With the 4.1.0 release out, we could then move to looking at > > > > > > > > > > the > > > > > > > > > > Kafka 4.0 upgrade (there is already a PR [3] for that). The > > > > > > > > > > main point > > > > > > > > > > with > > > > > > > > > > the Kafka 4.0 upgrade is that it drops support for Kafka > > > > > > > > > > brokers > > > > > > > > > > running > > > > > > > > > > version 2.0.0 and lower. Given this, I think it would make > > > > > > > > > > sense to > > > > > > > > > > move > > > > > > > > > > the Connector version to 5.0.0 and maybe even move to Flink > > > > > > > > > > 2.1.0 > > > > > > > > > > (which > > > > > > > > > > should be available in a month or so). This 5.0.0 release > > > > > > > > > > could also > > > > > > > > > > remove > > > > > > > > > > all the Zookeeper specific test infra and move to KRaft > > > > > > > > > > based clusters > > > > > > > > > > for > > > > > > > > > > testing. We could also move to a new, updated Flink > > > > > > > > > > Connector Parent > > > > > > > > > > pom > > > > > > > > > > version [4] which would harmonise the java versions and > > > > > > > > > > plugins with > > > > > > > > > > the > > > > > > > > > > main Flink project. > > > > > > > > > > > > > > > > > > > > I think, if the above is acceptable, that these changes > > > > > > > > > > warrant > > > > > > > > > > a major version bump. Users of older Kafka clusters would > > > > > > > > > > still be > > > > > > > > > > able to > > > > > > > > > > use 4.1.0 (which is an argument for making sure that > > > > > > > > > > release has the > > > > > > > > > > most > > > > > > > > > > up-to-date dependencies). > > > > > > > > > > > > > > > > > > > > Anyway, I would love to hear what the community think of the > > > > > > > > > > above. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > Tom Cooper > > > > > > > > > > @tomcooper.dev | https://tomcooper.dev > > > > > > > > > > > > > > > > > > > > [0] https://github.com/apache/flink-connector-kafka/pull/180 > > > > > > > > > > [1] https://nvd.nist.gov/vuln/detail/CVE-2025-27817 > > > > > > > > > > [2] https://github.com/apache/flink-connector-kafka/pull/181 > > > > > > > > > > [3] https://github.com/apache/flink-connector-kafka/pull/161 > > > > > > > > > > [4] > > > > > > > > > > https://github.com/apache/flink-connector-shared-utils/pull/48 > > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: Building C, IBM Hursley Office, Hursley Park Road, > Winchester, Hampshire SO21 2JN Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
