Hi David, Yes, basically I was saying lets just do a new 5.0 release for Flink 2.1 - Kafka 4.0.0/4.1.0 and not backport the Kafka update to other releases. In future we just do new connector releases for Flink/Kafka versions as they come out. Backports can then be done on an as needed (critical CVEs etc) basis. Which is basically what we seem to be doing now anyway.
So now the job is to find a maintainer with the time to do the 5.0 release? Tom Cooper @tomcooper.dev | https://tomcooper.dev On Wednesday, 10 September 2025 at 15:25, David Radley <[email protected]> wrote: > Hi Tom, > I had assumed that for the existing Kafka connectors for Flink 1.20 and Flink > 2, we would not move up the Kafka client level, so there were no migration > issues. So we would want a new Version of the Kafka client to adopt Kafka > 4.1.0. When you say “ I think we should just release a new version of the > connector for every Flink and Kafka release “ - could we just do this for > Kafka connector 5 supporting Flink 2. > I am not sure we want a new Kafka connector version for Flink 1.20 - unless > there is a big demand for it. > > Kind regards, David. > > From: Tom Cooper [email protected] > > Date: Wednesday, 10 September 2025 at 15:00 > To: [email protected] [email protected] > > Subject: [EXTERNAL] RE: Flink Connector Kafka Releases > > Hi David, > > The Kafka connector with Flink 2.0 should work with Flink 2.1, but as the PR > [1] to do the update to 2.1 is already merged its a moot point now. > > Personally, I don't think we need a 4.0.1 release of the connector. That > would only really be useful for users with very old Kafka clusters who want > to use new versions of Flink. Frankly, users should be on a newer Kafka > version than 2.0 by now. > > The main branch of the connector is on Flink 2.1 and Kafka 4.0.0. I think we > should proceed with a 5.0 release with those. I do have a PR [2] open for > Kafka 4.1 support and at this point I don't see why we shouldn't ship the > most up to date client library version available, but that is "a nice to > have". > > I am interested in your "supported concurrent releases" comment. I know in > Flink we support multiple previous versions, but I couldn't find any > docs/reference to what we should do with connectors? > > Connectors are an interesting case as there is a Flink library version and > one or more significant external library (e.g. Kafka) versions, so the > support matrix becomes complex. We could say we support a connector for > current-N Flink versions and will provide updates to the latest available > Kafka versions for each one? The question then becomes what value N should > be. If it was 2 then, with the release of Kafka 4.1.0, we would have > something like this: > > - Kafka Connector 3.5.0 / Flink 1.20 / Kafka 4.1.0 > - Kafka Connector 4.1.0 / Flink 2.0 / Kafka 4.1.0 > - Kafka Connector 5.0.0 / Flink 2.1 / Kafka 4.1.0 > > Essentially the major version is for flink updates and the minor for Kafka > updates within each Flink version. When Flink 2.2 comes out we would stop > updating the 3.x (Flink 1.20) branch. This is just an example and would > actually be a lot of work, with N+1 release processes for every Flink or > Kafka release. > > Personally, I think we should just release a new version of the connector for > every Flink and Kafka release (maybe using the major/minor versions approach > above) and do other releases on an as-needed basis (which actually sounds > like the current situation). > > I am interested what you (and the community) think? > > Cheers, > > Tom Cooper > @tomcooper.dev | https://tomcooper.dev > > [1] https://github.com/apache/flink-connector-kafka/pull/187 > [2] https://github.com/apache/flink-connector-kafka/pull/190 > > > On Friday, 29 August 2025 at 14:28, David Radley [email protected] > wrote: > > > Hi Tom, > > Thinking about releases. I see: > > > > Kafka connector 3.3 and 3.4, Flink 1.20 > > Kafka connector 4.0, Kafka client 3.9.0 Flink 2.0 > > Kafka connector 4.0.1, Kafka client 3.9.1 Flink 2.0. (does this version > > work with Flink 2.1 or is [1] a show stopper?) > > > > I assume we might want: > > Kafka connector 4.0.2, Kafka client 3.9.1 Flink 2.1 > > Kafka connector 5 Kafka client 4.0 Flink 2.1 > > > > This would be a lot of concurrent releases that are supported, > > Kind regards, David. > > > > [1] https://github.com/apache/flink-connector-kafka/pull/187 > > > > From: Tom Cooper [email protected] > > > > Date: Friday, 29 August 2025 at 09:57 > > To: [email protected] [email protected] > > > > Subject: [EXTERNAL] Re: Flink Connector Kafka Releases > > > > FYI, I have posted a PR [1] for the Flink 2.1 update for the Kafka > > connector. > > > > The main change with this update is that Flink has removed [2] (without > > deprecating first) the stripRowPrefix method from the DataTypeUtilsTest > > class. I couldn't see any evidence that this logic was moved to any other > > public API so this PR adds that logic into a util class within the > > connector. > > > > Also note that with the update to Flink 2.1.0 the Kafka connector drops > > support for Python 3.8, I added testing for Python 3.11 but 3.12-cython > > (Flink 2.1 added support for Python 3.12) is not yet available. > > > > If we can get his PR merged and the Kafka 4.0.0 update PR [3], we will be > > in a good position for the 5.0.0 release? > > > > Thanks, > > > > Tom Cooper > > @tomcooper.dev | https://tomcooper.dev > > > > [1] https://github.com/apache/flink-connector-kafka/pull/187 > > [2] https://github.com/apache/flink/pull/26784 > > > > On Thursday, 14 August 2025 at 21:59, Weiqing Yang [email protected] > > wrote: > > > > > Thanks, Fabian. This is helpful to know. > > > > > > On Thu, Aug 14, 2025 at 1:50 AM Fabian Paul [email protected] wrote: > > > > > > > Hi Weiqing, > > > > > > > > So far, there is no concrete timeline to publish the Kafka 5.0 > > > > release. We are still releasing 4.0.1 at the moment. Regarding the > > > > Flink 2.1 support, you should be able to use the flink kafka connector > > > > that supports 2.0 also with 2.1. The underlying connector library is > > > > stable across minor versions. This also means that a new Flink minor > > > > release doesn't necessarily warrant a new connector release. > > > > > > > > Best, > > > > Fabian > > > > > > > > On Wed, Aug 13, 2025 at 6:57 PM Weiqing Yang [email protected] > > > > wrote: > > > > > > > > > Hi Tom, Fabian, > > > > > > > > > > Thanks for the updates on the v4.0.1 release. I noticed that v4.0.1 is > > > > > going out with Flink 2.0 (link > > > > > < > > > > > https://github.com/apache/flink-connector-kafka/blob/v4.0.1-rc2/pom.xml#L56 > > > > > ). > > > > > Do you have a timeline or target window in mind for the Flink > > > > > Connector > > > > > Kafka 5.0 release that will include Flink 2.1 support? > > > > > > > > > > Thanks, > > > > > Weiqing > > > > > > > > > > On Mon, Aug 11, 2025 at 7:41 AM Tom Cooper [email protected] wrote: > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > Sorry, for the late reply, this message somehow ended up in my spam > > > > > > filter!? > > > > > > > > > > > > I think having the Flink 2.1 upgrade included in the move to Flink > > > > > > Connector Kafka 5.0 makes sense. > > > > > > I am hoping to find the time to work on the upgrade to Flink 2.1 at > > > > > > end of > > > > > > this week or next. > > > > > > Unless, of course, you are plan to work on that? > > > > > > > > > > > > Regards, > > > > > > > > > > > > Tom Cooper > > > > > > @tomcooper.dev | https://tomcooper.dev > > > > > > > > > > > > On Tuesday, 29 July 2025 at 09:08, Fabian Paul > > > > > > [email protected] > > > > > > wrote: > > > > > > > > > > > > > Hi Tom, > > > > > > > > > > > > > > Sounds good to me, I can start with the 4.0.1 release. > > > > > > > Regarding the 5.0 release, I am not super sure yet what to > > > > > > > include. > > > > > > > Since releasing always takes some effort, I would also be okay > > > > > > > with > > > > > > > doing the 5.0 release with incorporating Flink 2.1. The connector > > > > > > > already offers a release that is compatible with Flink 2.0, and in > > > > > > > theory, 2.1 should not introduce breaking changes that affect the > > > > > > > connector. > > > > > > > > > > > > > > Best, > > > > > > > Fabian > > > > > > > > > > > > > > On Mon, Jul 28, 2025 at 11:03 AM Tom Cooper [email protected] > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > > > > > You make a good point, as there are only dependency updates, a > > > > > > > > 4.0.1 > > > > > > > > release makes more sense. > > > > > > > > > > > > > > > > At this point the 5.0 connector release could include the soon > > > > > > > > to > > > > > > > > be > > > > > > > > released Kafka 4.0.1 client libraries (the RC for that is out > > > > > > > > already). > > > > > > > > I assume we would want to leave the flink 2.1 upgrade to a > > > > > > > > future > > > > > > > > 5.1 > > > > > > > > release? > > > > > > > > > > > > > > > > Thanks for looking at this. > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > Tom Cooper > > > > > > > > @tomcooper.dev | https://tomcooper.dev > > > > > > > > > > > > > > > > On Monday, 28 July 2025 at 09:51, Fabian Paul [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Tom, > > > > > > > > > > > > > > > > > > Thanks for starting this discussion. I think it's a good idea > > > > > > > > > to > > > > > > > > > do > > > > > > > > > another 4.1.0 release before proceeding with 5.0 to offer a > > > > > > > > > release > > > > > > > > > with the vulnerability fixed without requiring users to > > > > > > > > > upgrade > > > > > > > > > to > > > > > > > > > Kafka 4.0. Is there a reason you prefer to do the 4.1.0 > > > > > > > > > release > > > > > > > > > instead of the 4.0.1 release? I reviewed the changes between > > > > > > > > > the > > > > > > > > > current main and the release 4.0.0 [1], and they are mostly > > > > > > > > > dependency > > > > > > > > > upgrades and some fixes, but without any new features. What do > > > > > > > > > you > > > > > > > > > think about doing a 4.0.1 release and then kicking off 5.0.0 > > > > > > > > > with the > > > > > > > > > Kafka client upgrade? > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Fabian > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > https://github.com/apache/flink-connector-kafka/compare/v4.0...main > > > > > > > > > > > > > > > > > > On Fri, Jul 25, 2025 at 11:58 AM Tom Cooper [email protected] > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Bumping this thread as we are now ready to merge the Kafka > > > > > > > > > > 4.0.0 > > > > > > > > > > client update PR [1]. This will bump the major version of > > > > > > > > > > the > > > > > > > > > > connector to > > > > > > > > > > 5.0, as we are dropping support for Kafka brokers running > > > > > > > > > > version > > > > > > > > > > 2.0.0 or > > > > > > > > > > earlier. > > > > > > > > > > > > > > > > > > > > However, I still think it would be worth doing a 4.1.0 > > > > > > > > > > release > > > > > > > > > > of > > > > > > > > > > the connector (with the Kafka 3.9.1 client), before the > > > > > > > > > > Kafka 4.0.0 > > > > > > > > > > client > > > > > > > > > > update is merged. > > > > > > > > > > > > > > > > > > > > The current Flink Kafka Connector (4.0) has a critical CVE > > > > > > > > > > [2], > > > > > > > > > > which is patched in the 3.9.1 Kafka client library (which > > > > > > > > > > the current > > > > > > > > > > main > > > > > > > > > > branch of the Flink connector is using). Doing a 4.1 > > > > > > > > > > release of the > > > > > > > > > > connector would cover any users of older Kafka versions > > > > > > > > > > that want this > > > > > > > > > > CVE > > > > > > > > > > patched and also give a stable release of the connector > > > > > > > > > > using a point > > > > > > > > > > release of the Kafka client (with all the bug fixes that > > > > > > > > > > entails). This > > > > > > > > > > would be a good option for users who don't want to jump > > > > > > > > > > straight onto > > > > > > > > > > the > > > > > > > > > > new major Kafka client version. > > > > > > > > > > > > > > > > > > > > What do people think? > > > > > > > > > > > > > > > > > > > > Tom Cooper > > > > > > > > > > @tomcooper.dev | https://tomcooper.dev > > > > > > > > > > > > > > > > > > > > [1] https://github.com/apache/flink-connector-kafka/pull/161 > > > > > > > > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2025-27817 > > > > > > > > > > > > > > > > > > > > On Wednesday, 9 July 2025 at 09:35, Tom Cooper > > > > > > > > > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > I would like to start a conversation about releases for > > > > > > > > > > > the > > > > > > > > > > > Flink Connector Kafka project. > > > > > > > > > > > > > > > > > > > > > > We have recently updated [0] to version 3.9.1 of the Kafka > > > > > > > > > > > client library, which fixes a critical CVE [1]. With that > > > > > > > > > > > in mind, I > > > > > > > > > > > think > > > > > > > > > > > it would be prudent to have a 4.1.0 release as soon as > > > > > > > > > > > possible that > > > > > > > > > > > includes this. It would also be good to include the > > > > > > > > > > > dependency bumps > > > > > > > > > > > from > > > > > > > > > > > this PR [2] in that release. > > > > > > > > > > > > > > > > > > > > > > With the 4.1.0 release out, we could then move to looking > > > > > > > > > > > at > > > > > > > > > > > the > > > > > > > > > > > Kafka 4.0 upgrade (there is already a PR [3] for that). > > > > > > > > > > > The main point > > > > > > > > > > > with > > > > > > > > > > > the Kafka 4.0 upgrade is that it drops support for Kafka > > > > > > > > > > > brokers > > > > > > > > > > > running > > > > > > > > > > > version 2.0.0 and lower. Given this, I think it would > > > > > > > > > > > make sense to > > > > > > > > > > > move > > > > > > > > > > > the Connector version to 5.0.0 and maybe even move to > > > > > > > > > > > Flink 2.1.0 > > > > > > > > > > > (which > > > > > > > > > > > should be available in a month or so). This 5.0.0 release > > > > > > > > > > > could also > > > > > > > > > > > remove > > > > > > > > > > > all the Zookeeper specific test infra and move to KRaft > > > > > > > > > > > based clusters > > > > > > > > > > > for > > > > > > > > > > > testing. We could also move to a new, updated Flink > > > > > > > > > > > Connector Parent > > > > > > > > > > > pom > > > > > > > > > > > version [4] which would harmonise the java versions and > > > > > > > > > > > plugins with > > > > > > > > > > > the > > > > > > > > > > > main Flink project. > > > > > > > > > > > > > > > > > > > > > > I think, if the above is acceptable, that these changes > > > > > > > > > > > warrant > > > > > > > > > > > a major version bump. Users of older Kafka clusters would > > > > > > > > > > > still be > > > > > > > > > > > able to > > > > > > > > > > > use 4.1.0 (which is an argument for making sure that > > > > > > > > > > > release has the > > > > > > > > > > > most > > > > > > > > > > > up-to-date dependencies). > > > > > > > > > > > > > > > > > > > > > > Anyway, I would love to hear what the community think of > > > > > > > > > > > the > > > > > > > > > > > above. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > Tom Cooper > > > > > > > > > > > @tomcooper.dev | https://tomcooper.dev > > > > > > > > > > > > > > > > > > > > > > [0] > > > > > > > > > > > https://github.com/apache/flink-connector-kafka/pull/180 > > > > > > > > > > > [1] https://nvd.nist.gov/vuln/detail/CVE-2025-27817 > > > > > > > > > > > [2] > > > > > > > > > > > https://github.com/apache/flink-connector-kafka/pull/181 > > > > > > > > > > > [3] > > > > > > > > > > > https://github.com/apache/flink-connector-kafka/pull/161 > > > > > > > > > > > [4] > > > > > > > > > > > https://github.com/apache/flink-connector-shared-utils/pull/48 > > > > Unless otherwise stated above: > > > > IBM United Kingdom Limited > > Registered in England and Wales with number 741598 > > Registered office: Building C, IBM Hursley Office, Hursley Park Road, > > Winchester, Hampshire SO21 2JN > > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: Building C, IBM Hursley Office, Hursley Park Road, > Winchester, Hampshire SO21 2JN
