Unfortunately, due to my organization's policies and setup, I'm not able to test neither the RC nor the SNAPSHOT. We are currently running over 50 Kafka Streams applications, and the OOM symptoms are most dominant in applications using a GlobalStore and having input topics with hundreds of thousands records per minute. As stated before, reverting from 4.1.0 to 4.0.0 removes the OOM issue.
tor. 11. sep. 2025 kl. 20:18 skrev Matthias J. Sax <[email protected]>: > Thanks Herman, > > I would like to clarify: while the observed issue is in AK 4.1.0, we > need to verify if 4.0.1 might introduce the same issue. That's why we > wanted to report this early. > > Herman told me, that they will test 4.0.1 and report back. > > We do suspect https://issues.apache.org/jira/browse/KAFKA-19398 to be > the root cause, and the commit for this ticket is only in 4.1.0. But > it's not confirmed that it really the root cause. > > We also have a fix with > https://issues.apache.org/jira/browse/KAFKA-19679 which we merged into > 4.1 branch for 4.1.1 release. > > But the fix is not confirmed either at this point. Herman also wanted to > test 4.1.1-SNAPSHOT to confirm if the issue is fixed there. > > So at this point, it's all just an FYI, and Herman told me he will > report back in the next couple of days about what they found. > > > -Matthias > > On 9/11/25 10:31 AM, Herman K. Jakobsen wrote: > > Hi, > > > > I have found a memory leak in Kafka 4.1.0, and was asked by @mjsax to > > file the issue in this thread. > > > > Relevant Slack thread can be found here: https:// > > confluentcommunity.slack.com/archives/C48AHTCUQ/p1757431640123829 > > > > *In summary* > > By running a Kafka Stream application implementing a Stream-GlobalKtable > > join, there has been found situations where the memory and CPU usage are > > continuously increasing. By doing a heap dump of the running > > application, it was shown that > > > `org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl` > has a `storeLevel` map that grows indefinitely. > > > > I have attached screenshot of the application´s > > `kafka_stream_thread_process_latency_avg` metric showing the application > > running slower and slower until it hits an OutOfmemoryException and then > > restarts. The other screenshots contains the results of the analysed > > heap dump. > > > > Best regards, > > Herman > > > > heap_class.pngheap_path.pngproblem_suspect_1.pngprocess.png > > > > > > > > On 2025/09/11 10:09:00 Christo Lolov wrote: > > > Hello Kafka users, developers and client-developers, > > > > > > This is the third candidate for release of Apache Kafka 4.0.1. > > > > > > Since this is a patch version it contains the bug fixes outlined in > the > > > release notes below! > > > > > > Release notes for the 4.0.1 release: > > > > https://dist.apache.org/repos/dist/dev/kafka/4.0.1-rc2/RELEASE_NOTES.html > > > > > > *** Please download, test and vote by Wednesday September 17. > > > > > > Kafka's KEYS file containing PGP keys we use to sign the release: > > > https://kafka.apache.org/KEYS > > > > > > * Release artifacts to be voted upon (source and binary): > > > https://dist.apache.org/repos/dist/dev/kafka/4.0.1-rc2/ > > > > > > * Docker release artifacts to be voted upon: > > > apache/kafka:4.0.1-rc2 > > > apache/kafka-native:4.0.1-rc2 > > > > > > * Maven artifacts to be voted upon: > > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > > > * Javadoc: > > > https://dist.apache.org/repos/dist/dev/kafka/4.0.1-rc2/javadoc/ > > > > > > * Tag to be voted upon (off 4.0 branch) is the 4.0.1 tag: > > > https://github.com/apache/kafka/releases/tag/4.0.1-rc2 > > > > > > * Successful CI builds for the 4.0 branch: > > > Unit/integration tests: > > > https://github.com/apache/kafka/actions/runs/17586456207 > > > > > > * Successful Docker Image Github Actions Pipeline for 4.0 branch: > > > Docker Build Test Pipeline (JVM): > > > https://github.com/apache/kafka/actions/runs/17586731957 > > > Docker Build Test Pipeline (Native): > > > https://github.com/apache/kafka/actions/runs/17587113226 > > > > > > * Documentation updates for website and Javadocs > > > - https://github.com/apache/kafka-site/pull/719 > > > - https://github.com/apache/kafka-site/pull/723 > > > > > > Thanks, > > > Christo > > > > >
