Thanks Herman,
I would like to clarify: while the observed issue is in AK 4.1.0, we
need to verify if 4.0.1 might introduce the same issue. That's why we
wanted to report this early.
Herman told me, that they will test 4.0.1 and report back.
We do suspect https://issues.apache.org/jira/browse/KAFKA-19398 to be
the root cause, and the commit for this ticket is only in 4.1.0. But
it's not confirmed that it really the root cause.
We also have a fix with
https://issues.apache.org/jira/browse/KAFKA-19679 which we merged into
4.1 branch for 4.1.1 release.
But the fix is not confirmed either at this point. Herman also wanted to
test 4.1.1-SNAPSHOT to confirm if the issue is fixed there.
So at this point, it's all just an FYI, and Herman told me he will
report back in the next couple of days about what they found.
-Matthias
On 9/11/25 10:31 AM, Herman K. Jakobsen wrote:
Hi,
I have found a memory leak in Kafka 4.1.0, and was asked by @mjsax to
file the issue in this thread.
Relevant Slack thread can be found here: https://
confluentcommunity.slack.com/archives/C48AHTCUQ/p1757431640123829
*In summary*
By running a Kafka Stream application implementing a Stream-GlobalKtable
join, there has been found situations where the memory and CPU usage are
continuously increasing. By doing a heap dump of the running
application, it was shown that
`org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl` has a `storeLevel` map that grows indefinitely.
I have attached screenshot of the application´s
`kafka_stream_thread_process_latency_avg` metric showing the application
running slower and slower until it hits an OutOfmemoryException and then
restarts. The other screenshots contains the results of the analysed
heap dump.
Best regards,
Herman
heap_class.pngheap_path.pngproblem_suspect_1.pngprocess.png
On 2025/09/11 10:09:00 Christo Lolov wrote:
> Hello Kafka users, developers and client-developers,
>
> This is the third candidate for release of Apache Kafka 4.0.1.
>
> Since this is a patch version it contains the bug fixes outlined in the
> release notes below!
>
> Release notes for the 4.0.1 release:
> https://dist.apache.org/repos/dist/dev/kafka/4.0.1-rc2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Wednesday September 17.
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://dist.apache.org/repos/dist/dev/kafka/4.0.1-rc2/
>
> * Docker release artifacts to be voted upon:
> apache/kafka:4.0.1-rc2
> apache/kafka-native:4.0.1-rc2
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://dist.apache.org/repos/dist/dev/kafka/4.0.1-rc2/javadoc/
>
> * Tag to be voted upon (off 4.0 branch) is the 4.0.1 tag:
> https://github.com/apache/kafka/releases/tag/4.0.1-rc2
>
> * Successful CI builds for the 4.0 branch:
> Unit/integration tests:
> https://github.com/apache/kafka/actions/runs/17586456207
>
> * Successful Docker Image Github Actions Pipeline for 4.0 branch:
> Docker Build Test Pipeline (JVM):
> https://github.com/apache/kafka/actions/runs/17586731957
> Docker Build Test Pipeline (Native):
> https://github.com/apache/kafka/actions/runs/17587113226
>
> * Documentation updates for website and Javadocs
> - https://github.com/apache/kafka-site/pull/719
> - https://github.com/apache/kafka-site/pull/723
>
> Thanks,
> Christo
>
