[
https://issues.apache.org/jira/browse/KAFKA-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422805#comment-15422805
]
ASF GitHub Bot commented on KAFKA-3937:
---------------------------------------
GitHub user wiyu opened a pull request:
https://github.com/apache/kafka/pull/1743
KAFKA-3937: Kafka Clients Leak Native Memory For Longer Than Needed With
Compressed Messages
@ijuma - Creating this PR against 0.9.0 as this is what we're using in
prod. I can modify this for trunk if the logic solution looks good.
I modified the record iterator to inherit from closable and implemented the
method to call streams.close(). The stream will now be closed when we are done
traversing the inner iterator. I had to move the `try {} catch{}` out of the
`if` portion to deal with IOException coming from `innerDone()`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wiyu/kafka compressor_memory_leak_in_fetcher
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/1743.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1743
----
commit b56ec1563e34d40c338224970c6b8cd3802e4dcd
Author: William Yu <[email protected]>
Date: 2016-08-16T14:09:28Z
KAFKA-3937: Kafka Clients Leak Native Memory For Longer Than Needed With
Compressed Messages
----
> Kafka Clients Leak Native Memory For Longer Than Needed With Compressed
> Messages
> --------------------------------------------------------------------------------
>
> Key: KAFKA-3937
> URL: https://issues.apache.org/jira/browse/KAFKA-3937
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Affects Versions: 0.8.2.2, 0.9.0.1, 0.10.0.0
> Environment: Linux, latest oracle java-8
> Reporter: Tom Crayford
> Priority: Minor
>
> In https://issues.apache.org/jira/browse/KAFKA-3933, we discovered that
> brokers can crash when performing log recovery, as they leak native memory
> whilst decompressing compressed segments, and that native memory isn't
> cleaned up rapidly enough by garbage collection and finalizers. The work to
> fix that in the brokers is taking part in
> https://github.com/apache/kafka/pull/1598. As part of that PR, Ismael Juma
> asked me to fix similar issues in the client. Rather than have one large PR
> that fixes everything, I'd rather break this work up into seperate things, so
> I'm filing this JIRA to track the followup work. I should get to a PR on this
> at some point relatively soon, once the other PR has landed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
