[ https://issues.apache.org/jira/browse/RATIS-840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
runzhiwang updated RATIS-840: ----------------------------- Description: *What's the problem ?* When run hadoop-ozone for 4 days, datanode memory leak. When dump heap, I found there are 460710 instances of GrpcLogAppender. But there are only 6 instances of SenderList, and each SenderList contains 1-2 instance of GrpcLogAppender. So there are a lot of GrpcLogAppender did not stop the Daemon Thread when removed from senders. !image-2020-04-06-14-27-28-485.png! !image-2020-04-06-14-27-39-582.png! *What's the reason ?* >From the code, when >[removeSender|https://github.com/apache/incubator-ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderState.java#L431], > it did not call >[LogAppender::stopAppender|https://github.com/apache/incubator-ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/impl/LogAppender.java#L164]. *How to fix ?* To avoid forgetting stopAppender, I stopAppender in [SenderList ::removeAll|https://github.com/apache/incubator-ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderState.java#L173]. was: *What's the problem ?* When run hadoop-ozone for 4 days, datanode memory leak. When dump heap, I found there are 460710 instances of GrpcLogAppender. But there are only 6 instances of SenderList, and each SenderList contains 1-2 instance of GrpcLogAppender. So there are a lot of GrpcLogAppender did not stop the Daemon Thread when removed from senders. !image-2020-04-06-14-27-28-485.png! !image-2020-04-06-14-27-39-582.png! *What's the reason ?* >From the code, when >[removeSender|https://github.com/apache/incubator-ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderState.java#L431], > it did not call LogAppender::stopAppender. *How to fix ?* To avoid forgetting stopAppender, I stopAppender in [SenderList ::removeAll|https://github.com/apache/incubator-ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderState.java#L173]. > Memory leak of LogAppender > -------------------------- > > Key: RATIS-840 > URL: https://issues.apache.org/jira/browse/RATIS-840 > Project: Ratis > Issue Type: Bug > Reporter: runzhiwang > Priority: Major > Attachments: RATIS-840.001.patch, image-2020-04-06-14-27-28-485.png, > image-2020-04-06-14-27-39-582.png > > > *What's the problem ?* > When run hadoop-ozone for 4 days, datanode memory leak. When dump heap, I > found there are 460710 instances of GrpcLogAppender. But there are only 6 > instances of SenderList, and each SenderList contains 1-2 instance of > GrpcLogAppender. So there are a lot of GrpcLogAppender did not stop the > Daemon Thread when removed from senders. > !image-2020-04-06-14-27-28-485.png! > !image-2020-04-06-14-27-39-582.png! > > *What's the reason ?* > From the code, when > [removeSender|https://github.com/apache/incubator-ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderState.java#L431], > it did not call > [LogAppender::stopAppender|https://github.com/apache/incubator-ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/impl/LogAppender.java#L164]. > > *How to fix ?* > To avoid forgetting stopAppender, I stopAppender in [SenderList > ::removeAll|https://github.com/apache/incubator-ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderState.java#L173]. -- This message was sent by Atlassian Jira (v8.3.4#803005)