[ https://issues.apache.org/jira/browse/RATIS-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17852521#comment-17852521 ]
Duong commented on RATIS-2104: ------------------------------ [~szetszwo] Even if LogAppender is still running in MiniRaftCluster.shutdown(), once GrpcServerProtocol receives an appendEntries request, it should at least finish processing that request. I added some naive logs like the following: {code:java} public CompletableFuture<AppendEntriesReplyProto> appendEntriesAsync( ReferenceCountedObject<AppendEntriesRequestProto> requestRef) { AppendEntriesRequestProto request = requestRef.retain(); LOG.info("Starting {}", request.hashCode()); try { final RaftGroupId groupId = ProtoUtils.toRaftGroupId(request.getServerRequest().getRaftGroupId()); return getImplFuture(groupId) .thenCompose(impl -> impl.executeSubmitServerRequestAsync(() -> impl.appendEntriesAsync(requestRef))); } finally { LOG.info("Ending {}", request.hashCode()); requestRef.release(); } } {code} I can confirm that when the MiniRaftCluster is shutting down (JVM doesn't exist yet), the "finally" block is sometimes skipped for the last couple of requests and the tests fail due to leak detection assertion. {code:java} Starting 1697315093 .... // no Ending LEAK: A org.apache.ratis.proto.RaftProtos$AppendEntriesRequestProto-1697315093 is not released properly.{code} The guarantee that "finally" blocks always run (unless JVM exit) is the basic assumption for the correctness around ReferenceCountedObject. > TestLeaderInstallSnapshot may fail with java.lang.IllegalStateException: > allLeaks.size = 4 > ------------------------------------------------------------------------------------------ > > Key: RATIS-2104 > URL: https://issues.apache.org/jira/browse/RATIS-2104 > Project: Ratis > Issue Type: Bug > Reporter: Tsz-wo Sze > Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > I can reproduce it with ratis-3.1.0-rc0 > {code} > [INFO] Running org.apache.ratis.grpc.TestLeaderInstallSnapshot > [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: > 65.892 s <<< FAILURE! - in org.apache.ratis.grpc.TestLeaderInstallSnapshot > [ERROR] > org.apache.ratis.grpc.TestLeaderInstallSnapshot.testInstallSnapshotLeaderSwitch(Boolean)[1] > Time elapsed: 20.055 s <<< ERROR! > java.lang.IllegalStateException: allLeaks.size = 4 > at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:77) > at > org.apache.ratis.util.LeakDetector.assertNoLeaks(LeakDetector.java:107) > at > org.apache.ratis.server.impl.MiniRaftCluster.shutdown(MiniRaftCluster.java:869) > at > org.apache.ratis.grpc.MiniRaftClusterWithGrpc.shutdown(MiniRaftClusterWithGrpc.java:93) > at > org.apache.ratis.server.impl.MiniRaftCluster$Factory$Get.runWithNewCluster(MiniRaftCluster.java:149) > at > org.apache.ratis.server.impl.MiniRaftCluster$Factory$Get.runWithNewCluster(MiniRaftCluster.java:121) > at > org.apache.ratis.InstallSnapshotFromLeaderTests.testInstallSnapshotLeaderSwitch(InstallSnapshotFromLeaderTests.java:94) > at > org.apache.ratis.grpc.TestLeaderInstallSnapshot.testInstallSnapshotLeaderSwitch(TestLeaderInstallSnapshot.java:53) > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)