[ https://issues.apache.org/jira/browse/GEODE-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16137473#comment-16137473 ]
ASF GitHub Bot commented on GEODE-3276: --------------------------------------- GitHub user nabarunnag opened a pull request: https://github.com/apache/geode/pull/732 GEODE-3276: Managing race conditions while the senders are stopped * When a connection is initialized, a readAckThread may be alive from a previous incarnation. * This AckThread will be stuck on a read socket with no timeout as nothing was dispatched. * Also while it was stuck on the read, it will hold a connection lifecycle read lock * The initialize connection needs a connection life cycle write lock to start the connection but the read lock is held by the ack thread. * This results in a deadlock and eventually a hang. * Another situation is that we set the flag isStopped for the event processor before actually shutting down the diapatcher and ack thread. * So after the flag is set and before actually shutting down the dispatcher and ackThread, a gateway proxy stomper thread gets in between these two steps of execution. * The stomper thread checks the isStopped flag, which was set to true, and proceeds to destroy the connection pool. However the dispatcher and ackThread were still running. * This results in a out of heap memory exception while the ack thread is reading from the socket while connection pool was destroyed. * To solve this issue, the stomper thread checks if the event processor and dispatcher exists, if true then we close the input streams before destroying the connection pool. Thank you for submitting a contribution to Apache Geode. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Has your PR been rebased against the latest commit within the target branch (typically `develop`)? - [ ] Is your initial contribution a single, squashed commit? - [ ] Does `gradlew build` run cleanly? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. If you need help, please send an email to d...@geode.apache.org. You can merge this pull request into a Git repository by running: $ git pull https://github.com/nabarunnag/incubator-geode feature/GEODE-3276 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/geode/pull/732.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #732 ---- commit 9618f69d2620f5a3d3a6a8576631906a61b512f9 Author: nabarun <n...@pivotal.io> Date: 2017-08-17T17:51:15Z GEODE-3276: Managing race conditions while the senders are stopped * When a connection is initialized, a readAckThread may be alive from a previous incarnation. * This AckThread will be stuck on a read socket with no timeout as nothing was dispatched. * Also while it was stuck on the read, it will hold a connection lifecycle read lock * The initialize connection needs a connection life cycle write lock to start the connection but the read lock is held by the ack thread. * This results in a deadlock and eventually a hang. * Another situation is that we set the flag isStopped for the event processor before actually shutting down the diapatcher and ack thread. * So after the flag is set and before actually shutting down the dispatcher and ackThread, a gateway proxy stomper thread gets in between these two steps of execution. * The stomper thread checks the isStopped flag, which was set to true, and proceeds to destroy the connection pool. However the dispatcher and ackThread were still running. * This results in a out of heap memory exception while the ack thread is reading from the socket while connection pool was destroyed. * To solve this issue, the stomper thread checks if the event processor and dispatcher exists, if true then we close the input streams before destroying the connection pool. ---- > CI failure: > org.apache.geode.internal.cache.wan.parallel.ParallelWANStatsDUnitTest > > testParallelPropagationWithRemoteRegionDestroy FAILED > ------------------------------------------------------------------------------------------------------------------------------------------ > > Key: GEODE-3276 > URL: https://issues.apache.org/jira/browse/GEODE-3276 > Project: Geode > Issue Type: Bug > Components: statistics, wan > Affects Versions: 1.2.0 > Reporter: Shelley Lynn Hughes-Godfrey > Assignee: nabarun > > {noformat} > org.apache.geode.internal.cache.wan.parallel.ParallelWANStatsDUnitTest > > testParallelPropagationWithRemoteRegionDestroy FAILED > java.lang.AssertionError: An exception occurred during asynchronous > invocation. > Caused by: > java.lang.OutOfMemoryError: Java heap space > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)