[
https://issues.apache.org/jira/browse/GEODE-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243475#comment-17243475
]
ASF GitHub Bot commented on GEODE-8696:
---------------------------------------
dschneider-pivotal commented on a change in pull request #5728:
URL: https://github.com/apache/geode/pull/5728#discussion_r535537581
##########
File path:
geode-core/src/main/java/org/apache/geode/management/internal/FederatingManager.java
##########
@@ -79,26 +82,39 @@
* unbounded in practical situation as number of members will be a finite
set at any given point
* of time
*/
- private ExecutorService executorService;
+ private final AtomicReference<ExecutorService> executorService = new
AtomicReference<>();
+ private boolean pendingStartup = true;
Review comment:
The concern I have with pendingStartup being set to true when
FederatingManager is created is that removeMember could build up a big list of
pending tasks if the call of startManager is delayed.
In some cases we are careful to call start right after create. But in
others, based on how the gemfire properties configure jmx, it can be delayed.
If you look in ManagementAdapter this code:` if
(cache.getInternalDistributedSystem().getConfig().getJmxManager()) {
service.createManager();
if
(cache.getInternalDistributedSystem().getConfig().getJmxManagerStart()) {
service.startManager();
}
}
`
will not always call startManager.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Startup of JMX Manager may hang during crash of other members
> -------------------------------------------------------------
>
> Key: GEODE-8696
> URL: https://issues.apache.org/jira/browse/GEODE-8696
> Project: Geode
> Issue Type: Bug
> Components: jmx, management
> Reporter: Kirk Lund
> Assignee: Kirk Lund
> Priority: Major
> Labels: GeodeOperationAPI, pull-request-available
>
> The fix for GEODE-7400 removed final from the executorService field to
> introduce a Supplier<ExecutorService>. I think this hang was caused by adding
> synchronized to FederatingManager.executeTask instead of making
> executorService a volatile field.
> _vm_3_thr_3_client2_host2_21145_ hung while synchronized on
> *0x00000000f6316520*:
> {noformat}
> "vm_3_thr_3_client2_host2_21145" #56 daemon prio=5 os_prio=0
> tid=0x00007f1854002000 nid=0x5326 waiting on condition [0x00007f18520e2000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000f6364030> (a
> java.util.concurrent.FutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:244)
> at
> org.apache.geode.management.internal.FederatingManager.startManagingActivity(FederatingManager.java:256)
> at
> org.apache.geode.management.internal.FederatingManager.startManager(FederatingManager.java:121)
> - locked <0x00000000f6316520> (a
> org.apache.geode.management.internal.FederatingManager)
> at
> org.apache.geode.management.internal.SystemManagementService.startManager(SystemManagementService.java:373)
> - locked <0x00000000fed91a70> (a java.util.HashMap)
> at
> org.apache.geode.management.internal.beans.ManagementAdapter.handleCacheCreation(ManagementAdapter.java:197)
> at
> org.apache.geode.management.internal.beans.ManagementListener.handleEvent(ManagementListener.java:127)
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.notifyResourceEventListeners(InternalDistributedSystem.java:2089)
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.handleResourceEvent(InternalDistributedSystem.java:643)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1363)
> at
> org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:191)
> - locked <0x00000000e11065f0> (a java.lang.Class for
> org.apache.geode.internal.cache.GemFireCacheImpl)
> - locked <0x00000000e1101220> (a java.lang.Class for
> org.apache.geode.internal.cache.InternalCacheBuilder)
> at
> org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:158)
> - locked <0x00000000e1101220> (a java.lang.Class for
> org.apache.geode.internal.cache.InternalCacheBuilder)
> at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:142)
> at
> hydra.CacheVersionHelper.configureAndCreateCache(CacheVersionHelper.java:51)
> at hydra.CacheHelper.createCacheWithHttpService(CacheHelper.java:127)
> - locked <0x00000000fecab060> (a java.lang.Class for
> hydra.CacheHelper)
> at hydra.CacheHelper.createCache(CacheHelper.java:87)
> at
> splitBrain.NetworkPartitionTest.initialize(NetworkPartitionTest.java:293)
> at
> splitBrain.NetworkPartitionTest.initializeInstance(NetworkPartitionTest.java:228)
> - locked <0x00000000e14a2db0> (a java.lang.Class for
> splitBrain.NetworkPartitionTest)
> at
> splitBrain.NetworkPartitionTest.HydraTask_initialize(NetworkPartitionTest.java:203)
> - locked <0x00000000e14a2db0> (a java.lang.Class for
> splitBrain.NetworkPartitionTest)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at hydra.MethExecutor.execute(MethExecutor.java:173)
> at hydra.MethExecutor.execute(MethExecutor.java:141)
> at hydra.TestTask.execute(TestTask.java:197)
> at hydra.RemoteTestModule$1.run(RemoteTestModule.java:213)
> {noformat}
> _vm_3_thr_3_client2_host2_21145_ is waiting for _FederatingManager1_ to
> complete:
> {noformat}
> "FederatingManager1" #66 daemon prio=5 os_prio=0 tid=0x00007f1874327800
> nid=0x5331 waiting on condition [0x00007f18515d9000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000f63b2590> (a
> java.util.concurrent.CountDownLatch$Sync)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at
> org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:731)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForReplies(ReplyProcessor21.java:639)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForReplies(ReplyProcessor21.java:620)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForReplies(ReplyProcessor21.java:534)
> at
> org.apache.geode.internal.cache.StateFlushOperation.flush(StateFlushOperation.java:244)
> at
> org.apache.geode.internal.cache.InitialImageOperation.getFromOne(InitialImageOperation.java:432)
> at
> org.apache.geode.internal.cache.DistributedRegion.getInitialImageAndRecovery(DistributedRegion.java:1236)
> at
> org.apache.geode.internal.cache.DistributedRegion.initialize(DistributedRegion.java:1082)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:2971)
> at
> org.apache.geode.internal.cache.InternalCacheForClientAccess.createInternalRegion(InternalCacheForClientAccess.java:255)
> at
> org.apache.geode.management.internal.FederatingManager.addMemberArtifacts(FederatingManager.java:448)
> - locked <0x00000000fec1a480> (a
> org.apache.geode.distributed.internal.membership.InternalDistributedMember)
> at
> org.apache.geode.management.internal.FederatingManager$GIITask.call(FederatingManager.java:560)
> at
> org.apache.geode.management.internal.FederatingManager$GIITask.call(FederatingManager.java:550)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> _DM-MemberEventInvoker_ is preventing _FederatingManager1_ from completing
> its call to {{ReplyProcessor21.basicWait}} while waiting to lock
> *0x00000000f6316520* which is held by _vm_3_thr_3_client2_host2_21145_:
> {noformat}
> "DM-MemberEventInvoker" #28 daemon prio=5 os_prio=0 tid=0x00007f1860356000
> nid=0x5313 waiting for monitor entry [0x00007f18531f2000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.geode.management.internal.FederatingManager.executeTask(FederatingManager.java:230)
> - waiting to lock <0x00000000f6316520> (a
> org.apache.geode.management.internal.FederatingManager)
> at
> org.apache.geode.management.internal.FederatingManager.removeMember(FederatingManager.java:160)
> at
> org.apache.geode.management.internal.ManagementMembershipListener.memberDeparted(ManagementMembershipListener.java:57)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberCrashedEvent.handleEvent(ClusterDistributionManager.java:2519)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:2424)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:2413)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:1401)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.access$200(ClusterDistributionManager.java:108)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:1433)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
