Aled Sage created BROOKLYN-66:
---------------------------------
Summary: Deadlock in group.addMember() when unmanaging other
member of group
Key: BROOKLYN-66
URL: https://issues.apache.org/jira/browse/BROOKLYN-66
Project: Brooklyn
Issue Type: Bug
Affects Versions: 0.7.0-M1
Reporter: Aled Sage
When running `ServiceReplacerTest.testSetsOnFireWhenFailToReplaceMember` lots
of times, I hit this deadlock.
In brief...
When method is synchronized, hit deadlock:
1. thread called unmanage() on a member of a group, so we got the lock on
LocalEntityManager
and called group.removeMember; this ties to synchronize on
AbstractGroupImpl.members
2. another thread was doing AbstractGroupImpl.addMember, which is synchronized
on AbstractGroupImpl.members;
it tries to call Entities.manage(child) which calls
LocalEntityManager.getEntity(), which is
synchronized on this.
We MUST NOT call alien code from within the management framework while holding
locks.
The AbstractGroup.removeMember is effectively alien because a user could
override it, and because
it is entity specific. Therefore this should be fixed in LocalEntityManager.
Found one Java-level deadlock:
=============================
"brooklyn-execmanager-Lol69DXC-10":
waiting to lock monitor 7f9774943fc0 (object 7f3303f70, a
brooklyn.management.internal.LocalEntityManager),
which is held by "brooklyn-execmanager-Lol69DXC-2"
"brooklyn-execmanager-Lol69DXC-2":
waiting to lock monitor 7f977a88b5f0 (object 7f36e75d0, a
brooklyn.util.collections.SetFromLiveMap),
which is held by "brooklyn-execmanager-Lol69DXC-10"
Java stack information for the threads listed above:
===================================================
"brooklyn-execmanager-Lol69DXC-10":
at
brooklyn.management.internal.LocalEntityManager.getEntity(LocalEntityManager.java:198)
- waiting to lock <7f3303f70> (a
brooklyn.management.internal.LocalEntityManager)
at
brooklyn.management.internal.LocalEntityManager.isManaged(LocalEntityManager.java:207)
at
brooklyn.management.internal.LocalEntityManager.manage(LocalEntityManager.java:235)
at brooklyn.entity.basic.Entities.manage(Entities.java:768)
at
brooklyn.entity.basic.AbstractGroupImpl.addMember(AbstractGroupImpl.java:132)
- locked <7f36e75d0> (a brooklyn.util.collections.SetFromLiveMap)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:104)
at com.sun.proxy.$Proxy57.addMember(Unknown Source)
at
brooklyn.entity.group.DynamicClusterImpl.quarantineFailedNodes(DynamicClusterImpl.java:683)
at
brooklyn.entity.group.DynamicClusterImpl.addInEachLocation(DynamicClusterImpl.java:670)
at
brooklyn.entity.group.DynamicClusterImpl.addInSingleLocation(DynamicClusterImpl.java:615)
at
brooklyn.entity.group.DynamicClusterImpl.replaceMember(DynamicClusterImpl.java:484)
- locked <7f34800e8> (a [Ljava.lang.Object;)
at
brooklyn.entity.group.DynamicClusterImpl.replaceMember(DynamicClusterImpl.java:474)
- locked <7f34800e8> (a [Ljava.lang.Object;)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
at
groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
at
brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.groovy:144)
at
brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:251)
at
brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:274)
at
brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:238)
at brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:149)
at brooklyn.entity.basic.AbstractEffector.call(AbstractEffector.java:64)
at
brooklyn.entity.basic.AbstractEffector$1$1.call(AbstractEffector.java:83)
at
brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:318)
at
brooklyn.util.task.BasicExecutionManager$2.call(BasicExecutionManager.java:389)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
"brooklyn-execmanager-Lol69DXC-2":
at
brooklyn.entity.basic.AbstractGroupImpl.removeMember(AbstractGroupImpl.java:148)
- waiting to lock <7f36e75d0> (a
brooklyn.util.collections.SetFromLiveMap)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:104)
at com.sun.proxy.$Proxy57.removeMember(Unknown Source)
at
brooklyn.management.internal.LocalEntityManager.unmanageNonRecursive(LocalEntityManager.java:408)
- locked <7f3303f70> (a brooklyn.management.internal.LocalEntityManager)
at
brooklyn.management.internal.LocalEntityManager.unmanage(LocalEntityManager.java:284)
at brooklyn.entity.basic.Entities.unmcaanage(Entities.java:851)
at
brooklyn.entity.group.DynamicClusterImpl.discardNode(DynamicClusterImpl.java:836)
at
brooklyn.entity.group.DynamicClusterImpl.shrink(DynamicClusterImpl.java:609)
at
brooklyn.entity.group.DynamicClusterImpl.stop(DynamicClusterImpl.java:393)
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
at
groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
at
brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.groovy:144)
at
brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:251)
at
brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:274)
at
brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:238)
at
brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:102)
at com.sun.proxy.$Proxy56.stop(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:166)
at brooklyn.entity.basic.AbstractEffector.call(AbstractEffector.java:64)
at
brooklyn.entity.basic.AbstractEffector$1$1.call(AbstractEffector.java:83)
at
brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:318)
at
brooklyn.util.task.BasicExecutionManager$2.call(BasicExecutionManager.java:389)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Found 1 deadlock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
