keith-turner opened a new issue, #5597:
URL: https://github.com/apache/accumulo/issues/5597
**Describe the bug**
Sometimes when running ChaoticBalancerIT the following deadlock will happen
in a tablet server.
```
Found one Java-level deadlock:
=============================
"accumulo.pool.general.server-Worker-2":
waiting to lock monitor 0x00007f672003c100 (object 0x00000000f4c0c2c8, a
org.apache.accumulo.tserver.tablet.Tablet),
which is held by "Minor compacting +r<<"
"Minor compacting +r<<":
waiting for ownable synchronizer 0x00000000f4c5cda8, (a
java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "accumulo.pool.tabletserver.client-Worker-1"
"accumulo.pool.tabletserver.client-Worker-1":
waiting to lock monitor 0x00007f672003c100 (object 0x00000000f4c0c2c8, a
org.apache.accumulo.tserver.tablet.Tablet),
which is held by "Minor compacting +r<<"
Java stack information for the threads listed above:
===================================================
"accumulo.pool.general.server-Worker-2":
at
org.apache.accumulo.tserver.tablet.Tablet.updateRates(Tablet.java:1146)
- waiting to lock <0x00000000f4c0c2c8> (a
org.apache.accumulo.tserver.tablet.Tablet)
at
org.apache.accumulo.tserver.TabletServer.lambda$new$0(TabletServer.java:288)
at
org.apache.accumulo.tserver.TabletServer$$Lambda$342/0x0000000840349c40.run(Unknown
Source)
at
org.apache.accumulo.core.util.threads.NamedRunnable.run(NamedRunnable.java:41)
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at
java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:515)
at
java.util.concurrent.FutureTask.runAndReset([email protected]/FutureTask.java:305)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run([email protected]/ScheduledThreadPoolExecutor.java:305)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at java.lang.Thread.run([email protected]/Thread.java:829)
"Minor compacting +r<<":
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000000f4c5cda8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at
java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt([email protected]/AbstractQueuedSynchronizer.java:885)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued([email protected]/AbstractQueuedSynchronizer.java:917)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:1240)
at
java.util.concurrent.locks.ReentrantLock.lock([email protected]/ReentrantLock.java:267)
at
org.apache.accumulo.tserver.tablet.Tablet.bringMinorCompactionOnline(Tablet.java:1562)
at
org.apache.accumulo.tserver.tablet.Tablet.minorCompact(Tablet.java:409)
at
org.apache.accumulo.tserver.tablet.MinorCompactionTask.run(MinorCompactionTask.java:114)
at
org.apache.accumulo.tserver.tablet.Tablet.completeClose(Tablet.java:966)
- locked <0x00000000f4c0c2c8> (a
org.apache.accumulo.tserver.tablet.Tablet)
at org.apache.accumulo.tserver.tablet.Tablet.close(Tablet.java:787)
at
org.apache.accumulo.tserver.UnloadTabletHandler.run(UnloadTabletHandler.java:90)
at
org.apache.accumulo.tserver.TabletServerResourceManager.addMigration(TabletServerResourceManager.java:858)
at
org.apache.accumulo.tserver.TabletClientHandler.unloadTablet(TabletClientHandler.java:1078)
at jdk.internal.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke([email protected]/Method.java:566)
at
org.apache.accumulo.core.trace.TraceUtil.lambda$wrapService$0(TraceUtil.java:203)
at
org.apache.accumulo.core.trace.TraceUtil$$Lambda$372/0x000000084035d040.invoke(Unknown
Source)
at com.sun.proxy.$Proxy27.unloadTablet(Unknown Source)
at
org.apache.accumulo.core.tablet.thrift.TabletManagementClientService$Processor$unloadTablet.getResult(TabletManagementClientService.java:347)
at
org.apache.accumulo.core.tablet.thrift.TabletManagementClientService$Processor$unloadTablet.getResult(TabletManagementClientService.java:325)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:40)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40)
at
org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:147)
at
org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:50)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:492)
at
org.apache.accumulo.server.rpc.CustomNonBlockingServer$CustomFrameBuffer.invoke(CustomNonBlockingServer.java:129)
at org.apache.thrift.server.Invocation.run(Invocation.java:18)
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at java.lang.Thread.run([email protected]/Thread.java:829)
"accumulo.pool.tabletserver.client-Worker-1":
at
org.apache.accumulo.tserver.tablet.Tablet.refreshMetadata(Tablet.java:1684)
- waiting to lock <0x00000000f4c0c2c8> (a
org.apache.accumulo.tserver.tablet.Tablet)
at
org.apache.accumulo.tserver.tablet.Tablet$RefreshSession.refreshMetadata(Tablet.java:1634)
at
org.apache.accumulo.tserver.TabletClientHandler.refreshTablets(TabletClientHandler.java:1226)
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0([email protected]/Native
Method)
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke([email protected]/NativeMethodAccessorImpl.java:62)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke([email protected]/Method.java:566)
at
org.apache.accumulo.core.trace.TraceUtil.lambda$wrapService$0(TraceUtil.java:203)
at
org.apache.accumulo.core.trace.TraceUtil$$Lambda$372/0x000000084035d040.invoke(Unknown
Source)
at com.sun.proxy.$Proxy27.refreshTablets(Unknown Source)
at
org.apache.accumulo.core.tabletserver.thrift.TabletServerClientService$Processor$refreshTablets.getResult(TabletServerClientService.java:1596)
at
org.apache.accumulo.core.tabletserver.thrift.TabletServerClientService$Processor$refreshTablets.getResult(TabletServerClientService.java:1573)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:40)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40)
at
org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:147)
at
org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:50)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:492)
at
org.apache.accumulo.server.rpc.CustomNonBlockingServer$CustomFrameBuffer.invoke(CustomNonBlockingServer.java:129)
at org.apache.thrift.server.Invocation.run(Invocation.java:18)
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at java.lang.Thread.run([email protected]/Thread.java:829)
Found 1 deadlock.
```
**To Reproduce**
Repeatedly run ChaoticBalancerIT with the fixes in #5596 against
82e9891610b985edce604cebcc65dbf9029be3c1 until the test hangs then jstack the
two tablet servers.
**Expected behavior**
No deadlock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
