[
https://issues.apache.org/jira/browse/HBASE-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854826#action_12854826
]
stack commented on HBASE-2322:
------------------------------
I just ran a PE sequentialWrite against small cluster using 0.20_pre_durability
branch and deadlocked pretty soon into the job. See below.
{code}
Java stack information for the threads listed above:
===================================================
"IPC Server handler 59 on 60020":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f8d8a3a30c8> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1292)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1274)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1794)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
"IPC Server handler 45 on 60020":
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.request(MemStoreFlusher.java:172)
- waiting to lock <0x00007f8d71145b08> (a java.util.HashSet)
at
org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1543)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1528)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1311)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1274)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1794)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
"regionserver/10.20.20.189:60020.cacheFlusher":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f8d7113f0d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:253)
- locked <0x00007f8d71145b08> (a java.util.HashSet)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)
"IPC Server handler 31 on 60020":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f8d8a3a30c8> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:972)
at
org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:865)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:259)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushSomeRegions(MemStoreFlusher.java:379)
- locked <0x00007f8d7111ba30> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:348)
- locked <0x00007f8d7111ba30> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1788)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Found 1 deadlock.
{code}
> deadlock between put and cacheflusher in 0.20 branch
> ----------------------------------------------------
>
> Key: HBASE-2322
> URL: https://issues.apache.org/jira/browse/HBASE-2322
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.20.4, 0.21.0
>
> Attachments: hbase-2322.png
>
>
> {code}
> Found one Java-level deadlock:
> =============================
> "IPC Server handler 59 on 60020":
> waiting for ownable synchronizer 0x00007fec9eb050f8, (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
> which is held by "IPC Server handler 54 on 60020"
> "IPC Server handler 54 on 60020":
> waiting to lock monitor 0x000000004190e950 (object 0x00007fec64f25258, a
> java.util.HashSet),
> which is held by "regionserver/10.20.20.186:60020.cacheFlusher"
> "regionserver/10.20.20.186:60020.cacheFlusher":
> waiting for ownable synchronizer 0x00007fec651df998, (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync),
> which is held by "IPC Server handler 19 on 60020"
> "IPC Server handler 19 on 60020":
> waiting for ownable synchronizer 0x00007fec9eb050f8, (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
> which is held by "IPC Server handler 54 on 60020"
> Java stack information for the threads listed above:
> ===================================================
> "IPC Server handler 59 on 60020":
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007fec9eb050f8> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
> at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1299)
> at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1281)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1789)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> "IPC Server handler 54 on 60020":
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.request(MemStoreFlusher.java:172)
> - waiting to lock <0x00007fec64f25258> (a java.util.HashSet)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1549)
> at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1534)
> at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
> at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1281)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1789)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> "regionserver/10.20.20.186:60020.cacheFlusher":
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007fec651df998> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
> at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:235)
> - locked <0x00007fec64f25258> (a java.util.HashSet)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)
> "IPC Server handler 19 on 60020":
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007fec9eb050f8> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:980)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:873)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushSomeRegions(MemStoreFlusher.java:352)
> - locked <0x00007fec64ed96f0> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:321)
> - locked <0x00007fec64ed96f0> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1783)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> Found 1 deadlock.
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
