[
https://issues.apache.org/jira/browse/HBASE-23076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938398#comment-16938398
]
Wellington Chevreuil commented on HBASE-23076:
----------------------------------------------
Thanks for finding and reviewing it, [~elserj]!
> [HBOSS] ZKTreeLockManager shouldn't try to acquire a lock from the
> InterProcessMutex instance when checking if other processes hold it.
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-23076
> URL: https://issues.apache.org/jira/browse/HBASE-23076
> Project: HBase
> Issue Type: Improvement
> Affects Versions: hbase-filesystem-1.0.0-alpha1
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Critical
> Fix For: hbase-filesystem-1.0.0-alpha2
>
>
> While going through some internal tests, our [~elserj] had faced major
> bottleneck problems when creating tables with reasonable large number of
> pre-split tables:
> {noformat} create 'josh', 'f1', {SPLITS=> (1..500).map {|i|
> "user#{1000+i*(9999-1000)/500}"}}
> {noformat}
> The above resulted in RSes taking long time to complete all assignments,
> leading to APs timeout failures from master point of view, which in turn
> submits further APs, in a cascade fashion, until RSes RPC queues got flood
> and started throwing CallQueueFullException, leaving Master with loads of
> procedures to complete and many RITs.
> Jstack analysis pointed to potential lock contentions inside
> *ZKTreeLockManager.isLocked* method. To quote [~elserj] report:
> {quote}Specifically, lots of threads that look like this:
> {noformat}
> "RpcServer.priority.FPBQ.Fifo.handler=8,queue=0,port=16020" #100 daemon
> prio=5 os_prio=0 tid=0x00007f5d6dc3a000 nid=0x6b1 waiting for monitor entry
> [0x00007f5d3bafb000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hbase.oss.thirdparty.org.apache.curator.framework.recipes.locks.LockInternals.internalLockLoop(LockInternals.java:289)
> - waiting to lock <0x000000074ddd0d10> (a
> org.apache.hadoop.hbase.oss.thirdparty.org.apache.curator.framework.recipes.locks.LockInternals)
> at
> org.apache.hadoop.hbase.oss.thirdparty.org.apache.curator.framework.recipes.locks.LockInternals.attemptLock(LockInternals.java:219)
> at
> org.apache.hadoop.hbase.oss.thirdparty.org.apache.curator.framework.recipes.locks.InterProcessMutex.internalLock(InterProcessMutex.java:237)
> at
> org.apache.hadoop.hbase.oss.thirdparty.org.apache.curator.framework.recipes.locks.InterProcessMutex.acquire(InterProcessMutex.java:108)
> at
> org.apache.hadoop.hbase.oss.sync.ZKTreeLockManager.isLocked(ZKTreeLockManager.java:310)
> at
> org.apache.hadoop.hbase.oss.sync.ZKTreeLockManager.writeLockAbove(ZKTreeLockManager.java:183)
> at
> org.apache.hadoop.hbase.oss.sync.TreeLockManager.treeReadLock(TreeLockManager.java:282)
> at
> org.apache.hadoop.hbase.oss.sync.TreeLockManager.lock(TreeLockManager.java:449)
> at
> org.apache.hadoop.hbase.oss.HBaseObjectStoreSemantics.open(HBaseObjectStoreSemantics.java:181)
> at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:166)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:911)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:566)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:559)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:545)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:241)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.executeOpenRegionProcedures(RSRpcServices.java:3626)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.lambda$executeProcedures$2(RSRpcServices.java:3694)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$$Lambda$107/1985163471.accept(Unknown
> Source)
> at java.util.ArrayList.forEach(ArrayList.java:1257)
> at
> java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3694)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29774)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> Locked ownable synchronizers:
> - None
> {noformat}
> This means that we can only open Regions in a table one at a time now (across
> all regionservers). That's pretty bad and would explain why that part was so
> slow.
> Two thoughts already:
> 1) Having to grab the lock to determine if it's held is sub-optimal. That's
> what the top of this stacktrace is and I think we need to come up with some
> other approach because this doesn't scale.
> 2) We're all blocked in reading the TableDescriptor. Maybe the Master can
> include the TableDescriptor in the OpenRegionRequest so the RS's don't have
> to read it back?
> {quote}
> From [~elserj] suggestion above, #2 would require changes at hbase project
> side, but we still can try optmize hboss *ZKTreeLockManager.isLocked* method
> as mentioned in #1.
> Looking at curator's *InterProcessMutex*, we can use its
> *getParticipantNodes()* method for checking if there's any process locking
> the given node.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
