+user@phoenix Hi Vishnu,
Can you try restarting the region server where you are seeing the timeout on rowlocks. Would be helpful if you share RS logs. Can you provide the details like what kind of operations done before restart and would you like the share the schemas tables. Thanks, Rajeshbabu. On Thu, Jun 23, 2016 at 9:31 AM, vishnu rao <jaihind...@gmail.com> wrote: > i tried the following: > > 1) truncating system stats did not work. > 2) phoenix.stats.useCurrentTime=false > > but no luck - the wait time increased even further > > On Thu, Jun 23, 2016 at 9:04 AM, vishnu rao <jaihind...@gmail.com> wrote: > > > Hi Biju > > > > Yes local index > > > > . It all started when 1 box crashed. > > > > When I brought up a new one the error was localized to the new box. > > > > After cluster restart - it's spread to all servers. > > > > I shall attempt to clear system stats and increase meta cache size > > Vishnu, > > Are you using "local index" on any of the tables? We have seen > similar > > issues while using "local index". > > > > On Wed, Jun 22, 2016 at 12:25 PM, vishnu rao <jaihind...@gmail.com> > wrote: > > > > > the server dies when trying to take the thread dump. > > > > > > i believe i am experiencing this bug > > > > > > https://issues.apache.org/jira/browse/PHOENIX-2508 > > > > > > On Wed, Jun 22, 2016 at 5:03 PM, Heng Chen <heng.chen.1...@gmail.com> > > > wrote: > > > > > > > which thread hold the row lock? could you dump the jstack with > 'jstack > > -l > > > > pid' ? > > > > > > > > 2016-06-22 16:14 GMT+08:00 vishnu rao <jaihind...@gmail.com>: > > > > > > > > > hi Heng. > > > > > > > > > > 2016-06-22 08:13:42,256 WARN > > > > > [B.defaultRpcServer.handler=32,queue=2,port=16020] > > > regionserver.HRegion: > > > > > Failed getting lock in batch put, > > > > > row=\x01\xD6\xFD\xC9\xDC\xE4\x08\xC4\x0D\xBESM\xC2\x82\x14Z > > > > > > > > > > java.io.IOException: Timed out waiting for lock for row: > > > > > \x01\xD6\xFD\xC9\xDC\xE4\x08\xC4\x0D\xBESM\xC2\x82\x14Z > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5051) > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2944) > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801) > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743) > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692) > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654) > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213) > > > > > > > > > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > > > > > > > > > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) > > > > > > > > > > at > > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > > > > > > > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > > > On Wed, Jun 22, 2016 at 3:50 PM, Heng Chen < > heng.chen.1...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > Could you paste the whole jstack and relates rs log? It seems > row > > > > write > > > > > > lock was occupied by some thread. Need more information to find > > it. > > > > > > > > > > > > 2016-06-22 13:48 GMT+08:00 vishnu rao <jaihind...@gmail.com>: > > > > > > > > > > > > > need some help. this has happened for 2 of my servers > > > > > > > ------------- > > > > > > > > > > > > > > *[B.defaultRpcServer.handler=2,queue=2,port=16020] > > > > > regionserver.HRegion: > > > > > > > Failed getting lock in batch put, > > > > > > > row=a\xF7\x1D\xCBdR\xBC\xEC_\x18D>\xA2\xD0\x95\xFF* > > > > > > > > > > > > > > *java.io.IOException: Timed out waiting for lock for row: > > > > > > > a\xF7\x1D\xCBdR\xBC\xEC_\x18D>\xA2\xD0\x95\xFF* > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5051) > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2944) > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801) > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743) > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692) > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654) > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213) > > > > > > > > > > > > > > at > > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > > > > > > > > > > > > > > at > > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) > > > > > > > > > > > > > > at > > > > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > > > > > > > > > > > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > > > > > > > -- > > > > > > > with regards, > > > > > > > ch Vishnu > > > > > > > mash213.wordpress.com > > > > > > > doodle-vishnu.blogspot.in > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > with regards, > > > > > ch Vishnu > > > > > mash213.wordpress.com > > > > > doodle-vishnu.blogspot.in > > > > > > > > > > > > > > > > > > > > > -- > > > with regards, > > > ch Vishnu > > > mash213.wordpress.com > > > doodle-vishnu.blogspot.in > > > > > > > > > -- > with regards, > ch Vishnu > mash213.wordpress.com > doodle-vishnu.blogspot.in >
