i tried the following: 1) truncating system stats did not work. 2) phoenix.stats.useCurrentTime=false
but no luck - the wait time increased even further On Thu, Jun 23, 2016 at 9:04 AM, vishnu rao <jaihind...@gmail.com> wrote: > Hi Biju > > Yes local index > > . It all started when 1 box crashed. > > When I brought up a new one the error was localized to the new box. > > After cluster restart - it's spread to all servers. > > I shall attempt to clear system stats and increase meta cache size > Vishnu, > Are you using "local index" on any of the tables? We have seen similar > issues while using "local index". > > On Wed, Jun 22, 2016 at 12:25 PM, vishnu rao <jaihind...@gmail.com> wrote: > > > the server dies when trying to take the thread dump. > > > > i believe i am experiencing this bug > > > > https://issues.apache.org/jira/browse/PHOENIX-2508 > > > > On Wed, Jun 22, 2016 at 5:03 PM, Heng Chen <heng.chen.1...@gmail.com> > > wrote: > > > > > which thread hold the row lock? could you dump the jstack with 'jstack > -l > > > pid' ? > > > > > > 2016-06-22 16:14 GMT+08:00 vishnu rao <jaihind...@gmail.com>: > > > > > > > hi Heng. > > > > > > > > 2016-06-22 08:13:42,256 WARN > > > > [B.defaultRpcServer.handler=32,queue=2,port=16020] > > regionserver.HRegion: > > > > Failed getting lock in batch put, > > > > row=\x01\xD6\xFD\xC9\xDC\xE4\x08\xC4\x0D\xBESM\xC2\x82\x14Z > > > > > > > > java.io.IOException: Timed out waiting for lock for row: > > > > \x01\xD6\xFD\xC9\xDC\xE4\x08\xC4\x0D\xBESM\xC2\x82\x14Z > > > > > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5051) > > > > > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2944) > > > > > > > > at > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801) > > > > > > > > at > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743) > > > > > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692) > > > > > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654) > > > > > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) > > > > > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213) > > > > > > > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > > > > > > > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > > > > > > > > at > > > > > > > > > > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) > > > > > > > > at > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > > > > > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > On Wed, Jun 22, 2016 at 3:50 PM, Heng Chen <heng.chen.1...@gmail.com > > > > > > wrote: > > > > > > > > > Could you paste the whole jstack and relates rs log? It seems row > > > write > > > > > lock was occupied by some thread. Need more information to find > it. > > > > > > > > > > 2016-06-22 13:48 GMT+08:00 vishnu rao <jaihind...@gmail.com>: > > > > > > > > > > > need some help. this has happened for 2 of my servers > > > > > > ------------- > > > > > > > > > > > > *[B.defaultRpcServer.handler=2,queue=2,port=16020] > > > > regionserver.HRegion: > > > > > > Failed getting lock in batch put, > > > > > > row=a\xF7\x1D\xCBdR\xBC\xEC_\x18D>\xA2\xD0\x95\xFF* > > > > > > > > > > > > *java.io.IOException: Timed out waiting for lock for row: > > > > > > a\xF7\x1D\xCBdR\xBC\xEC_\x18D>\xA2\xD0\x95\xFF* > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5051) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2944) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213) > > > > > > > > > > > > at > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > > > > > > > > > > > > at > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) > > > > > > > > > > > > at > > > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > > > > > > > > > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > > > > > -- > > > > > > with regards, > > > > > > ch Vishnu > > > > > > mash213.wordpress.com > > > > > > doodle-vishnu.blogspot.in > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > with regards, > > > > ch Vishnu > > > > mash213.wordpress.com > > > > doodle-vishnu.blogspot.in > > > > > > > > > > > > > > > -- > > with regards, > > ch Vishnu > > mash213.wordpress.com > > doodle-vishnu.blogspot.in > > > -- with regards, ch Vishnu mash213.wordpress.com doodle-vishnu.blogspot.in
