Hello, We find region server abort with the following exception:
2018-05-15 08:23:23,920 ERROR [RpcServer.default.FPBQ.Fifo.handler=27,queue=7,port=16020] regionserver.HRegion: Asked to modify this region's (continuum,R\x0C\xF6\xF2\xBD\xD4L"\xB5\xFC\ xC6b\x8D\xD7\xC8x$\x7F\xFA\x9F\xA4\x92,1524491062878.117704beb050dcd3920335b4b290a898.) memstoreSize to a negative value which is incorrect. Current memstoreSize=-1533230, delta=480 java.lang.Exception at org.apache.hadoop.hbase.regionserver.HRegion.addAndGetGlobalMemstoreSize(HRegion.java:1205) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3534) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3102) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3044) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:894) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:822) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2376) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36621) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2352) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) 2018-05-15 08:23:23,922 FATAL [regionserver/dn-35.hadoop.B.GRA.infra.metrics.ovh.net/10.0.0.35:16020-splits-1525859600420] regionserver.HRegionServer: ABORTING region server dn-35.hadoo p.b.gra.infra.metrics.ovh.net,16020,1525859460041: Assertion failed while closing store continuum,RH\xFD\xA6\x88\xD7\xFB5\xBBq\xD9\xE8|\xF2I_r\x7F\xFA\xB0Y\x88,1509095154547.51aea042f53 655350c0d098fd378ab9b. v. flushableSize expected=0, actual= 23429. Current memstoreSize=-34925. Maybe a coprocessor operation failed and left the memstore in a partially updated state. 2018-05-15 08:23:23,922 FATAL [regionserver/dn-35.hadoop.B.GRA.infra.metrics.ovh.net/10.0.0.35:16020-splits-1525859600420] regionserver.HRegionServer: RegionServer abort: loaded coproce ssors are: [org.apache.hadoop.hbase.coprocessor.example.BulkDeleteEndpoint The error about memstoreSize becoming negative appear at a steady rate before abort (hundreds/sec) Any ideas? Thanks, Kevin
