Hi,
We have a large 60 node CDH 5.5.2 Hbase 1.0.0 cluster that take a very
heavy write load. For increased performance, we are using the
BufferedMutator class in hbase-client, although we're using hbase-client
version 1.2.0 because it has a small performance fix to this class.
It seems to be working fine whether the Hbase region servers are undergoing
minor compaction or not, but at some point a lot of them decide to start
auto-splitting. We configure the tables at the beginning with an explicit
set of splits for a total of 1024 regions at the beginning. By now we are
up to 4096 or so regions based on auto-splitting.
We have observed that as soon as auto-splitting starts, the mutations in
the BufferedMutator start throwing exceptions of the following form (after
retries I presume, we are configured for 5 retries and a 3 second RPC
timeout):
BufferedMutator Failed to send mutation
{"totalColumns":1,"row":"0998-f0b05361-5983-46d0-9443-4ca845987aad","families":{"s1":[{"qualifier":"s-10000123-20240198","vlen":28,"tag":[],"timestamp":1459655306487}]},"ttl":72693513}
. org.apache.hadoop.hbase.exceptions.FailedSanityCheckException:
org.apache.hadoop.hbase.exceptions.FailedSanityCheckException: Requested
row out of range for doMiniBatchMutation on HRegion
[region-actual-value-removed],
startKey='0998-UP27f54301-cf7b-11e5-858a-00163ebce99d', getEndKey()='0999',
row='0759-UPbe8df720-c9d5-11e5-947f-00163ee0563a' at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:688)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:639)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:1931)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
Does anyone know the cause for this? I am new to Hbase, in most other
distributed systems I have worked with, the system handles rerouting, so I
was expecting that the Hbase region server would automatically route to the
correct region server/region during splits.
Any help that anyone can provide would be much appreciated. I can provide
more config settings if they turn out to be relevant but there are a ton of
them in Hbase of course and I am not sure which ones would be relevant here.
Thanks in advance.
----
Saad
