Hi,
Recently we faced an issue in our HBase Cluster and following is the
exception trace in client. Any idea what could be the cause and how to
rectify this?
RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35},
java.io.IOException: java.io.IOException: Could not reseek
StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoreFileReader$1@2e508d2b,
cur=xxxxxxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/
1498885978888/Maximum/vlen=0/seqid=0] to key xxxxxxx/dataCF:5f1e00cd-cbea-
45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
reseek(StoreFileScanner.java:217)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
enforceSeek(StoreFileScanner.java:408)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
pollRealKV(KeyValueHeap.java:363)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
next(KeyValueHeap.java:123)
at org.apache.hadoop.hbase.regionserver.StoreScanner.
next(StoreScanner.java:588)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
next(KeyValueHeap.java:150)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
populateResult(HRegion.java:5731)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
nextInternal(HRegion.java:5894)
at org.apache.hadoop.hbase.regionserver.HRegion$
RegionScannerImpl.nextRaw(HRegion.java:5668)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(
HRegion.java:5645)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(
HRegion.java:5631)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.
get(RSRpcServices.java:2049)
at org.apache.hadoop.hbase.protobuf.generated.
ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: On-disk size without header provided is
36901, but block header contains 859321901. Block offset: 440164791, data
starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
at org.apache.hadoop.hbase.io.hfile.HFileBlock.
validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700(
HFileBlock.java:88)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
readBlockDataInternal(HFileBlock.java:1665)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
readBlockData(HFileBlock.java:1532)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
HFileReaderV2.java:452)
at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.
loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
AbstractScannerV2.seekTo(HFileReaderV2.java:649)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
AbstractScannerV2.reseekTo(HFileReaderV2.java:629)
at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.
reseekTo(HalfStoreFileReader.java:293)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
reseekAtOrAfter(StoreFileScanner.java:288)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
reseek(StoreFileScanner.java:200)
... 19 more
ThanksU
Regards,
Mukund Murrali
