Hi, Recently we faced an issue in our HBase Cluster and following is the exception trace in client. Any idea what could be the cause and how to rectify this?
RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35}, java.io.IOException: java.io.IOException: Could not reseek StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoreFileReader$1@2e508d2b, cur=xxxxxxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/ 1498885978888/Maximum/vlen=0/seqid=0] to key xxxxxxx/dataCF:5f1e00cd-cbea- 45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0 at org.apache.hadoop.hbase.regionserver.StoreFileScanner. reseek(StoreFileScanner.java:217) at org.apache.hadoop.hbase.regionserver.StoreFileScanner. enforceSeek(StoreFileScanner.java:408) at org.apache.hadoop.hbase.regionserver.KeyValueHeap. pollRealKV(KeyValueHeap.java:363) at org.apache.hadoop.hbase.regionserver.KeyValueHeap. next(KeyValueHeap.java:123) at org.apache.hadoop.hbase.regionserver.StoreScanner. next(StoreScanner.java:588) at org.apache.hadoop.hbase.regionserver.KeyValueHeap. next(KeyValueHeap.java:150) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl. populateResult(HRegion.java:5731) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl. nextInternal(HRegion.java:5894) at org.apache.hadoop.hbase.regionserver.HRegion$ RegionScannerImpl.nextRaw(HRegion.java:5668) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next( HRegion.java:5645) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next( HRegion.java:5631) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807) at org.apache.hadoop.hbase.regionserver.RSRpcServices. get(RSRpcServices.java:2049) at org.apache.hadoop.hbase.protobuf.generated. ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop( RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: On-disk size without header provided is 36901, but block header contains 859321901. Block offset: 440164791, data starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01 at org.apache.hadoop.hbase.io.hfile.HFileBlock. validateOnDiskSizeWithoutHeader(HFileBlock.java:521) at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700( HFileBlock.java:88) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl. readBlockDataInternal(HFileBlock.java:1665) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl. readBlockData(HFileBlock.java:1532) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock( HFileReaderV2.java:452) at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader. loadDataBlockWithScanInfo(HFileBlockIndex.java:271) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ AbstractScannerV2.seekTo(HFileReaderV2.java:649) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ AbstractScannerV2.reseekTo(HFileReaderV2.java:629) at org.apache.hadoop.hbase.io.HalfStoreFileReader$1. reseekTo(HalfStoreFileReader.java:293) at org.apache.hadoop.hbase.regionserver.StoreFileScanner. reseekAtOrAfter(StoreFileScanner.java:288) at org.apache.hadoop.hbase.regionserver.StoreFileScanner. reseek(StoreFileScanner.java:200) ... 19 more ThanksU Regards, Mukund Murrali