(What version of hbase?) Unfortunately the exception does not print the file name. Can you figure which file or region it is? Could try reading content w/ hfile tool to see if it complains also.
Problem is: On-disk size without header provided is 36901, but block header contains 859321901. Block offset: 440164791, data starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01 at org.apache.hadoop.hbase.io.hfile.HFileBlock. validateOnDiskSizeWithoutHeader(HFileBlock.java:521) I see the above exception in HBASE-13830 where Nils perhaps fixes it by upgrading hbase (see end comment). St.Ack On Wed, Jul 5, 2017 at 1:00 AM, mukund murrali <mukundmurra...@gmail.com> wrote: > Hi, > > Recently we faced an issue in our HBase Cluster and following is the > exception trace in client. Any idea what could be the cause and how to > rectify this? > > RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35}, > java.io.IOException: java.io.IOException: Could not reseek > StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoreFileReader$1@2e508d2b > , > cur=xxxxxxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/ > 1498885978888/Maximum/vlen=0/seqid=0] to key xxxxxxx/dataCF:5f1e00cd-cbea- > 45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0 > at org.apache.hadoop.hbase.regionserver.StoreFileScanner. > reseek(StoreFileScanner.java:217) > at org.apache.hadoop.hbase.regionserver.StoreFileScanner. > enforceSeek(StoreFileScanner.java:408) > at org.apache.hadoop.hbase.regionserver.KeyValueHeap. > pollRealKV(KeyValueHeap.java:363) > at org.apache.hadoop.hbase.regionserver.KeyValueHeap. > next(KeyValueHeap.java:123) > at org.apache.hadoop.hbase.regionserver.StoreScanner. > next(StoreScanner.java:588) > at org.apache.hadoop.hbase.regionserver.KeyValueHeap. > next(KeyValueHeap.java:150) > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl. > populateResult(HRegion.java:5731) > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl. > nextInternal(HRegion.java:5894) > at org.apache.hadoop.hbase.regionserver.HRegion$ > RegionScannerImpl.nextRaw(HRegion.java:5668) > at org.apache.hadoop.hbase.regionserver.HRegion$ > RegionScannerImpl.next( > HRegion.java:5645) > at org.apache.hadoop.hbase.regionserver.HRegion$ > RegionScannerImpl.next( > HRegion.java:5631) > at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829) > at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807) > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > get(RSRpcServices.java:2049) > at org.apache.hadoop.hbase.protobuf.generated. > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop( > RpcExecutor.java:133) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: On-disk size without header provided is > 36901, but block header contains 859321901. Block offset: 440164791, data > starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01 > at org.apache.hadoop.hbase.io.hfile.HFileBlock. > validateOnDiskSizeWithoutHeader(HFileBlock.java:521) > at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700( > HFileBlock.java:88) > at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl. > readBlockDataInternal(HFileBlock.java:1665) > at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl. > readBlockData(HFileBlock.java:1532) > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock( > HFileReaderV2.java:452) > at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader. > loadDataBlockWithScanInfo(HFileBlockIndex.java:271) > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ > AbstractScannerV2.seekTo(HFileReaderV2.java:649) > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ > AbstractScannerV2.reseekTo(HFileReaderV2.java:629) > at org.apache.hadoop.hbase.io.HalfStoreFileReader$1. > reseekTo(HalfStoreFileReader.java:293) > at org.apache.hadoop.hbase.regionserver.StoreFileScanner. > reseekAtOrAfter(StoreFileScanner.java:288) > at org.apache.hadoop.hbase.regionserver.StoreFileScanner. > reseek(StoreFileScanner.java:200) > ... 19 more > > ThanksU > Regards, > > Mukund Murrali >