Hi Stack
We are currently using hbase-1.2.5(Stable). I was not able to read the file
using HFile tool. It throws the same error. But I downloaded the file using
dfs commands. The file had very less data. Ya I saw that comment. More
details on the issue and fix will be helpful. Adding him to this thread.
Regards,
Mukund Murrali
On Wed, Jul 5, 2017 at 8:30 PM, Stack <st...@duboce.net> wrote:
> (What version of hbase?)
>
> Unfortunately the exception does not print the file name. Can you figure
> which file or region it is? Could try reading content w/ hfile tool to see
> if it complains also.
>
> Problem is:
>
> On-disk size without header provided is
> 36901, but block header contains 859321901. Block offset: 440164791, data
> starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
> at org.apache.hadoop.hbase.io.hfile.HFileBlock.
> validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
>
> I see the above exception in HBASE-13830 where Nils perhaps fixes it by
> upgrading hbase (see end comment).
>
> St.Ack
>
>
>
> On Wed, Jul 5, 2017 at 1:00 AM, mukund murrali <mukundmurra...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Recently we faced an issue in our HBase Cluster and following is the
> > exception trace in client. Any idea what could be the cause and how to
> > rectify this?
> >
> > RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35},
> > java.io.IOException: java.io.IOException: Could not reseek
> > StoreFileScanner[org.apache.hadoop.hbase.io.
> HalfStoreFileReader$1@2e508d2b
> > ,
> > cur=xxxxxxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/
> > 1498885978888/Maximum/vlen=0/seqid=0] to key
> xxxxxxx/dataCF:5f1e00cd-cbea-
> > 45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > reseek(StoreFileScanner.java:217)
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > enforceSeek(StoreFileScanner.java:408)
> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> > pollRealKV(KeyValueHeap.java:363)
> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> > next(KeyValueHeap.java:123)
> > at org.apache.hadoop.hbase.regionserver.StoreScanner.
> > next(StoreScanner.java:588)
> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> > next(KeyValueHeap.java:150)
> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> > populateResult(HRegion.java:5731)
> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> > nextInternal(HRegion.java:5894)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.nextRaw(HRegion.java:5668)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.next(
> > HRegion.java:5645)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.next(
> > HRegion.java:5631)
> > at org.apache.hadoop.hbase.regionserver.HRegion.get(
> HRegion.java:6829)
> > at org.apache.hadoop.hbase.regionserver.HRegion.get(
> HRegion.java:6807)
> > at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > get(RSRpcServices.java:2049)
> > at org.apache.hadoop.hbase.protobuf.generated.
> > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
> > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> > RpcExecutor.java:133)
> > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> java:108)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.io.IOException: On-disk size without header provided is
> > 36901, but block header contains 859321901. Block offset: 440164791, data
> > starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock.
> > validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700(
> > HFileBlock.java:88)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> > readBlockDataInternal(HFileBlock.java:1665)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> > readBlockData(HFileBlock.java:1532)
> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
> > HFileReaderV2.java:452)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$
> BlockIndexReader.
> > loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> > AbstractScannerV2.seekTo(HFileReaderV2.java:649)
> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> > AbstractScannerV2.reseekTo(HFileReaderV2.java:629)
> > at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.
> > reseekTo(HalfStoreFileReader.java:293)
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > reseekAtOrAfter(StoreFileScanner.java:288)
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > reseek(StoreFileScanner.java:200)
> > ... 19 more
> >
> > ThanksU
> > Regards,
> >
> > Mukund Murrali
> >
>
