We are again facing the issue for a different region. Any suggestions will
be highly helpful
Thanks
On Thu, 6 Jul 2017 at 11:00 AM, mukund murrali <mukundmurra...@gmail.com>
wrote:
> Hi Stack
>
> We are currently using hbase-1.2.5(Stable). I was not able to read the
> file using HFile tool. It throws the same error. But I downloaded the file
> using dfs commands. The file had very less data. Ya I saw that comment.
> More details on the issue and fix will be helpful. Adding him to this
> thread.
>
> Regards,
>
> Mukund Murrali
>
> On Wed, Jul 5, 2017 at 8:30 PM, Stack <st...@duboce.net> wrote:
>
>> (What version of hbase?)
>>
>> Unfortunately the exception does not print the file name. Can you figure
>> which file or region it is? Could try reading content w/ hfile tool to see
>> if it complains also.
>>
>> Problem is:
>>
>> On-disk size without header provided is
>> 36901, but block header contains 859321901. Block offset: 440164791, data
>> starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
>> at org.apache.hadoop.hbase.io.hfile.HFileBlock.
>> validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
>>
>> I see the above exception in HBASE-13830 where Nils perhaps fixes it by
>> upgrading hbase (see end comment).
>>
>> St.Ack
>>
>>
>>
>> On Wed, Jul 5, 2017 at 1:00 AM, mukund murrali <mukundmurra...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > Recently we faced an issue in our HBase Cluster and following is the
>> > exception trace in client. Any idea what could be the cause and how to
>> > rectify this?
>> >
>> > RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35},
>> > java.io.IOException: java.io.IOException: Could not reseek
>> > StoreFileScanner[org.apache.hadoop.hbase.io
>> .HalfStoreFileReader$1@2e508d2b
>> > ,
>> > cur=xxxxxxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/
>> > 1498885978888/Maximum/vlen=0/seqid=0] to key
>> xxxxxxx/dataCF:5f1e00cd-cbea-
>> > 45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
>> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
>> > reseek(StoreFileScanner.java:217)
>> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
>> > enforceSeek(StoreFileScanner.java:408)
>> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
>> > pollRealKV(KeyValueHeap.java:363)
>> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
>> > next(KeyValueHeap.java:123)
>> > at org.apache.hadoop.hbase.regionserver.StoreScanner.
>> > next(StoreScanner.java:588)
>> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
>> > next(KeyValueHeap.java:150)
>> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
>> > populateResult(HRegion.java:5731)
>> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
>> > nextInternal(HRegion.java:5894)
>> > at org.apache.hadoop.hbase.regionserver.HRegion$
>> > RegionScannerImpl.nextRaw(HRegion.java:5668)
>> > at org.apache.hadoop.hbase.regionserver.HRegion$
>> > RegionScannerImpl.next(
>> > HRegion.java:5645)
>> > at org.apache.hadoop.hbase.regionserver.HRegion$
>> > RegionScannerImpl.next(
>> > HRegion.java:5631)
>> > at
>> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829)
>> > at
>> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807)
>> > at org.apache.hadoop.hbase.regionserver.RSRpcServices.
>> > get(RSRpcServices.java:2049)
>> > at org.apache.hadoop.hbase.protobuf.generated.
>> > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
>> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
>> > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
>> > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
>> > RpcExecutor.java:133)
>> > at
>> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
>> > at java.lang.Thread.run(Thread.java:745)
>> > Caused by: java.io.IOException: On-disk size without header provided is
>> > 36901, but block header contains 859321901. Block offset: 440164791,
>> data
>> > starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
>> > at org.apache.hadoop.hbase.io.hfile.HFileBlock.
>> > validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
>> > at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700(
>> > HFileBlock.java:88)
>> > at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
>> > readBlockDataInternal(HFileBlock.java:1665)
>> > at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
>> > readBlockData(HFileBlock.java:1532)
>> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
>> > HFileReaderV2.java:452)
>> > at org.apache.hadoop.hbase.io
>> .hfile.HFileBlockIndex$BlockIndexReader.
>> > loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
>> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
>> > AbstractScannerV2.seekTo(HFileReaderV2.java:649)
>> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
>> > AbstractScannerV2.reseekTo(HFileReaderV2.java:629)
>> > at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.
>> > reseekTo(HalfStoreFileReader.java:293)
>> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
>> > reseekAtOrAfter(StoreFileScanner.java:288)
>> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
>> > reseek(StoreFileScanner.java:200)
>> > ... 19 more
>> >
>> > ThanksU
>> > Regards,
>> >
>> > Mukund Murrali
>> >
>>
>
> --
Regards,
Mukund Murrali
