[
https://issues.apache.org/jira/browse/HBASE-22433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842896#comment-16842896
]
Anoop Sam John commented on HBASE-22433:
----------------------------------------
Seems related issues.. For checking whether an already cached item, we get the
existing block and compare and in finally return block. Seems an unwanted
refCount decr happening here which causing the premature eviction of the block
and so corrupted the HFile block buffer.
> Corrupt hfile data
> ------------------
>
> Key: HBASE-22433
> URL: https://issues.apache.org/jira/browse/HBASE-22433
> Project: HBase
> Issue Type: Bug
> Affects Versions: 2.2.0
> Reporter: binlijin
> Priority: Critical
>
> We use 2.2.0 version and encounter corrupt cell data.
> {code}
> 2019-05-15 22:53:59,354 ERROR
> [regionserver/hb-mbasedata-14:16020-longCompactions-1557048533421]
> regionserver.CompactSplit: Compaction failed
> region=mktdm_id_src,99999990,1557681762973.255e9adde013e370deb595c59a7285c3.,
> storeName=o, priority=196, startTime=1557931927314
> java.lang.IllegalStateException: Invalid currKeyLen 1700752997 or
> currValueLen 2002739568. Block offset: 70452918, block length: 66556,
> position: 42364 (without header).
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.checkKeyValueLen(HFileReaderImpl.java:1182)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readKeyValueLen(HFileReaderImpl.java:628)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl._next(HFileReaderImpl.java:1080)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.next(HFileReaderImpl.java:1097)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:208)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:120)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:644)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:386)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:326)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1429)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2231)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:629)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:671)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2019-05-15 23:14:24,143 ERROR
> [regionserver/hb-mbasedata-14:16020-longCompactions-1557048533422]
> regionserver.CompactSplit: Compaction failed
> region=mktdm_id_src,9fdee4,1557681762973.1782aebb83eae551e7bdfc2bfa13eb3d.,
> storeName=o, priority=194, startTime=1557932726849
> java.lang.RuntimeException: Unknown code 98
> at org.apache.hadoop.hbase.KeyValue$Type.codeToType(KeyValue.java:274)
> at org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(CellUtil.java:1307)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.getMidpoint(HFileWriterImpl.java:383)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.finishBlock(HFileWriterImpl.java:343)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:603)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:376)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.abortWriter(DefaultCompactor.java:98)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.abortWriter(DefaultCompactor.java:42)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:335)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1429)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2231)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:629)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:671)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
