[
https://issues.apache.org/jira/browse/HBASE-26467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447565#comment-17447565
]
Hudson commented on HBASE-26467:
--------------------------------
Results for branch branch-2
[build #398 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/398/]:
(x) *{color:red}-1 overall{color}*
----
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/398/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/398/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/398/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/398/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Wrong Cell Generated by MemStoreLABImpl.forceCopyOfBigCellInto when Cell size
> bigger than data chunk size
> ----------------------------------------------------------------------------------------------------------
>
> Key: HBASE-26467
> URL: https://issues.apache.org/jira/browse/HBASE-26467
> Project: HBase
> Issue Type: Bug
> Components: in-memory-compaction
> Reporter: zhuobin zheng
> Assignee: zhuobin zheng
> Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9
>
>
> In our company 2.X cluster. I found some region compaction keeps failling
> because some cell can't construct succefully. In fact , we even can't read
> these cell.
> From follow stack , we can found the bug cause KeyValue can't constructed.
> Simple Log and Stack:
> {code:java}
> // code placeholder
> 2021-11-18 16:50:47,708 ERROR [regionserver/xxxx:60020-longCompactions-4]
> regionserver.CompactSplit: Compaction failed
> region=xx_table,3610ff49595a0fc4a824f2a575f37a31,1570874723992.dac703ceb35e8d8703233bebf34ae49f.,
> storeName=c, priority=-319, startTime=1637225447127
> java.lang.IllegalArgumentException: Invalid tag length at position=4659867,
> tagLength=0,
> at
> org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueTagBytes(KeyValueUtil.java:685)
> at
> org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:643)
> at org.apache.hadoop.hbase.KeyValue.<init>(KeyValue.java:345)
> at
> org.apache.hadoop.hbase.SizeCachedKeyValue.<init>(SizeCachedKeyValue.java:43)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.getCell(HFileReaderImpl.java:981)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:233)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:418)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:322)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:288)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:487)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor$1.createScanner(Compactor.java:248)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:318)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
> at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1468)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2266)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:624)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:666)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748) {code}
> From further observation, I found the following characteristics:
> # Cell size more than 2M
> # We can reproduce the bug only after in memory compact
> # Cell bytes end with \x00\x02\x00\x00
>
> In fact, the root reason is method (MemStoreLABImpl.forceCopyOfBigCellInto)
> which only invoked when cell bigger than data chunk size construct cell with
> wrong length. So there are 4 bytes (chunk head size) append end of the cell
> bytes.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
