[
https://issues.apache.org/jira/browse/HBASE-29131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk resolved HBASE-29131.
----------------------------------
Resolution: Fixed
Pushed to branch-2.5+.
> Introduce the option for post-compaction validation of HFiles
> -------------------------------------------------------------
>
> Key: HBASE-29131
> URL: https://issues.apache.org/jira/browse/HBASE-29131
> Project: HBase
> Issue Type: New Feature
> Components: Compaction
> Reporter: Nick Dimiduk
> Assignee: Nick Dimiduk
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.0.0-beta-2, 2.6.3, 2.5.12
>
>
> After enabling zstd compression on a table, we experienced an incident where
> data files produced during compaction were subsequently not readable. The
> client experienced an exception, the call stack for which is is below.
> Restoring after this incident was quite annoying, we'd like to avoid
> repeating the experience.
> The idea is that the region server should read back the files it writes out
> at the end of the compaction. This extra precaution would function as a
> last-ditch verification that the output file is wholesome before committing
> the compaction. If the file is not readable, the compaction would be aborted
> and we'd try again the next time a compaction is requested.
> {noformat}
> ... INFO o.a.h.h.c.AsyncRequestFutureImpl - id=1, table=XXX, attempt=4/13,
> failureCount=250ops, last exception=java.io.IOException: java.io.IOException:
> Src size is incorrect
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:504)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at
> org.apache.hadoop.hbase.ipc.CallRunnerWithContext.run(CallRunnerWithContext.java:103)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
> Caused by: com.github.luben.zstd.ZstdException: Src size is incorrect
> at
> com.github.luben.zstd.ZstdDecompressCtx.decompressDirectByteBuffer(ZstdDecompressCtx.java:172)
> at
> com.github.luben.zstd.ZstdDecompressCtx.decompress(ZstdDecompressCtx.java:241)
> at
> org.apache.hadoop.hbase.io.compress.zstd.ZstdDecompressor.decompress(ZstdDecompressor.java:73)
> at
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:88)
> at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
> at
> java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:345)
> at
> java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
> at
> java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399)
> at
> org.apache.hadoop.hbase.io.util.BlockIOUtils.readFullyWithHeapBuffer(BlockIOUtils.java:151)
> at
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:104)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.unpack(HFileBlock.java:644)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1353)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1252)
> at
> org.apache.hadoop.hbase.io.hfile.NoOpIndexBlockEncoder$NoOpEncodedSeeker.loadDataBlockWithScanInfo(NoOpIndexBlockEncoder.java:346)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReaderV2.loadDataBlockWithScanInfo(HFileBlockIndex.java:514)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:674)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:659)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:338)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:256)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:457)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:357)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:302)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:266)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:1102)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekOrSkipToNextColumn(StoreScanner.java:833)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:734)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:151)
> at
> org.apache.hadoop.hbase.regionserver.RegionScannerImpl.populateResult(RegionScannerImpl.java:339)
> at
> org.apache.hadoop.hbase.regionserver.RegionScannerImpl.nextInternal(RegionScannerImpl.java:511)
> at
> org.apache.hadoop.hbase.regionserver.RegionScannerImpl.nextRaw(RegionScannerImpl.java:275)
> at
> org.apache.hadoop.hbase.regionserver.RegionScannerImpl.next(RegionScannerImpl.java:253)
> at
> org.apache.hadoop.hbase.regionserver.RegionScannerImpl.next(RegionScannerImpl.java:240)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2652)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:859)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2892)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:437)
> ... 4 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
