[
https://issues.apache.org/jira/browse/HBASE-26107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-26107.
-----------------------------------
Fix Version/s: 3.0.0-alpha-2
Resolution: Fixed
Thanks for the reviews [~zhangduo] and [~pankajkumar]! Pushed to master.
> MOB compaction with missing files catches incorrect exception
> -------------------------------------------------------------
>
> Key: HBASE-26107
> URL: https://issues.apache.org/jira/browse/HBASE-26107
> Project: HBase
> Issue Type: Bug
> Components: mob
> Affects Versions: 3.0.0-alpha-1
> Reporter: Peter Somogyi
> Assignee: Peter Somogyi
> Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> The MOB compaction catches FileNotFoundException when
> {{hbase.unsafe.mob.discard.miss}} is true to handle missing MOB cells. The
> FNFE is wrapped in DoNotRetryIOException so the compaction fails for the
> given region.
> {noformat}
> 2021-07-21 13:51:05,880 WARN
> org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor:
> hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only
> when first upgrading to a version with the distributed mob compaction feature
> on a cluster that has experienced MOB data corruption.
> 2021-07-21 13:51:05,880 WARN
> org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor:
> hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only
> when first upgrading to a version with the distributed mob compaction feature
> on a cluster that has experienced MOB data corruption.
> 2021-07-21 13:51:05,880 INFO
> org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true
> optimized configured=false optimized enabled=false maximum MOB file
> size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB
> family=test_cf region=3a2ee81f9244c39ba61d694e616c1a89]
> 2021-07-21 13:51:05,880 INFO
> org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true
> optimized configured=false optimized enabled=false maximum MOB file
> size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB
> family=test_cf region=7a96f55bb9ae04500a06cbaef02da6a3]
> 2021-07-21 13:51:05,888 INFO
> org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting
> IntegrationTestIngestWithMOB,77777777,1626787996628.c71cad04514b17ee86a407490bd27424.
> 2021-07-21 13:51:05,891 INFO
> org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting
> IntegrationTestIngestWithMOB,88888888,1626787996628.8fd002bda07755decda67b7084d1e0f6.
> 2021-07-21 13:51:05,895 ERROR org.apache.hadoop.hbase.regionserver.HMobStore:
> The mob file
> 1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
> could not be found in the locations
> [hdfs://example.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce5
> 3c2/test_cf,
> hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf]
> or it is corrupt
> 2021-07-21 13:51:05,895 INFO
> org.apache.hadoop.hbase.regionserver.throttle.PressureAwareThroughputController:
> 7a96f55bb9ae04500a06cbaef02da6a3#test_cf#compaction#1 average throughput is
> 0.07 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active
> operations remaining, total limit is 10.00 MB/second
> 2021-07-21 13:51:05,908 INFO
> org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting
> IntegrationTestIngestWithMOB,dddddddd,1626787996628.53186ca5008e3a964eee5f96ee3f1b26.
> 2021-07-21 13:51:05,997 ERROR
> org.apache.hadoop.hbase.regionserver.CompactSplit: Compaction failed
> Request=regionName=IntegrationTestIngestWithMOB,11111111,1626787996628.7a96f55bb9ae04500a06cbaef02da6a3.,
> storeName=test_cf, fileCount=1, fileSize=110.6 M (110.6 M), priority=1,
> time=1626875465819
> java.io.IOException: Mob compaction failed for region:
> 7a96f55bb9ae04500a06cbaef02da6a3
> at
> org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:575)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327)
> at
> org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.compact(DefaultMobStoreCompactor.java:227)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
> at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2183)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:633)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:675)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hbase.DoNotRetryIOException:
> java.io.FileNotFoundException: File does not exist:
> hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:438)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:328)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:307)
> at
> org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:366)
> ... 10 more
> Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1500)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1493)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1508)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:140)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:104)
> at
> org.apache.hadoop.hbase.regionserver.HStoreFile.<init>(HStoreFile.java:220)
> at
> org.apache.hadoop.hbase.mob.CachedMobFile.create(CachedMobFile.java:49)
> at
> org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:220)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:399)
> ... 13 more
> 2021-07-21 13:51:06,013 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting compaction of test_cf in
> IntegrationTestIngestWithMOB,77777777,1626787996628.c71cad04514b17ee86a407490bd27424.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
