Peter Somogyi created HBASE-26107:
-------------------------------------
Summary: MOB compaction with missing files catches incorrect
exception
Key: HBASE-26107
URL: https://issues.apache.org/jira/browse/HBASE-26107
Project: HBase
Issue Type: Bug
Components: mob
Affects Versions: 3.0.0-alpha-1
Reporter: Peter Somogyi
Assignee: Peter Somogyi
The MOB compaction catches FileNotFoundException when
{{hbase.unsafe.mob.discard.miss}} is true to handle missing MOB cells. The FNFE
is wrapped in DoNotRetryIOException so the compaction fails for the given
region.
{noformat}
2021-07-21 13:51:05,880 WARN
org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor:
hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only
when first upgrading to a version with the distributed mob compaction feature
on a cluster that has experienced MOB data corruption.
2021-07-21 13:51:05,880 WARN
org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor:
hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only
when first upgrading to a version with the distributed mob compaction feature
on a cluster that has experienced MOB data corruption.
2021-07-21 13:51:05,880 INFO
org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true
optimized configured=false optimized enabled=false maximum MOB file
size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB
family=test_cf region=3a2ee81f9244c39ba61d694e616c1a89]
2021-07-21 13:51:05,880 INFO
org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true
optimized configured=false optimized enabled=false maximum MOB file
size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB
family=test_cf region=7a96f55bb9ae04500a06cbaef02da6a3]
2021-07-21 13:51:05,888 INFO
org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting
IntegrationTestIngestWithMOB,77777777,1626787996628.c71cad04514b17ee86a407490bd27424.
2021-07-21 13:51:05,891 INFO
org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting
IntegrationTestIngestWithMOB,88888888,1626787996628.8fd002bda07755decda67b7084d1e0f6.
2021-07-21 13:51:05,895 ERROR org.apache.hadoop.hbase.regionserver.HMobStore:
The mob file
1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
could not be found in the locations
[hdfs://example.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce5
3c2/test_cf,
hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf]
or it is corrupt
2021-07-21 13:51:05,895 INFO
org.apache.hadoop.hbase.regionserver.throttle.PressureAwareThroughputController:
7a96f55bb9ae04500a06cbaef02da6a3#test_cf#compaction#1 average throughput is
0.07 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active
operations remaining, total limit is 10.00 MB/second
2021-07-21 13:51:05,908 INFO
org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting
IntegrationTestIngestWithMOB,dddddddd,1626787996628.53186ca5008e3a964eee5f96ee3f1b26.
2021-07-21 13:51:05,997 ERROR
org.apache.hadoop.hbase.regionserver.CompactSplit: Compaction failed
Request=regionName=IntegrationTestIngestWithMOB,11111111,1626787996628.7a96f55bb9ae04500a06cbaef02da6a3.,
storeName=test_cf, fileCount=1, fileSize=110.6 M (110.6 M), priority=1,
time=1626875465819
java.io.IOException: Mob compaction failed for region:
7a96f55bb9ae04500a06cbaef02da6a3
at
org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:575)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327)
at
org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.compact(DefaultMobStoreCompactor.java:227)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407)
at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2183)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:633)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:675)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException:
java.io.FileNotFoundException: File does not exist:
hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
at
org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:438)
at
org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:328)
at
org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:307)
at
org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:366)
... 10 more
Caused by: java.io.FileNotFoundException: File does not exist:
hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1500)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1493)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1508)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:140)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:104)
at
org.apache.hadoop.hbase.regionserver.HStoreFile.<init>(HStoreFile.java:220)
at
org.apache.hadoop.hbase.mob.CachedMobFile.create(CachedMobFile.java:49)
at
org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:220)
at
org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:399)
... 13 more
2021-07-21 13:51:06,013 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction of test_cf in
IntegrationTestIngestWithMOB,77777777,1626787996628.c71cad04514b17ee86a407490bd27424.
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
