[
https://issues.apache.org/jira/browse/HBASE-22072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16805785#comment-16805785
]
Pavel commented on HBASE-22072:
-------------------------------
Sorry for a long time to answer, I've made some investigations catching
unclosed scanners.
Each StoreFile has an AtomicInteger refCount, serving by StoreFileReader
incrementing by each StoreFileScanner and decreeing when reading completes.
This refCount==0 alows dropping StoreFile after compaction. I added identifier
to StoreFileScanner class as follows (all code examples from rel/2.1.4)
{code:java}
public StoreFileScanner(StoreFileReader reader, HFileScanner hfs, boolean
useMVCC,
boolean hasMVCC, long readPt, long scannerOrder, boolean
canOptimizeForNonNullColumn) {
this.readPt = readPt;
this.reader = reader;
this.hfs = hfs;
this.enforceMVCC = useMVCC;
this.hasMVCCInfo = hasMVCC;
this.scannerOrder = scannerOrder;
this.canOptimizeForNonNullColumn = canOptimizeForNonNullColumn;
this.identifier = new Timestamp(System.currentTimeMillis()) + "$" +
System.identityHashCode(this);
this.reader.incrementRefCount(identifier);
}
{code}
and pass it to incrementRefCount of StoreFileReader who now has
{code:java}
private Set<String> scanners = ConcurrentHashMap.newKeySet();{code}
to get more information about active scanners
{code:java}
void incrementRefCount(String storeFileScannerName) {
int rc = refCount.incrementAndGet();
scanners.add(storeFileScannerName);
if (LOG.isDebugEnabled()){
String message = "Increment refCount of "
+ reader.getPath()
+ "(" + rc + ") by " + storeFileScannerName + ": "
+ Thread.currentThread().getId() + "#"
+ Thread.currentThread().getName();
LOG.debug(org.apache.commons.lang.exception.ExceptionUtils.getFullStackTrace(new
NullPointerException(message)));
}
}
{code}
StoreFileScanner removing from Set after scanner closing
{code:java}
void readCompleted(String storeFileScannerName) {
int rc = refCount.decrementAndGet();
scanners.remove(storeFileScannerName);
if (LOG.isDebugEnabled()){
String message = "Decrement refCount of "
+ reader.getPath()
+ "(" + rc + ") by " + storeFileScannerName + ": "
+ Thread.currentThread().getId() + "#"
+ Thread.currentThread().getName();
LOG.debug(message);
.....
{code}
I also added this set of scanners to log message of Chore service and finally
got unclosed StoreFileScanner, created as follows
{noformat}
org.apache.hadoop.hbase.regionserver.StoreFileReader.incrementRefCount(StoreFileReader.java:172)
org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:97)
org.apache.hadoop.hbase.regionserver.StoreFileReader.getStoreFileScanner(StoreFileReader.java:155)
org.apache.hadoop.hbase.regionserver.HStoreFile.getPreadScanner(HStoreFile.java:504)
org.apache.hadoop.hbase.regionserver.StoreFileScanner.getScannersForStoreFiles(StoreFileScanner.java:147)
org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1309)
org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1276)
org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:891)
org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1195)
org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1171)
org.apache.hadoop.hbase.regionserver.HStore.access$600(HStore.java:131)
org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2302)
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2741)
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2467)
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2439)
org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2329)
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:612)
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:581)
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68)
org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:361)
java.lang.Thread.run(Thread.java:748)
{noformat}
It looks like then we have active StoreScanners on MemStore, which is flushing
to StoreFile, we close all MemStoreScanners and open new StoreFileScanners to
make it possible for client scanner to continue reading. And for some reason
this StoreFileScanners remain unclosed. I could not find certain place in code,
working wrong and ask dev community for assist.
I can give more details if needed by changing logging though it require new
build and production regionserver restart as far as I can not reproduce this
behavior on test installation.
> High read/write intensive regions may cause long crash recovery
> ---------------------------------------------------------------
>
> Key: HBASE-22072
> URL: https://issues.apache.org/jira/browse/HBASE-22072
> Project: HBase
> Issue Type: Bug
> Components: Performance, Recovery
> Affects Versions: 2.1.2
> Reporter: Pavel
> Priority: Major
>
> Compaction of high read loaded region may leave compacted files undeleted
> because of existing scan references:
> INFO org.apache.hadoop.hbase.regionserver.HStore - Can't archive compacted
> file hdfs://hdfs-ha/hbase... because of either isCompactedAway=true or file
> has reference, isReferencedInReads=true, refCount=1, skipping for now
> If region is either high write loaded this happens quite often and region may
> have few storefiles and tons of undeleted compacted hdfs files.
> Region keeps all that files (in my case thousands) untill graceful region
> closing procedure, which ignores existing references and drop obsolete files.
> It works fine unless consuming some extra hdfs space, but only in case of
> normal region closing. If region server crashes than new region server,
> responsible for that overfiling region, reads hdfs folder and try to deal
> with all undeleted files, producing tons of storefiles, compaction tasks and
> consuming abnormal amount of memory, wich may lead to OutOfMemory Exception
> and further region servers crash. This stops writing to region because number
> of storefiles reach *hbase.hstore.blockingStoreFiles* limit, forces high GC
> duty and may take hours to compact all files into working set of files.
> Workaround is a periodically check hdfs folders files count and force region
> assign for ones with too many files.
> It could be nice if regionserver had a setting similar to
> hbase.hstore.blockingStoreFiles and invoke attempt to drop undeleted
> compacted files if number of files reaches this setting.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
