[
https://issues.apache.org/jira/browse/HBASE-22072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806509#comment-16806509
]
Anoop Sam John commented on HBASE-22072:
----------------------------------------
This is great investigation work.. Good one..
Reading the code I can see some issues very likely to happen.
1. StoreScanner#close(boolean withDelayedScannersClose)
{code}
if (withDelayedScannersClose) {
clearAndClose(scannersForDelayedClose);
clearAndClose(memStoreScannersAfterFlush);
clearAndClose(flushedstoreFileScanners);
if (this.heap != null) {
this.heap.close();
this.currentScanners.clear();
this.heap = null; // CLOSED!
}
} else {
if (this.heap != null) {
this.scannersForDelayedClose.add(this.heap);
this.currentScanners.clear();
this.heap = null;
}
}
{code}
Ideally we should add memStoreScannersAfterFlush and flushedstoreFileScanners
also into scannersForDelayedClose in else block. Mostly these will be empty by
then.
2. StoreScanner#reopenAfterFlush
{code}
List<KeyValueScanner> allScanners =
new ArrayList<>(flushedstoreFileScanners.size() +
memStoreScannersAfterFlush.size());
allScanners.addAll(flushedstoreFileScanners);
allScanners.addAll(memStoreScannersAfterFlush);
scanners = selectScannersFrom(store, allScanners);
{code}
These selected 'scanners' will get added to heap so there is no leak on them.
But it is not always sure that the selectScannersFrom call will return both the
scanners. Am not sure in your case this happening or not. But if we filter out
any of the scanners at this step, we need immediate close them.
And your case seems to mostly because of the race. I think u have a point
StoreScanner#close() Also taking flushLock.lock() might solve this?
Still I have some other Qs
reopenAfterFlush() -> Ideally this should open scanners on new SF and MS after
a flush. But seems it is done much before. If it was taken in this method only,
u would not have faced the issue. So not just that flushLock.lock() is enough
I guess. Because this reopenAfterFlush() method is called when the
StoreScanner was not really closed but a next or seek call happens. On a
closed/closing scanner it wont happen anyway. But we do take the Scanners open
in updateReaders itself. Remember any specific reason [~ram_krish]
> High read/write intensive regions may cause long crash recovery
> ---------------------------------------------------------------
>
> Key: HBASE-22072
> URL: https://issues.apache.org/jira/browse/HBASE-22072
> Project: HBase
> Issue Type: Bug
> Components: Performance, Recovery
> Affects Versions: 2.1.2
> Reporter: Pavel
> Priority: Major
>
> Compaction of high read loaded region may leave compacted files undeleted
> because of existing scan references:
> INFO org.apache.hadoop.hbase.regionserver.HStore - Can't archive compacted
> file hdfs://hdfs-ha/hbase... because of either isCompactedAway=true or file
> has reference, isReferencedInReads=true, refCount=1, skipping for now
> If region is either high write loaded this happens quite often and region may
> have few storefiles and tons of undeleted compacted hdfs files.
> Region keeps all that files (in my case thousands) untill graceful region
> closing procedure, which ignores existing references and drop obsolete files.
> It works fine unless consuming some extra hdfs space, but only in case of
> normal region closing. If region server crashes than new region server,
> responsible for that overfiling region, reads hdfs folder and try to deal
> with all undeleted files, producing tons of storefiles, compaction tasks and
> consuming abnormal amount of memory, wich may lead to OutOfMemory Exception
> and further region servers crash. This stops writing to region because number
> of storefiles reach *hbase.hstore.blockingStoreFiles* limit, forces high GC
> duty and may take hours to compact all files into working set of files.
> Workaround is a periodically check hdfs folders files count and force region
> assign for ones with too many files.
> It could be nice if regionserver had a setting similar to
> hbase.hstore.blockingStoreFiles and invoke attempt to drop undeleted
> compacted files if number of files reaches this setting.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
