[
https://issues.apache.org/jira/browse/HBASE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987603#comment-16987603
]
Anoop Sam John commented on HBASE-23349:
----------------------------------------
bq.Agree, because of this, our auto-reopen of region is not happening based on
max refCount among compacted store files.
The ref count based decision system wont work at all I fear! Because what value
u will give for this config? when compaction happened and so the file become
eligible for arhival, there can be 1 or more (Any number) scanners over this
file. So the ref count would have been 1+. Later it is possible that many of
these scanners are properly closed say apart from one. So the ref count would
become 1. Any reopen decision based on a larger value than 1 wont help us
then!. The point is that once the compaction is over, these files are moved
out of this Store's HFiles. It just sits there. The newer scanners wont even
touch that and so possibly increase the ref counts. So the leaks wont make this
ref count to grow large over the time. Said that any decision based on ref
count wont work !
My thinking is we should do a time based decision. We have the Discharger
thread running at intervals which will archive the files with 0 ref count. Even
if the ref count is >0 and the file is already compacted away before a long
time (Configurable) we should force archive it. We should NOT do Region open
for solving this at all. Infact no need for that. A region reopen will cause
many other issues like invalidation of its data from block cache! In the past,
before this Discharger thread way of archiving files, we used to archive
compacted away files immediately after a compaction op. The current scanners
used to reopen the scanners on the new files. So it is possible that even if
some valid old scans still running over the compacted away files can still
continue its scan. So we would need to bring back the old way (not completely
though)
> Expose max refCount among all compacted store files of a region
> ---------------------------------------------------------------
>
> Key: HBASE-23349
> URL: https://issues.apache.org/jira/browse/HBASE-23349
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 3.0.0, 2.3.0, 1.6.0
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
> Attachments: HBASE-23349.master.000.patch,
> HBASE-23349.master.001.patch, HBASE-23349.master.002.patch
>
>
> We should expose a region level metric that represents max refCount among
> refCounts of all compacted store files under the region. For successful
> archival of compacted store files, it is important for this metric count to
> be 0 eventually if not immediately. If it is >0 for a considerably high
> amount of time, it indicates some issue i.e. reader refCount leak on some
> compacted store files and in such case, archival would not be successful.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
